UrlsController/src/main/java/eu/openaire/urls_controller
Lampros Smyrnaios b702cf4484 Upgrade the "findAssignmentsQuery":
- Retrieve the assignments by checking only the publication-urls against the "attempt", "assignment" and "payload" tables, not the IDs. This change allow us to: a) avoid re-attempting urls which have already been attempted multiple times (by different id-url pairs), b) avoid aggregating urls which are already inside the "payload" or "assignment" tables, even when they are related with other IDs.
In the end, we only care about the urls when choosing which records should be aggregated.
- Improve performance by using the "anti join" operator, where it fits, in order to allow the engine to use the faster "hash" operations.
2023-10-04 13:43:15 +03:00
..
components - Add bulk-import support for non-Authoritative data-sources. 2023-09-26 18:02:48 +03:00
configuration - Rename "ImpalaConnector.java" to "DatabaseConnector.java". 2023-08-23 16:55:23 +03:00
controllers - Add bulk-import support for non-Authoritative data-sources. 2023-09-26 18:02:48 +03:00
models Update Bulk-Import API: 2023-07-25 11:59:47 +03:00
payloads - Process the WorkerReports in background Jobs and post the reportResults to the Workers. 2023-05-24 13:52:28 +03:00
security - Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated. 2022-06-27 21:41:32 +03:00
services Upgrade the "findAssignmentsQuery": 2023-10-04 13:43:15 +03:00
util Handle the case when the "upload-file-to-S3" operation fails with a "ConnectException". In this case, all remaining upload operations for the files of that particular batch or segment, are canceled. 2023-10-04 13:01:13 +03:00
Application.java - Fix missing changes. 2023-07-24 19:55:37 +03:00