UrlsController/src/main/java/eu/openaire/urls_controller/util
Lampros Smyrnaios 8dfb58ee63 Avoid assigning the same publications multiple times to the Workers, after the recent "parallelization enchantment".
After that enchantment, each worker could request multiple assignment-batches, before its previous batches were processed by the Controller. This means that for each batch that was processed, the Controller was deleting from the "assignment" table, all the assignments (-batches) delivered to the Worker that brought that batch, even though the "attempt" and "payload" records for the rest of the batches were not inserted in the DB yet. So in a new assignments-batch request, the same publications that were already under processing, were delivered to the same or other Workers.
Now, for each finished batch, only the assignments of that batch are deleted from the "assignment" table.
2023-07-11 17:27:23 +03:00
..
FileDecompressor.java Fix a bug, which caused the full-text files to never close. 2023-05-04 13:03:28 +03:00
FileUtils.java Avoid assigning the same publications multiple times to the Workers, after the recent "parallelization enchantment". 2023-07-11 17:27:23 +03:00
GenericUtils.java - Increase the initialDelay for the "checkIfServiceIsReadyForShutdown" scheduled-task, in production, to 10 minutes. 2023-06-06 16:49:53 +03:00
ParquetFileUtils.java - Fix not using actual the "currentAssignmentsBatch" of the workerReport itself, when creating the parquetFileNames and when reporting to the user the initialization of the "addition of the workerReport". 2023-06-27 16:08:01 +03:00
S3ObjectStore.java Add an extra precaution-check to allow the emptying or deletion of an S3-Object-Store bucket, only when the app runs in "TestEnvironment". 2023-02-01 16:42:22 +02:00
TestFileUtils.java - Optimize file-related tasks. 2022-05-26 15:43:59 +03:00
UriBuilder.java - Add the "getWorkersInfo" endpoint. 2023-05-23 14:57:15 +03:00