UrlsController/src/main/java/eu/openaire/urls_controller
Lampros Smyrnaios 15224c6468 Improve performance in the "getUrls"-endpoint, and more:
- Optimize the "findAssignmentsQuery" by using an inner limit (larger than the outer).
- Save a ton of time from inserting the assignments into the database, by using a temporal table to hold the new assignments, in order for them to be easily accessible both from the Controller (which processes them and send them to the Worker) and the database itself, in order to "import" them into the "assignment"-table.
- Replace the "Date" with "Timestamp", in order to hold more detailed information.
- Code cleanup.
2021-11-30 19:59:46 +02:00
..
components - Implement the "getUrls" and "addWorkerReport" endpoints with full database-handling. 2021-11-09 23:59:27 +02:00
configuration - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction. 2021-11-30 13:26:19 +02:00
controllers Improve performance in the "getUrls"-endpoint, and more: 2021-11-30 19:59:46 +02:00
models Improve performance in the "getUrls"-endpoint, and more: 2021-11-30 19:59:46 +02:00
payloads - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction. 2021-11-30 13:26:19 +02:00
security - Add the "isControllerAlive"-endpoint. 2021-09-23 15:08:52 +03:00
util - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 2021-11-30 18:23:27 +02:00
Application.java - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 2021-11-30 18:23:27 +02:00