UrlsController/src/main/java/eu/openaire/urls_controller
Lampros Smyrnaios dc8f0f2bd1 - Heavily reduce the maximum amount of space needed, by deleting the files of each full-texts batch, right after they are uploaded to the S3 Object Store.
- Add a check for when the retrieved full-texts-batch is missing some requested files and show a warn-log.
- Update dependencies.
2023-01-23 20:23:21 +02:00
..
components - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 2021-12-21 15:55:27 +02:00
configuration - Fix not finding the parquet-schema files when the app was run inside a Docker Container. 2022-12-08 12:16:05 +02:00
controllers - Use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits on compression rate and speed. 2023-01-10 13:34:54 +02:00
models - Apply error-checking on individual CallableTasks and in tasks-batches related to the creation and upload of all the data related to the "attempt" and "payload" table. So, if no data could be uploaded for one or both tables, no "load"-queries will be executed for that/those tables. 2022-12-09 12:46:06 +02:00
payloads - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction. 2021-11-30 13:26:19 +02:00
security - Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated. 2022-06-27 21:41:32 +03:00
util - Heavily reduce the maximum amount of space needed, by deleting the files of each full-texts batch, right after they are uploaded to the S3 Object Store. 2023-01-23 20:23:21 +02:00
Application.java Improve performance of the hash-checking algorithm by using multithreading. 2022-12-15 18:34:28 +02:00