Commit Graph

8 Commits

Author SHA1 Message Date
Lampros Smyrnaios 18cc9e0e68 - Improve error-handling in file-compression.
- Update dependencies.
2023-10-04 16:08:38 +03:00
Lampros Smyrnaios 903032f454 - After a WorkerReport has been sent, ask for new assignments immediately. So, the Worker does not have to wait for hours for the Controller to check for duplicate files in the DB, retrieve and upload the full-texts and insert the records to the DB.
- Special care is taken to delete the delivered full-texts as soon as possible.
- Write the workerReport to a json-file, in case something goes wrong, and keep it until the Controller notifies the Worker that the processing was successful.
2023-05-23 22:19:41 +03:00
Lampros Smyrnaios ec4d084972 Reduce memory usage. 2023-04-28 17:59:36 +03:00
Lampros Smyrnaios 0997558347 Update dependencies. 2023-04-20 15:39:15 +03:00
Lampros Smyrnaios 839a797124 - Improve performance of full-texts transferring to the Controller, by preloading some bytes for faster response to the Controller's read requests.
- Optimize directories-creation process by eliminating the additive check for existence, as that check already takes place inside the "mkdirs()" method.
- Remove the obsolete code which in case the specific assignments' subdirectory failed to be created, then a different base-dir was used instead. Since the user-defined baseDir is already been successfully created upon initialization, any problem on creating subdirectories inside that base-directory will most likely persist even when changing the base directory. Additionally, even if the subdirectory with the changed base-directory succeeded, the "FullTextsController.getFullTexts()" method would not use it, resulting in errors.
- Code polishing.
2023-03-08 13:12:17 +02:00
Lampros Smyrnaios 4da54e7a7d - Show a warning, in case the number of archived files is different from the number of requested files.
- Code polishing.
- Update Gradle.
2023-03-07 16:25:10 +02:00
Lampros Smyrnaios ba989484e4 Improve performance when archiving and compressing the full-texts. 2023-03-02 17:47:58 +02:00
Lampros Smyrnaios fd62ac567e - Add a new endpoint "getFullTextsImproved" which uses Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits on compression rate and speed.
- Remove some dependencies.
2023-01-09 15:48:30 +02:00