Commit Graph

9 Commits

Author SHA1 Message Date
Lampros Smyrnaios 9fdaa9503b - Delete any left-over full-texts after 36 hours.
- Upon shutting down, post a "shutdownReport" to the Controller.
2023-05-23 22:22:57 +03:00
Lampros Smyrnaios 903032f454 - After a WorkerReport has been sent, ask for new assignments immediately. So, the Worker does not have to wait for hours for the Controller to check for duplicate files in the DB, retrieve and upload the full-texts and insert the records to the DB.
- Special care is taken to delete the delivered full-texts as soon as possible.
- Write the workerReport to a json-file, in case something goes wrong, and keep it until the Controller notifies the Worker that the processing was successful.
2023-05-23 22:19:41 +03:00
Lampros Smyrnaios 4d90846261 - In case the specified "controllerIP" is actually a domain-name, find its IP-address, so that a proper IP-to-IP comparison can be performed and the "securityChecks" can pass.
- Increase the "read-timeout" when searching for the host's machine public-IP.
- Update dependencies.
- Code polishing.
2023-05-22 21:25:22 +03:00
Lampros Smyrnaios 839a797124 - Improve performance of full-texts transferring to the Controller, by preloading some bytes for faster response to the Controller's read requests.
- Optimize directories-creation process by eliminating the additive check for existence, as that check already takes place inside the "mkdirs()" method.
- Remove the obsolete code which in case the specific assignments' subdirectory failed to be created, then a different base-dir was used instead. Since the user-defined baseDir is already been successfully created upon initialization, any problem on creating subdirectories inside that base-directory will most likely persist even when changing the base directory. Additionally, even if the subdirectory with the changed base-directory succeeded, the "FullTextsController.getFullTexts()" method would not use it, resulting in errors.
- Code polishing.
2023-03-08 13:12:17 +02:00
Lampros Smyrnaios ec09ecc7ff - Refactor and Spring-ify the File-storage initialization process.
- Fix the problematic file-storage-path (it could not be used when the Controller was requesting the full-texts), which was produced when the user-defined path could not be created.
2023-03-07 16:21:32 +02:00
Lampros Smyrnaios ff4fd3d289 - Show the elapsed time for each assignments-request to be processed by the Worker.
- Update dependencies.
2023-03-02 17:34:44 +02:00
Lampros Smyrnaios 66d3f7bcb2 - Show a warning, in case the number of results is different from the number of the assignments (due to missing / double logging).
- Update Spring.
2023-02-24 23:27:02 +02:00
Lampros Smyrnaios 84a37bd4b7 - Handle the case, where an instance of a urlReport record (having the same id and sourceUrl), may have failed to give a docUrl, due to en error, even if another instance gives the docUrl and the docFile. The absence of that handling could lead to a record-instance, being assigned a "fileLocation" which was actually an error-message (comment), and as a result the real "fileLocation" would have never been reached to be assigned, so the payload would be lost.
- Improve exceptions-handling.
2023-02-21 15:22:49 +02:00
Lampros Smyrnaios 24b52fba63 - Refactor the initialization and configuration process and Spring-ify the project.
- Update Spring dependency.
2023-01-25 18:33:49 +02:00