Lampros Smyrnaios
4d90846261
- In case the specified "controllerIP" is actually a domain-name, find its IP-address, so that a proper IP-to-IP comparison can be performed and the "securityChecks" can pass.
...
- Increase the "read-timeout" when searching for the host's machine public-IP.
- Update dependencies.
- Code polishing.
2023-05-22 21:25:22 +03:00
Lampros Smyrnaios
bd0ead816d
Make the value of time-out for "restTemplateForReport", to scale along the "maxAssignmentsLimitPerBatch".
2023-05-16 19:08:59 +03:00
Lampros Smyrnaios
d5a997ad3d
Use restTemplates with different read timeouts depending on the operation. For the assignments-request we need a shorter read timeout, than the one we need for the worker-report. This guarantees that the connection does not hungs for so long, when the Controller crashes before sending the assignments.
2023-04-29 17:24:16 +03:00
Lampros Smyrnaios
0ba15dd31a
Increase the "requestReadTimeoutDuration" to 10 hours, as the number of full-texts to be transferred to the Controller keeps getting larger.
2023-04-26 15:08:46 +03:00
Lampros Smyrnaios
839a797124
- Improve performance of full-texts transferring to the Controller, by preloading some bytes for faster response to the Controller's read requests.
...
- Optimize directories-creation process by eliminating the additive check for existence, as that check already takes place inside the "mkdirs()" method.
- Remove the obsolete code which in case the specific assignments' subdirectory failed to be created, then a different base-dir was used instead. Since the user-defined baseDir is already been successfully created upon initialization, any problem on creating subdirectories inside that base-directory will most likely persist even when changing the base directory. Additionally, even if the subdirectory with the changed base-directory succeeded, the "FullTextsController.getFullTexts()" method would not use it, resulting in errors.
- Code polishing.
2023-03-08 13:12:17 +02:00
Lampros Smyrnaios
4da54e7a7d
- Show a warning, in case the number of archived files is different from the number of requested files.
...
- Code polishing.
- Update Gradle.
2023-03-07 16:25:10 +02:00
Lampros Smyrnaios
ff4fd3d289
- Show the elapsed time for each assignments-request to be processed by the Worker.
...
- Update dependencies.
2023-03-02 17:34:44 +02:00
Lampros Smyrnaios
81b61b530f
Drastically improve performance by applying a pre-processing algorithm for the assignments-list to open some "space" between assignments which have the same domain, which in return, causes the threads to block less during execution.
...
(The threads block, due to the mandatory "politeness-delay" before reconnecting with the same domain, in order to avoid overloading the remote servers.)
2023-02-24 23:23:37 +02:00
Lampros Smyrnaios
84a37bd4b7
- Handle the case, where an instance of a urlReport record (having the same id and sourceUrl), may have failed to give a docUrl, due to en error, even if another instance gives the docUrl and the docFile. The absence of that handling could lead to a record-instance, being assigned a "fileLocation" which was actually an error-message (comment), and as a result the real "fileLocation" would have never been reached to be assigned, so the payload would be lost.
...
- Improve exceptions-handling.
2023-02-21 15:22:49 +02:00
Lampros Smyrnaios
24b52fba63
- Refactor the initialization and configuration process and Spring-ify the project.
...
- Update Spring dependency.
2023-01-25 18:33:49 +02:00