lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 17 Wiki Activity
180 Commits 1 Branch 17 Tags 1.5 MiB
Commit Graph

180 Commits

Author SHA1 Message Date
Lampros Smyrnaios b70ae3ed58 - Improve error-handling 
- Use new class-model from PublicationsRetriever software.
- Set new version.
 2024-12-06 00:23:46 +02:00
Lampros Smyrnaios 83e6d761dd - Increase the initial delay for "checkAndDeleteOldFiles" to 54 hours, in order to avoid obsolete checks earlier, since a directory needs to have been last accessed at least 48 hours ago, while some additional hours are needed for the first batch to be finished. 
- Improve a log-message.
 2024-12-05 17:56:15 +02:00
Lampros Smyrnaios fa7b2da926 Make sure the "already-retrieved" fulltexts are deleted as early as possible. These are the files which have their hash been added in the Database sometime in the past and the Controller has not requested them now. 2024-12-04 00:59:00 +02:00
Lampros Smyrnaios bb521cc493 Update the logic behind sending the "shutdownReport" to the Controller: 
- Make sure the Controller is being informed for the about-to-shut-down state of the worker, even if it is being shut-down without the use of the api.
- Send the report as late as possible, in order for the Controller to know the exact time the Worker was shutdown.
 2024-12-04 00:45:22 +02:00
Lampros Smyrnaios 6e4f91f20d - Set next version. 
- Code polishing.
- Update dependencies.
- Optimize Gradle's configuration.
 2024-12-04 00:26:06 +02:00
Lampros Smyrnaios f05c98de20 Revert back the reduction in the amount of "requiredFreeSpace" needed to be available in order to accept new assignments, which happened in commit: 4af74d4581 2024-12-03 19:54:11 +02:00
Lampros Smyrnaios 393c9e037e Add missing import. 2024-11-13 20:02:03 +02:00
Lampros Smyrnaios 40ffc572e3 Add missing call. 2024-11-13 19:59:48 +02:00
Lampros Smyrnaios ef33e91553 - Use newer api form "PublicationsRetriever" software. 
- Optimize memory-de-allocation.
- Optimize Gradle settings.
- Set new version.
- Update dependencies.
 2024-11-13 19:10:05 +02:00
Lampros Smyrnaios 3eaeff468a Set new version. 2024-10-25 19:04:21 +03:00
Lampros Smyrnaios 0b3ee3e16e - Set next version. 
- Update dependencies.
 2024-10-24 20:22:40 +03:00
Lampros Smyrnaios ea17ec917b - Set new version. 
- Update dependencies.
 2024-06-11 11:59:46 +03:00
Lampros Smyrnaios d630f16198 Improve the compression of fulltext files: 
- Fix not using the big bufferSize it was supposed to use.
- Make sure the maximum compression-level is used. Before, the invalid value "bufferSize" was passed as the level, and it is unclear to which real-compression level it was changed to, inside the zstd-library (19 or 22 (only allowed though "ultra mode")), probably to the ultra-level though, as this "switch" seems to be required only through the cli.
- Exclude the possibly outdated "commons-compress" transitive dependency from the "publications_retriever" dependency.
 2024-06-10 18:21:35 +03:00
Lampros Smyrnaios 107908a733 - Fix not deleting the "assignments_*" directory, along with the potentially partially created zstd file, in case there was a compression error. 
- Show the number of files which were successfully compressed, in each batch.
- Fix the class-value used in the Logger-initializer, in "FullTextsController".
- Improve an error-log.
 2024-05-30 12:29:02 +03:00
Lampros Smyrnaios 4af74d4581 - Reduce the amount of "requiredFreeSpace" needed to be available in order to accept new assignments. 
- Increase the time to wait before rechecking the available free space, in order to get new assignments, to 30 minutes.
- Update dependencies.
- Code polishing.
 2024-05-28 23:10:52 +03:00
Lampros Smyrnaios c242f65518 - Improve error-handling in "ConnWithController.postShutdownReportToController()". 
- Update dependencies.
 2024-05-22 16:14:45 +03:00
Lampros Smyrnaios b40c72f78f - Fix the process of shutting down the worker, in case the user sends the relevant request, while the worker is stuck in a data-request error-loop. 
- Upload the updated gradle-wrapper.
 2024-04-29 17:08:40 +03:00
Lampros Smyrnaios 34407179fc Update Gradle in the install script. 2024-04-26 15:02:50 +03:00
Lampros Smyrnaios 7857ce1f05 Update Gradle. 2024-04-26 15:02:07 +03:00
Lampros Smyrnaios 795d6e7c93 - Update README. 
- Update dependencies.
 2024-04-26 13:36:41 +03:00
Lampros Smyrnaios 736d0f8526 Add a missing change in logback-spring.xml. 2024-02-08 20:04:45 +02:00
Lampros Smyrnaios cb736a8d66 Add the Jenkins' build-status badge in README. 2024-02-08 19:12:04 +02:00
Lampros Smyrnaios 5d7465df3c Add some gradle files to be used by Jenkins. 2024-02-08 19:06:54 +02:00
Lampros Smyrnaios 24c4a75acf - Use the "RollingFile" logs-appender by default. 
- Set the next version.
 2024-02-08 18:51:10 +02:00
Lampros Smyrnaios 50d756d582 - Automatically use the latest version of "publications_retriever" software from the Nexus maven-repository. 
- Update Gradle.
- Update License.
- Configure the destination of the logs in the "application.properties" file.
 2024-02-08 18:33:18 +02:00
Lampros Smyrnaios 3909104a1b - Update a dependency. 
- Set new version.
 2024-01-15 13:54:12 +02:00
Lampros Smyrnaios c4770ee716 Set new version. 2023-12-22 12:40:26 +02:00
Lampros Smyrnaios 066d6f665f - Take into account the new "errorMsg" value returned by "LoaderAndChecker.getWasValidAndCouldRetry()". 
- Update dependencies.
 2023-12-18 15:17:51 +02:00
Lampros Smyrnaios bad9544c58 - Improve error-handling. 
- Improve a log-message.
 2023-11-29 13:49:54 +02:00
Lampros Smyrnaios 5a9e7228ae - Set the upcoming version. 
- Update dependencies.
 2023-11-27 13:02:33 +02:00
Lampros Smyrnaios 9073f56227 Revert the "read-timeout" value back to 1 hour, as there is no that big of a problem with the load of either server, it's a frequent network-"lag" that causes the issue, which is not solved even with 2 hours of waiting. 2023-10-31 15:08:57 +02:00
Lampros Smyrnaios 69ea5b6d19 - Increase the "ReadTimeout" to 2 hours, as the Worker struggles to get the assignments-data in time. 
- Revert the change about special handling of the "RestClientException". The exMsg was appearing in a different line, in the logs, and was a "SocketTimeoutException".
 2023-10-27 18:39:10 +03:00
Lampros Smyrnaios bfa76e9484 - Show the full stacktrace in the weird case of a "RestClientException" without an exception-message. Also, in this case, retry immediately, as there is no long-lasting network problem that requires some time between requests, but most probably a random interruption. 
- Code polishing.
 2023-10-27 17:36:54 +03:00
Lampros Smyrnaios 10e39d79a4 - Improve a log-message. 
- Update dependencies.
 2023-10-20 17:35:39 +03:00
Lampros Smyrnaios 1b45f384a7 - In case a faulty "assignmentsCounter" was given to the "addReportResultToWorker"-endpoint, then return an explanatory error-message along with the HTTP-404 error. 
- Update Gradle.
 2023-10-06 15:45:53 +03:00
Lampros Smyrnaios 01e378ea66 - Add progress-report-log for assignments-processing. 
- Code polishing.
 2023-10-05 12:02:52 +03:00
Lampros Smyrnaios 18cc9e0e68 - Improve error-handling in file-compression. 
- Update dependencies.
 2023-10-04 16:08:38 +03:00
Lampros Smyrnaios 2895668417 - Add LICENSE. 
- Code polishing.
 2023-09-14 16:09:20 +03:00
Lampros Smyrnaios 49cd0c19c2 - Increase the "hoursToWaitBeforeDeletion" to 48. 
- Adjust the number and size of log files.
 2023-08-31 17:54:07 +03:00
Lampros Smyrnaios e85282d35b Update the "addReportResultToWorker"-endpoint to check if the given "assignmentsCounter" was handled by that worker, without considering the related full-texts directory, since that may have been deleted in the meantime. 2023-08-31 17:52:52 +03:00
Lampros Smyrnaios b579296ada - Code optimization and polishing. 
- Update dependencies.
 2023-08-28 16:11:26 +03:00
Lampros Smyrnaios dc97b323c9 - Show a warning, if the "numOfUnretrievedFiles" is over 50. 
- Delete gradle .zip file after installation.
- Code polishing.
 2023-08-04 15:33:48 +03:00
Lampros Smyrnaios 088cf73b30 - Update dependencies. 
- Code optimization and polishing.
 2023-07-27 17:46:17 +03:00
Lampros Smyrnaios 952bf7c035 - Update dependencies. 
- Code polishing.
 2023-07-06 13:22:09 +03:00
Lampros Smyrnaios 33df46f6f5 - Improve README. 
- Update and cleanup dependencies.
- Code polishing.
 2023-06-22 12:47:36 +03:00
Lampros Smyrnaios 9c897b8bf4 - Make use of the new Normalizer utilized by the PublicationRetriever plugin. 
- Code polishing.
 2023-06-10 02:40:45 +03:00
Lampros Smyrnaios 2aedae2367 - In case a serious error happened while processing the assignments, instead of shutting down immediately, now the Worker shuts down the executor service, registers that it will shut down soon and waits for the Controller to retrieve the already downloaded full-text files. 
- In case the full-texts' subdirectory could not be created, then terminate the "handleAssignment" method immediately. No posting of a faulty workerReport to the Controller should happen.
- Code polishing.
 2023-05-31 15:25:36 +03:00
Lampros Smyrnaios 4a95826f58 - Avoid processing the assignments, for which the assignments_full-texts subdirectory cannot be created. 
- Avoid a double-log.
 2023-05-31 02:27:24 +03:00
Lampros Smyrnaios 7f3ca80959 Bypass url-canonicalization for urls containing certain uncommon characters which cause the urls to get rejected. 2023-05-30 19:45:14 +03:00
Lampros Smyrnaios a9b1b20a51 - Prevent running out of space, by checking the available free space and stalling the acquisition of new assignments until more free space becomes available. 
- Fix missing change.
 2023-05-30 17:58:29 +03:00