Commit Graph

6 Commits

Author SHA1 Message Date
Lampros Smyrnaios 5035094e44 - Move the "shutdownOrCancelCode" input in the "inputDataFile" provided by the user, for convenience and to be able to make this "auth-code" mandatory. Previously, it was optional and the app could not be made to stop in a normal-manner, if this code was not provided.
- Improve the instructions and the error-messages for the "inputDataFile".
2022-06-28 16:00:11 +03:00
Lampros Smyrnaios 26cbb83b51 - Add the "shutdownWorker"-endpoint to accept requests for shutting-down the Worker, gracefully, after it completes its current work (including sending the publications-files to the Controller). A user-defined "auth-code" is required.
- Add the "cancelShutdownWorker"-endpoint to cancel a previous "shutdownWorker"-request. A user-defined "auth-code" is required.
2022-06-22 18:53:27 +03:00
Lampros Smyrnaios 20b71164d5 - The worker will store the files in its local file-system and will send them to the controller in batches, after the latter requests them. When all files from a given assignments-num are sent, the files will be deleted from the Worker, in a scheduled-job.
- Implement the "getFullTexts"-endpoint, which returns the requested full-texts in a zip file.
- Implement the "getFullText"-endpoint, which returns the requested full-text.
- Implement the "getHandledAssignmentsCounts"-endpoint which returns the assignments-numbers, which were handled by that worker.
- Make sure each urlReport has the same "Date" for a given assignments-number. Also, make sure the "size" and "hash" have a "null" value, in case the full-text was not found.
- Check and log thread-pool shutdown errors.
- Add the stack-trace in the error-logs, instead of the Stderr.
- Update SpringBoot dependency.
- Change log levels.
- Code cleanup.
2021-11-26 17:04:31 +02:00
Lampros Smyrnaios 6cc2673fca - Add the ability to upload the files on an S3-ObjectStore.
- Change the server's port and the port of the controller-api.
- Update dependencies.
2021-07-29 09:01:53 +03:00
Lampros Smyrnaios f6e53ca289 Integrate the "PublicationsRetriever" program as a plugin, which downloads the full-texts of the publications. Afterwards, the retrieved data info is transferred to the Controller.
The "PublicationsRetriever" can be installed locally as a library, using the "installPublicationsRetriever.sh" script.
2021-06-22 05:58:07 +03:00
Lampros Smyrnaios 08eabe6f08 Initial commit of UrlsWorker. 2021-03-16 18:38:53 +02:00