Lampros Smyrnaios
9cb43b3d94
- Improve startup speed, by using a faster remote server to get the host's machine public IP. This also reduces the risk of not being able to get the public IP at all.
...
- Set the App to gracefully shut down the WebServer and wait up to 2 minutes.
- Increase the waiting time for the "PublicationsRetriever.executor" to shut down, to 2 minutes.
2023-05-23 20:17:58 +03:00
Lampros Smyrnaios
cc55354e73
Show the worker-id when the worker starts.
2023-05-15 13:22:55 +03:00
Lampros Smyrnaios
29a54f0b30
Remove the "shutDownOrCancelCode" from security checks, since we have an IP whitelisting mechanism in place.
2023-05-03 15:15:46 +03:00
Lampros Smyrnaios
4eac7c5c66
Fix typo in property's name.
2023-04-29 18:10:35 +03:00
Lampros Smyrnaios
53ab51922a
Allow shutdown requests from the Controller.
2023-04-28 23:46:39 +03:00
Lampros Smyrnaios
b98ea92dec
Update/improve documentation.
2023-01-27 14:27:57 +02:00
Lampros Smyrnaios
24b52fba63
- Refactor the initialization and configuration process and Spring-ify the project.
...
- Update Spring dependency.
2023-01-25 18:33:49 +02:00
Lampros Smyrnaios
90a69686cf
- When the Worker is about to shut-down, after deleting all the handled assignments' files, check for remaining full-texts in the local storage and warn the user. If no remaining files were found, then delete the parent fulltexts' directory.
...
- Polish the code.
2022-11-02 02:27:04 +02:00
Lampros Smyrnaios
5035094e44
- Move the "shutdownOrCancelCode" input in the "inputDataFile" provided by the user, for convenience and to be able to make this "auth-code" mandatory. Previously, it was optional and the app could not be made to stop in a normal-manner, if this code was not provided.
...
- Improve the instructions and the error-messages for the "inputDataFile".
2022-06-28 16:00:11 +03:00
Lampros Smyrnaios
26cbb83b51
- Add the "shutdownWorker"-endpoint to accept requests for shutting-down the Worker, gracefully, after it completes its current work (including sending the publications-files to the Controller). A user-defined "auth-code" is required.
...
- Add the "cancelShutdownWorker"-endpoint to cancel a previous "shutdownWorker"-request. A user-defined "auth-code" is required.
2022-06-22 18:53:27 +03:00
Lampros Smyrnaios
20b71164d5
- The worker will store the files in its local file-system and will send them to the controller in batches, after the latter requests them. When all files from a given assignments-num are sent, the files will be deleted from the Worker, in a scheduled-job.
...
- Implement the "getFullTexts"-endpoint, which returns the requested full-texts in a zip file.
- Implement the "getFullText"-endpoint, which returns the requested full-text.
- Implement the "getHandledAssignmentsCounts"-endpoint which returns the assignments-numbers, which were handled by that worker.
- Make sure each urlReport has the same "Date" for a given assignments-number. Also, make sure the "size" and "hash" have a "null" value, in case the full-text was not found.
- Check and log thread-pool shutdown errors.
- Add the stack-trace in the error-logs, instead of the Stderr.
- Update SpringBoot dependency.
- Change log levels.
- Code cleanup.
2021-11-26 17:04:31 +02:00
Lampros Smyrnaios
6cc2673fca
- Add the ability to upload the files on an S3-ObjectStore.
...
- Change the server's port and the port of the controller-api.
- Update dependencies.
2021-07-29 09:01:53 +03:00
Lampros Smyrnaios
f6e53ca289
Integrate the "PublicationsRetriever" program as a plugin, which downloads the full-texts of the publications. Afterwards, the retrieved data info is transferred to the Controller.
...
The "PublicationsRetriever" can be installed locally as a library, using the "installPublicationsRetriever.sh" script.
2021-06-22 05:58:07 +03:00
Lampros Smyrnaios
08eabe6f08
Initial commit of UrlsWorker.
2021-03-16 18:38:53 +02:00
Lampros Smyrnaios