The Worker app of the PDF Aggregation Service.

Go to file

Lampros Smyrnaios 778dc6e25c - Improve the stability of "UriBuilder.getPublicIP()", by using a "HttpURLConnection" to increase the connection and read timeouts and avoid timeout-exceptions. - Show the number of assignments which are requested from the Controller, in the log-message. - Update Spring.		2023-01-03 18:43:26 +02:00
gradle/wrapper	- Set some optimization settings for gradle.	2022-11-30 16:25:57 +02:00
src	- Improve the stability of "UriBuilder.getPublicIP()", by using a "HttpURLConnection" to increase the connection and read timeouts and avoid timeout-exceptions.	2023-01-03 18:43:26 +02:00
.gitignore	- Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want.	2021-09-09 16:28:58 +03:00
README.md	- When the Worker is about to shut-down, after deleting all the handled assignments' files, check for remaining full-texts in the local storage and warn the user. If no remaining files were found, then delete the parent fulltexts' directory.	2022-11-02 02:27:04 +02:00
build.gradle	- Improve the stability of "UriBuilder.getPublicIP()", by using a "HttpURLConnection" to increase the connection and read timeouts and avoid timeout-exceptions.	2023-01-03 18:43:26 +02:00
gradle.properties	- Set some optimization settings for gradle.	2022-11-30 16:25:57 +02:00
installAndRun.sh	- Add an existence-check for the "publications_retriever"-JAR, before trying to make a backup, inside "installAndRun.sh".	2022-12-15 14:15:24 +02:00
settings.gradle	- Fix the project's name inside "settings.gradle".	2021-09-22 17:06:30 +03:00

README.md

UrlsWorker

The Worker's Application, requests assignments from the Controller and processes them, downloading the available full-texts.
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.
The Worker responds by compressing and sending the requested files, in each batch.

To install and run the application:

Run git clone and then cd UrlsWorker.
[Optional] Create the file inputData.txt , which contains just one line with the workerId, the maxAssignmentsLimitPerBatch, the maxAssignmentsBatchesToHandleBeforeRestart, the controller's base api-url and the shutdownOrCancelCode, all seperated by a comma "," .
For example: worker_1,1000,0,http://IP:PORT/api/,stopOrCancelCode.
The shutdownOrCancelCode is kind of an "auth-code", when receiving "shutdown" and "cancel-shutdown" requests.
Execute the installAndRun.sh script.
In case the above file (inputData.txt) does not exist, the script will request the required data from the user, and then it will create the inputData.txt file.

Notes:

If the "maxAssignmentsBatchesToHandleBeforeRestart" is zero or negative, then an infinite number of assignments-batches will be handled.
The above script, installs the PublicationsRetriever, as a library and then compiles and runs the whole Application.
If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1.