Lampros Smyrnaios
20b71164d5
- Implement the "getFullTexts"-endpoint, which returns the requested full-texts in a zip file. - Implement the "getFullText"-endpoint, which returns the requested full-text. - Implement the "getHandledAssignmentsCounts"-endpoint which returns the assignments-numbers, which were handled by that worker. - Make sure each urlReport has the same "Date" for a given assignments-number. Also, make sure the "size" and "hash" have a "null" value, in case the full-text was not found. - Check and log thread-pool shutdown errors. - Add the stack-trace in the error-logs, instead of the Stderr. - Update SpringBoot dependency. - Change log levels. - Code cleanup. |
||
---|---|---|
gradle/wrapper | ||
scripts | ||
src | ||
.gitignore | ||
README.md | ||
build.gradle | ||
installAndRun.sh | ||
settings.gradle |
README.md
UrlsWorker
This is the Worker's Application.
It requests assignments from the controller and processes them.
It posts the results to the controller, which in turn, puts them in a database.
To install and run the application:
- Run
git clone
and thencd UrlsWorker
. - Create the file
S3_minIO_credentials.txt
, which contains just one line with the S3_url, S3_username, S3_password, S3_server_region and the S3_bucket, separated by a comma,
. - [Optional] Create the file
inputData.txt
, which contains just one line with the workerId and the controller's base api-url, seperated by a comma,
. For example:worker_1,http://IP:PORT/api/
. - Execute the
installAndRun.sh
script. In case the above file (inputData.txt) does not exist, it will request the current worker's ID and the Controller's Url, and it will create the inputData.txt file.
That script, installs the PublicationsRetriever, as a library and then compiles and runs the whole Application.
If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1
.