You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Lampros Smyrnaios 0032a8018f - Improve search-accuracy of "alreadyDownloaded" full-texts. 5 months ago
gradle/wrapper - Allow the user to set a maximum number of assignments-batches for the Worker to handle. After handling those batches, the Worker will shut down. A number of < 0 > indicates an infinite number of batches. 6 months ago
scripts Initial commit of UrlsWorker. 1 year ago
src - Improve search-accuracy of "alreadyDownloaded" full-texts. 5 months ago
.gitignore - Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want. 10 months ago
README.md - Make sure the handled assignments - full-texts are deleted before the application exits. 6 months ago
build.gradle - Allow the user to set a maximum number of assignments-batches for the Worker to handle. After handling those batches, the Worker will shut down. A number of < 0 > indicates an infinite number of batches. 6 months ago
installAndRun.sh - Make sure the handled assignments - full-texts are deleted before the application exits. 6 months ago
settings.gradle - Fix the project's name inside "settings.gradle". 9 months ago

README.md

UrlsWorker

This is the Worker's Application.
It requests assignments from the controller and processes them.
It posts the results to the controller, which in turn, puts them in a database.

To install and run the application:

  • Run git clone and then cd UrlsWorker.
  • Create the file S3_minIO_credentials.txt , which contains just one line with the S3_url, S3_username, S3_password, S3_server_region and the S3_bucket, all separated by a comma ,.
  • [Optional] Create the file inputData.txt , which contains just one line with the workerId, the maxAssignmentsLimitPerBatch, the maxAssignmentsBatchesToHandleBeforeRestart and the controller's base api-url, all seperated by a comma , . For example: worker_1,http://IP:PORT/api/.
  • Execute the installAndRun.sh script. In case the above file (inputData.txt) does not exist, it will request the current worker's ID, the maxAssignmentsLimitPerBatch, the maxAssignmentsBatchesToHandleBeforeRestart and the Controller's Url, and it will create the inputData.txt file.

Note: If the "maxAssignmentsBatchesToHandleBeforeRestart" is zero or negative, then an infinite number of assignments-batches will be handled. That script, installs the PublicationsRetriever, as a library and then compiles and runs the whole Application.
If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1.