You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
LSmyrnaios 045788c728 - Use the "Timestamp" data-type instead of the "Date", in order to include more information. 2 days ago
gradle/wrapper - Integrate the latest changes from the "PublicationsRetriever"-plugin. 3 months ago
scripts Initial commit of UrlsWorker. 9 months ago
src - Use the "Timestamp" data-type instead of the "Date", in order to include more information. 2 days ago
.gitignore - Update the "" script to be able to just run the app (without re-installing), if you want. 3 months ago - Update the "": 2 months ago
build.gradle - The worker will store the files in its local file-system and will send them to the controller in batches, after the latter requests them. When all files from a given assignments-num are sent, the files will be deleted from the Worker, in a scheduled-job. 2 days ago - Add an HTTP-error-handler in "AssignmentHandler.requestAssignments()". 2 months ago
settings.gradle - Fix the project's name inside "settings.gradle". 2 months ago


This is the Worker's Application.
It requests assignments from the controller and processes them.
It posts the results to the controller, which in turn, puts them in a database.

To install and run the application:

  • Run git clone and then cd UrlsWorker.
  • Create the file S3_minIO_credentials.txt , which contains just one line with the S3_url, S3_username, S3_password, S3_server_region and the S3_bucket, separated by a comma ,.
  • [Optional] Create the file inputData.txt , which contains just one line with the workerId and the controller's base api-url, seperated by a comma , . For example: worker_1,http://IP:PORT/api/.
  • Execute the script. In case the above file (inputData.txt) does not exist, it will request the current worker's ID and the Controller's Url, and it will create the inputData.txt file.

That script, installs the PublicationsRetriever, as a library and then compiles and runs the whole Application.
If you want to just run the app, then run the script with the argument "1": ./ 1.