lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 11 Wiki Activity
The Worker app of the PDF Aggregation Service.
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
66 Commits
1 Branch
11 Tags
1.3 MiB
Java 96.9%
Shell 3.1%
 
 
master
2.1.9
2.1.8
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1
2.0
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'd91732bc16'
${ noResults }
Go to file
Lampros Smyrnaios d91732bc16 - Add deletion, of the cookies in the newly-supported CookieManager, after each batch. 
- Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated.
- Update dependencies.
- Code cleanup.
 		2 years ago
gradle/wrapper - Update the Worker's report to include the datasourceID for each record. It is used by the Controller inside the S3-fileNames. 2 years ago
src - Add deletion, of the cookies in the newly-supported CookieManager, after each batch. 2 years ago
.gitignore - Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want. 3 years ago
README.md Update the README.md 2 years ago
build.gradle - Add deletion, of the cookies in the newly-supported CookieManager, after each batch. 2 years ago
installAndRun.sh - Update the Worker's report to include the datasourceID for each record. It is used by the Controller inside the S3-fileNames. 2 years ago
settings.gradle - Fix the project's name inside "settings.gradle". 3 years ago

README.md

UrlsWorker

The Worker's Application, requests assignments from the Controller and processes them, downloading the available full-texts.
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.
The Worker responds by compressing and sending the requested files in each batch.

To install and run the application:

  • Run git clone and then cd UrlsWorker.
  • [Optional] Create the file inputData.txt , which contains just one line with the workerId, the maxAssignmentsLimitPerBatch, the maxAssignmentsBatchesToHandleBeforeRestart and the controller's base api-url, all seperated by a comma , . For example: worker_1,1000,0,http://IP:PORT/api/.
  • Execute the installAndRun.sh script. In case the above file (inputData.txt) does not exist, it will request the required data from the user, and then it will create the inputData.txt file.

Notes:

  • If the "maxAssignmentsBatchesToHandleBeforeRestart" is zero or negative, then an infinite number of assignments-batches will be handled.
  • The above script, installs the PublicationsRetriever, as a library and then compiles and runs the whole Application.
  • If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1.