Lampros Smyrnaios
b63ad87d00
- Fix a bug, where, in case it took too long to get the assignments from the Controller (possible when there are too many workers requesting at the same time or if the database is responding slowly), the Worker's scheduler would request for new assignments, in the meantime. - Fix a bug, where, if the "maxAssignmentsBatchesToHandleBeforeRestart" was set, the Worker's scheduler could request another batch, right before the Worker was about to shut down. - Fix a bug, where the condition of when to clear the over-sized data-structures was based on the "assignmentRequestCounter" send by the Controller (which is increased on each request by any worker and not for each individual one), and not on the "numHandledAssignmentsBatches" kept by each individual worker. This would result in much earlier cleanup, relative to the number of the Workers. |
||
|---|---|---|
| gradle/wrapper | ||
| src | ||
| .gitignore | ||
| README.md | ||
| build.gradle | ||
| installAndRun.sh | ||
| settings.gradle | ||
README.md
UrlsWorker
The Worker's Application, requests assignments from the Controller and processes them, downloading the available full-texts.
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.
The Worker responds by compressing and sending the requested files in each batch.
To install and run the application:
- Run
git cloneand thencd UrlsWorker. - [Optional] Create the file
inputData.txt, which contains just one line with the workerId, the maxAssignmentsLimitPerBatch, the maxAssignmentsBatchesToHandleBeforeRestart and the controller's base api-url, all seperated by a comma,. For example:worker_1,1000,0,http://IP:PORT/api/. - Execute the
installAndRun.shscript. In case the above file (inputData.txt) does not exist, it will request the required data from the user, and then it will create the inputData.txt file.
Notes:
- If the "maxAssignmentsBatchesToHandleBeforeRestart" is zero or negative, then an infinite number of assignments-batches will be handled.
- The above script, installs the PublicationsRetriever, as a library and then compiles and runs the whole Application.
- If you want to just run the app, then run the script with the argument "1":
./installAndRun.sh 1.