lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 12 Wiki Activity
The Worker app of the PDF Aggregation Service.
147 Commits 1 Branch 12 Tags 1.3 MiB
Java 96.9%
Shell 3.1%
Go to file
Lampros Smyrnaios 10e39d79a4 - Improve a log-message. 
- Update dependencies.
 		2023-10-20 17:35:39 +03:00
gradle/wrapper - In case a faulty "assignmentsCounter" was given to the "addReportResultToWorker"-endpoint, then return an explanatory error-message along with the HTTP-404 error. 2023-10-06 15:45:53 +03:00
src - Improve a log-message. 2023-10-20 17:35:39 +03:00
.gitignore - Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want. 2021-09-09 16:28:58 +03:00
LICENSE - Add LICENSE. 2023-09-14 16:09:20 +03:00
README.md - Improve README. 2023-06-22 12:47:36 +03:00
build.gradle - Improve a log-message. 2023-10-20 17:35:39 +03:00
createSwapStorage.sh Leave the Max-Heap-Size to 8Gb, we assume that enough swap space will be available on the host. 2023-04-29 17:55:03 +03:00
gradle.properties - Set some optimization settings for gradle. 2022-11-30 16:25:57 +02:00
installAndRun.sh - In case a faulty "assignmentsCounter" was given to the "addReportResultToWorker"-endpoint, then return an explanatory error-message along with the HTTP-404 error. 2023-10-06 15:45:53 +03:00
settings.gradle - Fix the project's name inside "settings.gradle". 2021-09-22 17:06:30 +03:00

README.md

UrlsWorker

The Worker's Application, requests assignments from the Controller and processes them with the help of the PublicationsRetriever software and downloads the available full-texts.
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.
The Worker responds by compressing and sending the requested files, in each batch.

Multiple instances of this app are deployed on the cloud.
We use Facebook's Zstandard compression algorithm, which brings very big benefits in compression rate and speed.


To install and run the application:

  • Run git clone and then cd UrlsWorker.
  • Set the preferable values inside the application.properties file.
  • Execute the installAndRun.sh script.

Notes:

  • The above script, installs the PublicationsRetriever software, as a library and then compiles and runs the whole Application.
  • If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1. In this scenario, apart from the SpringBoot-app not re-building, the PublicationsRetriever software, will not be re-installed as well.
  • If you want to avoid re-installing the PublicationsRetriever software, but still building the Worker's application, run the script with the argument "0", followed by the argument "1": ./installAndRun.sh 0 1, i.e. when using a development (non-published) version or when nothing has changed and wanting to avoid the time-overhead.