22 lines
1.4 KiB
Markdown
22 lines
1.4 KiB
Markdown
# UrlsWorker
|
|
### [](https://jenkins-dnet.d4science.org/job/UrlsWorker/)
|
|
|
|
The Worker's Application, requests assignments from the [**Controller**](https://code-repo.d4science.org/lsmyrnaios/UrlsController) and processes them with the help of the [__PublicationsRetriever__](https://github.com/LSmyrnaios/PublicationsRetriever) software and downloads the available full-texts.<br>
|
|
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.<br>
|
|
The Worker responds by compressing and sending the requested files, in each batch.<br>
|
|
<br>
|
|
Multiple instances of this app are deployed on the cloud.<br>
|
|
We use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits in compression rate and speed.
|
|
<br>
|
|
<br>
|
|
|
|
**To install and run the application**:
|
|
- Run ```git clone``` and then ```cd UrlsWorker```.
|
|
- Set the preferable values inside the [__application.properties__](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker/src/branch/master/src/main/resources/application.properties) file.
|
|
- Execute the ```installAndRun.sh``` script.<br>
|
|
<br>
|
|
|
|
**Notes**:
|
|
- If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```. In this scenario, the SpringBoot-app will not be re-built.<br>
|
|
<br>
|