UrlsWorker/README.md

# UrlsWorker

The Worker's Application, requests assignments from the [**Controller**](https://code-repo.d4science.org/lsmyrnaios/UrlsController) and processes them with the help of the [__PublicationsRetriever__](https://github.com/LSmyrnaios/PublicationsRetriever) software and downloads the available full-texts.<br>
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.<br>
The Worker responds by compressing and sending the requested files, in each batch.<br>
<br>
Multiple instances of this app are deployed on the cloud.<br>
We use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits in compression rate and speed.  
<br>
<br>

**To install and run the application**:
- Run ```git clone``` and then ```cd UrlsWorker```.
- Set the preferable values inside the [__application.properties__](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker/src/branch/master/src/main/resources/application.properties) file.
- Execute the ```installAndRun.sh``` script.<br>
<br>

**Notes**:
- The above script, installs the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, as a library and then compiles and runs the whole Application.<br>
- If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```. In this scenario, apart from the SpringBoot-app not re-building,
the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, will not be re-installed as well.<br>
- If you want to avoid re-installing the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, but still building the Worker's application, run the script with the argument "0", followed by the argument "1": ```./installAndRun.sh 0 1```, i.e. when using a development (non-published) version
or when nothing has changed and wanting to avoid the time-overhead.<br>
<br>
- Read the Controller's url from a file, when starting the Application. - Switch the "AssignmentsHandler.askForTest" to "false". - Get the size and the hash of a docFile which is previously downloaded by another ID in that batch. - Reset the "AssignmentHandler.urlReports" list after posting the results to the Controller. - Enhance logging and comments. - Add more guidelines in the README. - Disable the scheduled test-live job. - Code cleanup. 2021-09-21 15:21:39 +02:00			`# UrlsWorker`
- Add README.md - Avoid sending a null-Error-object, which increases the complexity on the controller's side. Instead, send an Error object with "null" members. 2021-08-05 19:41:32 +02:00
- Improve README. - Update and cleanup dependencies. - Code polishing. 2023-06-22 11:47:36 +02:00			`The Worker's Application, requests assignments from the [Controller](https://code-repo.d4science.org/lsmyrnaios/UrlsController) and processes them with the help of the [__PublicationsRetriever__](https://github.com/LSmyrnaios/PublicationsRetriever) software and downloads the available full-texts.<br>`
Update the README.md 2022-02-07 19:59:10 +01:00			`Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.<br>`
- When the Worker is about to shut-down, after deleting all the handled assignments' files, check for remaining full-texts in the local storage and warn the user. If no remaining files were found, then delete the parent fulltexts' directory. - Polish the code. 2022-11-02 01:27:04 +01:00			`The Worker responds by compressing and sending the requested files, in each batch.<br>`
- Read the Controller's url from a file, when starting the Application. - Switch the "AssignmentsHandler.askForTest" to "false". - Get the size and the hash of a docFile which is previously downloaded by another ID in that batch. - Reset the "AssignmentHandler.urlReports" list after posting the results to the Controller. - Enhance logging and comments. - Add more guidelines in the README. - Disable the scheduled test-live job. - Code cleanup. 2021-09-21 15:21:39 +02:00			`<br>`
- Improve README. - Update and cleanup dependencies. - Code polishing. 2023-06-22 11:47:36 +02:00			`Multiple instances of this app are deployed on the cloud.<br>`
- Add a new endpoint "getFullTextsImproved" which uses Facebook's [Zstandard](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits on compression rate and speed. - Remove some dependencies. 2023-01-09 14:48:30 +01:00			`We use Facebook's [Zstandard](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits in compression rate and speed.`
			`<br>`
Update/improve documentation. 2023-01-27 13:27:57 +01:00			`<br>`

			`To install and run the application:`
- Read the Controller's url from a file, when starting the Application. - Switch the "AssignmentsHandler.askForTest" to "false". - Get the size and the hash of a docFile which is previously downloaded by another ID in that batch. - Reset the "AssignmentHandler.urlReports" list after posting the results to the Controller. - Enhance logging and comments. - Add more guidelines in the README. - Disable the scheduled test-live job. - Code cleanup. 2021-09-21 15:21:39 +02:00			- Run ```git clone``` and then ```cd UrlsWorker```.
Update/improve documentation. 2023-01-27 13:27:57 +01:00			`- Set the preferable values inside the [__application.properties__](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker/src/branch/master/src/main/resources/application.properties) file.`
- Move the "shutdownOrCancelCode" input in the "inputDataFile" provided by the user, for convenience and to be able to make this "auth-code" mandatory. Previously, it was optional and the app could not be made to stop in a normal-manner, if this code was not provided. - Improve the instructions and the error-messages for the "inputDataFile". 2022-06-28 15:00:11 +02:00			- Execute the ```installAndRun.sh``` script.<br>
Update the README.md 2022-02-07 19:59:10 +01:00			`<br>`
- Add README.md - Avoid sending a null-Error-object, which increases the complexity on the controller's side. Instead, send an Error object with "null" members. 2021-08-05 19:41:32 +02:00
Update/improve documentation. 2023-01-27 13:27:57 +01:00			`Notes:`
			`- The above script, installs the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, as a library and then compiles and runs the whole Application.<br>`
			- If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```. In this scenario, apart from the SpringBoot-app not re-building,
			`the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, will not be re-installed as well.<br>`
- Increase the "max_heap_size". - Update a dependency. - Update README.md 2023-04-28 18:37:12 +02:00			- If you want to avoid re-installing the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, but still building the Worker's application, run the script with the argument "0", followed by the argument "1": ```./installAndRun.sh 0 1```, i.e. when using a development (non-published) version
			`or when nothing has changed and wanting to avoid the time-overhead.<br>`
- Add README.md - Avoid sending a null-Error-object, which increases the complexity on the controller's side. Instead, send an Error object with "null" members. 2021-08-05 19:41:32 +02:00			`<br>`