Lampros Smyrnaios
839a797124
- Optimize directories-creation process by eliminating the additive check for existence, as that check already takes place inside the "mkdirs()" method. - Remove the obsolete code which in case the specific assignments' subdirectory failed to be created, then a different base-dir was used instead. Since the user-defined baseDir is already been successfully created upon initialization, any problem on creating subdirectories inside that base-directory will most likely persist even when changing the base directory. Additionally, even if the subdirectory with the changed base-directory succeeded, the "FullTextsController.getFullTexts()" method would not use it, resulting in errors. - Code polishing. |
||
|---|---|---|
| gradle/wrapper | ||
| src | ||
| .gitignore | ||
| README.md | ||
| build.gradle | ||
| gradle.properties | ||
| installAndRun.sh | ||
| settings.gradle | ||
README.md
UrlsWorker
The Worker's Application, requests assignments from the Controller and processes them with the help of the PublicationsRetriever software and downloads the available full-texts.
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.
The Worker responds by compressing and sending the requested files, in each batch.
We use Facebook's Zstandard compression algorithm, which brings very big benefits in compression rate and speed.
To install and run the application:
- Run
git cloneand thencd UrlsWorker. - Set the preferable values inside the application.properties file.
- Execute the
installAndRun.shscript.
Notes:
- The above script, installs the PublicationsRetriever software, as a library and then compiles and runs the whole Application.
- If you want to just run the app, then run the script with the argument "1":
./installAndRun.sh 1. In this scenario, apart from the SpringBoot-app not re-building, the PublicationsRetriever software, will not be re-installed as well. - If you want to avoid re-installing the PublicationsRetriever software, i.e. when using a development (non-published) version
or when nothing has changed and wanting to avoid the time-overhead, run the script with the argument "0", followed by the argument "1":
./installAndRun.sh 0 1.