lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 30 Wiki Activity
The Worker app of the PDF Aggregation Service.
248 Commits 2 Branches 30 Tags 694 KiB
Java 97.4%
Shell 2.6%
Go to file
Lampros Smyrnaios 500101f305 Update dependencies. 2025-12-11 16:14:45 +02:00
gradle/wrapper - Release version 3.3.1 2025-11-20 18:20:43 +02:00
src - Add steps into fixing an issue causing the archives' stream to be truncated. Now the "TAOS"-stream is finalized before the underlying compression stream is closed. 2025-12-11 15:49:53 +02:00
.gitignore - Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want. 2021-09-09 16:28:58 +03:00
LICENSE - Set increased timeout for async requests. 2025-06-05 13:28:23 +03:00
README.md - Set next version. 2025-11-06 15:40:56 +02:00
build.gradle Update dependencies. 2025-12-11 16:14:45 +02:00
createSwapStorage.sh - Automatically use the latest version of "publications_retriever" software from the Nexus maven-repository. 2024-02-08 18:33:18 +02:00
gradle.properties - Set next version. 2024-12-04 00:26:06 +02:00
gradlew - Upgrade to Java 25 and use the more efficient Virtual Threads. 2025-10-23 17:33:54 +03:00
gradlew.bat - Upgrade to Java 25 and use the more efficient Virtual Threads. 2025-10-23 17:33:54 +03:00
installAndRun.sh Update dependencies. 2025-12-11 16:14:45 +02:00
settings.gradle - Fix the project's name inside "settings.gradle". 2021-09-22 17:06:30 +03:00

README.md

UrlsWorker

Jenkins build status

The Worker's Application, requests assignments from the Controller and processes them with the help of the PublicationsRetriever software and downloads the available full-texts or html-pages.
Then, it posts the results to the Controller, which in turn, requests from the Worker, the files which are not already found by other workers, in batches.
The Worker responds by compressing and sending the requested files, in each batch.

Multiple instances of this app are deployed on cloud VMs.
We use Facebook's Zstandard compression algorithm, which brings very big benefits in compression rate and speed and as a result, reduces cloud costs.

Requirements and installation

Requirements

  • Java 25
  • Gradle 9.2.0

Installation and execution

  • Run git clone and then cd UrlsWorker.
  • Set the preferable values inside the application.properties file. Mainly the "workerId" and "controllerIp" properties.
  • Execute the installAndRun.sh script.

Notes:

  • If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1. In this scenario, the SpringBoot-app will not be re-built.