lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 11 Wiki Activity
The Worker app of the PDF Aggregation Service.
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
163 Commits
1 Branch
11 Tags
1.3 MiB
Java 96.9%
Shell 3.1%
 
 
master
2.1.9
2.1.8
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1
2.0
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
Go to file
Lampros Smyrnaios 34407179fc Update Gradle in the install script. 16 hours ago
gradle/wrapper Update Gradle. 16 hours ago
src Add a missing change in logback-spring.xml. 3 months ago
.gitignore - Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want. 3 years ago
LICENSE - Automatically use the latest version of "publications_retriever" software from the Nexus maven-repository. 3 months ago
README.md - Update README. 18 hours ago
build.gradle - Update README. 18 hours ago
createSwapStorage.sh - Automatically use the latest version of "publications_retriever" software from the Nexus maven-repository. 3 months ago
gradle.properties - Set some optimization settings for gradle. 1 year ago
gradlew Add some gradle files to be used by Jenkins. 3 months ago
gradlew.bat Add some gradle files to be used by Jenkins. 3 months ago
installAndRun.sh Update Gradle in the install script. 16 hours ago
settings.gradle - Fix the project's name inside "settings.gradle". 3 years ago

README.md

UrlsWorker

Jenkins build status

The Worker's Application, requests assignments from the Controller and processes them with the help of the PublicationsRetriever software and downloads the available full-texts.
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.
The Worker responds by compressing and sending the requested files, in each batch.

Multiple instances of this app are deployed on the cloud.
We use Facebook's Zstandard compression algorithm, which brings very big benefits in compression rate and speed.


To install and run the application:

  • Run git clone and then cd UrlsWorker.
  • Set the preferable values inside the application.properties file.
  • Execute the installAndRun.sh script.

Notes:

  • If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1. In this scenario, the SpringBoot-app will not be re-built.