The Controller app of the PDF Aggregation Service.
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Go to file
Lampros Smyrnaios dea257b87f - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter".
- Fix a bug, which caused multiple workers to get assigned the same batch-counter, while the assignment-tasks where different.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the error-message returned from the Worker, when a getFullTexts-request fails.
- Improve some log-messages.
- Update dependencies.
- Code cleanup.
2 years ago
gradle/wrapper - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter". 2 years ago
scripts Initial commit of UrlsController. 3 years ago
src/main - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter". 2 years ago
README.md - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 2 years ago
build.gradle - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter". 2 years ago
installAndRun.sh - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter". 2 years ago
settings.gradle - Add the "isControllerAlive"-endpoint. 3 years ago

README.md

UrlsController

This is the Controller's Application.
It receives requests coming from the workers , constructs an assignments-list with data received from a database and returns the list to the workers.
Then it receives the "WorkerReports" and writes them into the database.
The database used is the Impala .
[...]

To install and run the application, run git clone. Then, provide a file "S3_minIO_credentials.txt", inside the working directory.
In the "S3_minIO_credentials.txt" file, you should provide the endpoint, the accessKey, the secretKey, the region and the bucket, in that order, separated by comma.
Afterwards, execute the installAndRun.sh script.
If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1.