lsmyrnaios
/
UrlsController
1
0
Fork 2
Code Issues Pull Requests Releases 18 Wiki Activity
The Controller app of the PDF Aggregation Service.
31 Commits 2 Branches 18 Tags 1.3 MiB
Java 98.3%
Shell 1.6%
Go to file
Lampros Smyrnaios 33ba3e8d91 - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 
- Fix not updating the fileLocation with the s3Url for records which share the same full-text.
- Set only one delete-order for each assignments-batch-files, not one (or more, by mistake) per zip-batch.
- Set the HttpStatus to "204 - NO_CONTENT", when no assignments are available to be returned to the Worker.
- Fix not unlocking the "dataBaseLock" in case of a "dataBase-connection"-error, in "addWorkerReport()".
- Improve some log-messages.
- Change the log-level for the "S3-bucket already exists" message.
- Update Gradle.
- Optimize imports.
- Code cleanup.
 		2021-12-21 15:55:27 +02:00
gradle/wrapper - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 2021-12-21 15:55:27 +02:00
scripts Initial commit of UrlsController. 2021-03-16 15:25:15 +02:00
src/main - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 2021-12-21 15:55:27 +02:00
README.md - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 2021-11-30 18:23:27 +02:00
build.gradle - Increase the lower and upper limits for the Java Heap Size. 2021-12-06 20:27:39 +02:00
installAndRun.sh - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 2021-12-21 15:55:27 +02:00
settings.gradle - Add the "isControllerAlive"-endpoint. 2021-09-23 15:08:52 +03:00

README.md

UrlsController

This is the Controller's Application.
It receives requests coming from the workers , constructs an assignments-list with data received from a database and returns the list to the workers.
Then it receives the "WorkerReports" and writes them into the database.
The database used is the Impala .
[...]

To install and run the application, run git clone. Then, provide a file "S3_minIO_credentials.txt", inside the working directory.
In the "S3_minIO_credentials.txt" file, you should provide the endpoint, the accessKey, the secretKey, the region and the bucket, in that order, separated by comma.
Afterwards, execute the installAndRun.sh script.
If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1.