michal.politowski
/
UrlsController
forked from lsmyrnaios/UrlsController
1
0
Fork 0
Code Pull Requests Activity
The Controller app of the PDF Aggregation Service.
33 Commits 3 Branches 14 Tags 1.1 MiB
Java 98.3%
Shell 1.6%
Go to file
Lampros Smyrnaios 2cf25b0d26 - Fix Impala "broken pipe" error, by closing the connection when not in need. The connection is reopened later with minimal overhead, as a connection pool is used. 
- Fix not closing the database-connection in case of a specific error (also in a commented error-case).
 		2022-01-13 00:47:15 +02:00
gradle/wrapper - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements. 2021-12-24 00:25:50 +02:00
scripts Initial commit of UrlsController. 2021-03-16 15:25:15 +02:00
src/main - Fix Impala "broken pipe" error, by closing the connection when not in need. The connection is reopened later with minimal overhead, as a connection pool is used. 2022-01-13 00:47:15 +02:00
README.md - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 2021-11-30 18:23:27 +02:00
build.gradle - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements. 2021-12-24 00:25:50 +02:00
installAndRun.sh - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements. 2021-12-24 00:25:50 +02:00
settings.gradle - Add the "isControllerAlive"-endpoint. 2021-09-23 15:08:52 +03:00

README.md

UrlsController

This is the Controller's Application.
It receives requests coming from the workers , constructs an assignments-list with data received from a database and returns the list to the workers.
Then it receives the "WorkerReports" and writes them into the database.
The database used is the Impala .
[...]

To install and run the application, run git clone. Then, provide a file "S3_minIO_credentials.txt", inside the working directory.
In the "S3_minIO_credentials.txt" file, you should provide the endpoint, the accessKey, the secretKey, the region and the bucket, in that order, separated by comma.
Afterwards, execute the installAndRun.sh script.
If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1.