38 Commits (3c9f8870d1573c0c6df495d63dfefb84439518fe)
 

Author SHA1 Message Date
Lampros Smyrnaios 3c9f8870d1 - Change the repository for the Impala JDBC Driver, as the previous one had networking issues. 5 months ago
Lampros Smyrnaios ff46839158 Fix not prioritizing the gradle version defined inside the "installAndRun.sh" script. 5 months ago
Lampros Smyrnaios 8d9336fa52 Update dependencies. 5 months ago
Lampros Smyrnaios ab99bc6168 - Make sure the temp table "current_assignment" from a cancelled previous execution, is dropped and purged on startup. 5 months ago
Lampros Smyrnaios 83f40a23d9 Bring back the prepared-statements for the insert-queries. After the fix of the "broken pipe"-error, they now work. Bringing them back, increases security and solves the "SQL syntax errors" caused by the values of some URLs. 6 months ago
Lampros Smyrnaios 2cf25b0d26 - Fix Impala "broken pipe" error, by closing the connection when not in need. The connection is reopened later with minimal overhead, as a connection pool is used. 6 months ago
Lampros Smyrnaios 82bf11b9b3 - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements. 6 months ago
Lampros Smyrnaios 33ba3e8d91 - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 6 months ago
Lampros Smyrnaios 0178e44574 - Increase security by sanitizing the value of the "workerId" before use it in sql-statements. Impala has bugs with some types of PreparedStatements. 7 months ago
Lampros Smyrnaios a46ab84f10 - Increase the lower and upper limits for the Java Heap Size. 7 months ago
Lampros Smyrnaios dea257b87f - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter". 7 months ago
Lampros Smyrnaios 15224c6468 Improve performance in the "getUrls"-endpoint, and more: 7 months ago
Lampros Smyrnaios 48eed20dd8 - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 7 months ago
Lampros Smyrnaios 780ed15ce2 - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction. 7 months ago
Lampros Smyrnaios d100af35d0 - Implement the "getUrls" and "addWorkerReport" endpoints with full database-handling. 8 months ago
Lampros Smyrnaios 0d47c33a08 - Improve logging configurations. 8 months ago
Lampros Smyrnaios 0540820817 -Update "installAndRun.sh": 9 months ago
Lampros Smyrnaios d931315ced - Add the "isControllerAlive"-endpoint. 9 months ago
Lampros Smyrnaios 983b900da7 - Add the "installAndRun.sh" script. 10 months ago
Lampros Smyrnaios d56e988518 - Process the Error of PDF-aggregation. Distinguish between "couldRetry" and "noRetry" cases. 11 months ago
Lampros Smyrnaios 25c566bf68 - Change server's port. 11 months ago
Lampros Smyrnaios 27375b9396 - Refactor the Assignment-creation. 12 months ago
Lampros Smyrnaios 5e7ccbd8c6 Add the "addWorkerReport" endpoint. 1 year ago
Lampros Smyrnaios 40763ec146 Update the "WorkerReport" request and the "UrlReport" and "Payload" models. 1 year ago
Lampros Smyrnaios c194af167f Allow handling of concurrent requests to the "getTestUrls"-endpoint. 1 year ago
Lampros Smyrnaios 308cab5ecd - Return an HTTP-500-error when the server cannot find the resourceFile requested by the "getTestUrls"-endpoint. 1 year ago
Lampros Smyrnaios 6729f51b03 Add an "assignmentId" field in the "Assignment"-class. 1 year ago
Lampros Smyrnaios 87044574b5 Update dependencies and add the "gradle-wrapper.properties" file which defines the gradle version. 1 year ago
Lampros Smyrnaios 787299b5b7 Add the "Datasource" inside the "Task" class and include it in the Assignment. 1 year ago
Lampros Smyrnaios d20fcf9cce - Update the "getUrls" and "getTestUrls" endpoints to take the data from the workers, in http-request-parameters instead of the http-body. 1 year ago
Lampros Smyrnaios e2cc320baf - Add the "getTestUrls"-endpoint which returns an "Assignment" with data retrieved from the added resource-file. 1 year ago
Lampros Smyrnaios d3588ea36b Add the "DownloadAttempt" class. 1 year ago
Lampros Smyrnaios a6ab810ad3 Update classes: "Publication" and "Payload". 1 year ago
Lampros Smyrnaios 85ecc4a36b Add classes: "AssignmentResponse", "WorkerReport", "WorkerRequest", "UrlReport". 1 year ago
Lampros Smyrnaios c2ea8a69de Update classes: "Assignment", "Task", "Error", "Payload", "UrlsRequest". 1 year ago
Lampros Smyrnaios 89c6a73a30 Add "Assignment", "Task" and "Error" classes. 1 year ago
Lampros Smyrnaios c6e12d3e95 - Update "addResults"-endpoint. 1 year ago
Lampros Smyrnaios 8a4376da9c Initial commit of UrlsController. 1 year ago