Commit Graph

201 Commits

Author SHA1 Message Date
Antonis Lempesis f5748434c7 Merge branch 'master' of https://code-repo.d4science.org/antonis.lempesis/UrlsController 2022-01-31 14:01:39 +02:00
Antonis Lempesis 9ac10fc4b3 fixed Value annotations 2022-01-31 14:01:26 +02:00
Antonis Lempesis 0772e9cdfb Merge branch 'master' of https://code-repo.d4science.org/antonis.lempesis/UrlsController 2022-01-31 13:49:34 +02:00
Antonis Lempesis 1c82088a7c fixed Value annotations 2022-01-31 13:49:14 +02:00
Antonis Lempesis 3da6fd98e9 added Dockerfile 2022-01-31 04:21:31 +02:00
Antonis Lempesis 6dde8c0faa finished merge 2022-01-31 04:17:16 +02:00
Antonis Lempesis e47fd8d97b merged refactor branch 2022-01-30 23:10:06 +02:00
Antonis Lempesis 3741cce886 springified project 2022-01-30 22:15:13 +02:00
Antonis Lempesis bf26bf955f springified project 2022-01-30 22:14:52 +02:00
Lampros Smyrnaios d0ab42e4fa - Change the scheme of the file-location URI.
- Move the old and the current database names in the "application.properties" file.
- Improve logging.
2022-01-28 07:24:42 +02:00
Lampros Smyrnaios 92b11baf93 - Update the repository for the Impala JDBC Driver.
- Code cleanup.
2022-01-28 00:59:19 +02:00
Antonis Lempesis 91f460ce51 moved impala jar to omtd repository and updated build file 2022-01-28 00:41:29 +02:00
Lampros Smyrnaios a01e11eef0 When all the data is processed, increase the number of "max-attempts" to retry some very old records, in the next requests. 2022-01-27 01:18:26 +02:00
Lampros Smyrnaios 3c9f8870d1 - Change the repository for the Impala JDBC Driver, as the previous one had networking issues.
- Optimize the "findAssignmentsQuery".
2022-01-26 19:52:46 +02:00
Lampros Smyrnaios ff46839158 Fix not prioritizing the gradle version defined inside the "installAndRun.sh" script. 2022-01-21 15:45:12 +02:00
Lampros Smyrnaios 8d9336fa52 Update dependencies. 2022-01-21 15:04:29 +02:00
Lampros Smyrnaios ab99bc6168 - Make sure the temp table "current_assignment" from a cancelled previous execution, is dropped and purged on startup.
- Improve logging.
- Code cleanup.
2022-01-19 01:37:47 +02:00
Lampros Smyrnaios 83f40a23d9 Bring back the prepared-statements for the insert-queries. After the fix of the "broken pipe"-error, they now work. Bringing them back, increases security and solves the "SQL syntax errors" caused by the values of some URLs. 2022-01-13 00:54:21 +02:00
Lampros Smyrnaios 2cf25b0d26 - Fix Impala "broken pipe" error, by closing the connection when not in need. The connection is reopened later with minimal overhead, as a connection pool is used.
- Fix not closing the database-connection in case of a specific error (also in a commented error-case).
2022-01-13 00:47:15 +02:00
Lampros Smyrnaios 82bf11b9b3 - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements.
- Update dependencies.
2021-12-24 00:25:50 +02:00
Lampros Smyrnaios 33ba3e8d91 - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches.
- Fix not updating the fileLocation with the s3Url for records which share the same full-text.
- Set only one delete-order for each assignments-batch-files, not one (or more, by mistake) per zip-batch.
- Set the HttpStatus to "204 - NO_CONTENT", when no assignments are available to be returned to the Worker.
- Fix not unlocking the "dataBaseLock" in case of a "dataBase-connection"-error, in "addWorkerReport()".
- Improve some log-messages.
- Change the log-level for the "S3-bucket already exists" message.
- Update Gradle.
- Optimize imports.
- Code cleanup.
2021-12-21 15:55:27 +02:00
Lampros Smyrnaios 0178e44574 - Increase security by sanitizing the value of the "workerId" before use it in sql-statements. Impala has bugs with some types of PreparedStatements.
- Improve reliability, by dropping the "current_assignment" table in case of an error, thus the next "getUrls"-request will not fail.
- Fix the "databaseLock" not being unlocked when the "addWorkerReport()" method returned early on some error-cases.
- Delete the "assignment"-data after inserting the related payloads and attempts in the database.
2021-12-10 21:47:58 +02:00
Lampros Smyrnaios a46ab84f10 - Increase the lower and upper limits for the Java Heap Size.
- Update the "ServerBaseURL" to the Public IP Address of the machine which is running the app.
2021-12-06 20:27:39 +02:00
Lampros Smyrnaios dea257b87f - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter".
- Fix a bug, which caused multiple workers to get assigned the same batch-counter, while the assignment-tasks where different.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the error-message returned from the Worker, when a getFullTexts-request fails.
- Improve some log-messages.
- Update dependencies.
- Code cleanup.
2021-12-06 20:18:30 +02:00
Lampros Smyrnaios 15224c6468 Improve performance in the "getUrls"-endpoint, and more:
- Optimize the "findAssignmentsQuery" by using an inner limit (larger than the outer).
- Save a ton of time from inserting the assignments into the database, by using a temporal table to hold the new assignments, in order for them to be easily accessible both from the Controller (which processes them and send them to the Worker) and the database itself, in order to "import" them into the "assignment"-table.
- Replace the "Date" with "Timestamp", in order to hold more detailed information.
- Code cleanup.
2021-11-30 19:59:46 +02:00
Lampros Smyrnaios 48eed20dd8 - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files.
- Move the "mergeParquetFiles" and "getCutBatchExceptionMessage" methods inside the "FileUtils" class.
- Code cleanup.
2021-11-30 18:23:27 +02:00
Lampros Smyrnaios 780ed15ce2 - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction.
- Fix the "null" representation of an "unknown" payload-size in the database.
- Remove the obsolete thread-locking for the "CreateDatabase" operation. This code is guaranteed to run BEFORE any other operation in the database.
- Implement the "handlePreparedStatementException" and "closeConnection" methods.
- Improve error-logs.
- Update dependencies.
- Code cleanup.
2021-11-30 13:26:19 +02:00
Lampros Smyrnaios d100af35d0 - Implement the "getUrls" and "addWorkerReport" endpoints with full database-handling.
- Add connectivity with an Impala-database and create a dedicated Controller for future statistics-requests.
- Optimize the "getTestUrls"-endpoint.
- Disable the "reportCurrentTime()" scheduled-task.
- Update dependencies and bump project's version to '1.0.0-SNAPSHOT'.
- Set the logging-appender to "File".
- Code cleanup.
2021-11-09 23:59:27 +02:00
Lampros Smyrnaios 0d47c33a08 - Improve logging configurations.
- Fix the "AssignmentResponse.toString()" method.
- Update SpringBoot to v.2.5.6
- Code cleanup.
2021-11-04 11:57:19 +02:00
Lampros Smyrnaios 0540820817 -Update "installAndRun.sh":
-- Check if a gradle installation with the given version already exists, before downloading and installing gradle.
-- Make sure the "unzip" package is installed, before trying unzipping the gradle package.
- Update the logs' fileName.
2021-10-14 02:46:33 +03:00
Lampros Smyrnaios d931315ced - Add the "isControllerAlive"-endpoint.
- Change the data-type of the "UrlReport.status" to be "enum StatusType", in order to increase consistency and comparability.
- Change the "Date" datatype in "Payload" to have the SQL's version.
- Fix the project's name inside "settings.gradle".
- Code cleanup.
2021-09-23 15:08:52 +03:00
Lampros Smyrnaios 983b900da7 - Add the "installAndRun.sh" script.
- Update the README.
- Update the dependencies.
2021-09-09 15:56:37 +03:00
Lampros Smyrnaios d56e988518 - Process the Error of PDF-aggregation. Distinguish between "couldRetry" and "noRetry" cases.
- Update the "RequestParam" for the getUrls endpoints.
- Fix the "assignmentCounter".
- Code cleanup.
2021-08-05 15:43:37 +03:00
Lampros Smyrnaios 25c566bf68 - Change server's port.
- Update dependencies.
2021-07-29 08:44:36 +03:00
Lampros Smyrnaios 27375b9396 - Refactor the Assignment-creation.
In order to match the database, now we have a list of Assignments sent through the AssignmentResponse, instead of a single Assignment having a list of tasks.
- Cleanup the members of the "Payload" model (also prepare for database integration).
2021-07-05 14:04:39 +03:00
Lampros Smyrnaios 5e7ccbd8c6 Add the "addWorkerReport" endpoint. 2021-06-22 05:38:48 +03:00
Lampros Smyrnaios 40763ec146 Update the "WorkerReport" request and the "UrlReport" and "Payload" models. 2021-06-19 07:07:36 +03:00
Lampros Smyrnaios c194af167f Allow handling of concurrent requests to the "getTestUrls"-endpoint. 2021-06-10 20:24:51 +03:00
Lampros Smyrnaios 308cab5ecd - Return an HTTP-500-error when the server cannot find the resourceFile requested by the "getTestUrls"-endpoint.
- Close the "inputScanner" after each use when retrieving the test-tasks.
- Show info-logs when sending an assignment to a worker.
- Code cleanup.
2021-06-10 14:21:39 +03:00
Lampros Smyrnaios 6729f51b03 Add an "assignmentId" field in the "Assignment"-class. 2021-06-09 05:48:54 +03:00
Lampros Smyrnaios 87044574b5 Update dependencies and add the "gradle-wrapper.properties" file which defines the gradle version. 2021-06-08 19:12:40 +03:00
Lampros Smyrnaios 787299b5b7 Add the "Datasource" inside the "Task" class and include it in the Assignment. 2021-05-20 02:50:50 +03:00
Lampros Smyrnaios d20fcf9cce - Update the "getUrls" and "getTestUrls" endpoints to take the data from the workers, in http-request-parameters instead of the http-body.
- Fix tasksLimit-check.
- Code cleanup.
2021-05-19 02:32:46 +03:00
Lampros Smyrnaios e2cc320baf - Add the "getTestUrls"-endpoint which returns an "Assignment" with data retrieved from the added resource-file.
- Update the "getUrls"-endpoint to be ready to retrieve data from the database, once it's added.
- Update the dependencies.
- Code cleanup.
2021-05-18 17:23:20 +03:00
Lampros Smyrnaios d3588ea36b Add the "DownloadAttempt" class. 2021-04-24 21:44:51 +03:00
Lampros Smyrnaios a6ab810ad3 Update classes: "Publication" and "Payload". 2021-04-24 21:40:10 +03:00
Lampros Smyrnaios 85ecc4a36b Add classes: "AssignmentResponse", "WorkerReport", "WorkerRequest", "UrlReport". 2021-04-24 21:06:52 +03:00
Lampros Smyrnaios c2ea8a69de Update classes: "Assignment", "Task", "Error", "Payload", "UrlsRequest". 2021-04-24 21:05:21 +03:00
Lampros Smyrnaios 89c6a73a30 Add "Assignment", "Task" and "Error" classes. 2021-04-15 03:36:08 +03:00
Lampros Smyrnaios c6e12d3e95 - Update "addResults"-endpoint.
- Add "UrlsRequest.java".
- Some minor updates in "build.gradle" and "application.properties".
2021-03-16 18:07:30 +02:00