Commit Graph

24 Commits (master)

Author SHA1 Message Date
Lampros Smyrnaios 1d821ed803 - Prepare version for next release.
- Fix typo of not using the "OpenAireID" in the S3 location of bulkImported files. Instead, the "fileNameID" was used, which in aggregation is the OpenAireID, but not in bulk-import.
- Update dependencies.
- Code polishing.
3 weeks ago
Lampros Smyrnaios 3563fd6e2a - Try to get the cause of the exception of the callable-tasks which handle parquet-files.
- Update License.
- Update dependencies.
2 months ago
Lampros Smyrnaios 2e60128084 - Allow to easily change the por used by workers.
- Show the number of active background-tasks and bulkImportDirs, which delay the Service's shutdown.
- Update dependencies.
- Code polishing.
4 months ago
Lampros Smyrnaios ebf8896005 - Fix getter and setter methods for the "isAuthoritative" field.
- Update Gradle.
7 months ago
Lampros Smyrnaios 44459c8681 - Rename "ImpalaConnector.java" to "DatabaseConnector.java".
- Update dependencies.
- Code polishing.
8 months ago
Lampros Smyrnaios cde6063d63 - Update dependencies.
- Code polishing.
9 months ago
Lampros Smyrnaios e8644cb64f - Optimize the "insertAssignmentsQuery".
- Add documentation about the Prometheus Metrics, in README.
- Update Dependencies.
- Code polishing.
10 months ago
Lampros Smyrnaios 1b14a7e554 - Add profiles to docker-services to selectively run the additional "Prometheus" and "Grafana" services or not.
- Update Gradle.
1 year ago
Lampros Smyrnaios 68759e3023 Update dependencies. 1 year ago
Lampros Smyrnaios 38643c76a3 - Code polishing.
- Update Gradle.
1 year ago
Lampros Smyrnaios b7f6056032 - Improve an error-message.
- Update Gradle.
1 year ago
Lampros Smyrnaios a1c16ffc19 - Exclude empty and null urls in the assignments.
- Update the "getFullTextsImproved"-call to "getFullTexts", now that the "improved" version is stable.
- Update Gradle.
- Code polishing.
1 year ago
Lampros Smyrnaios 577ea983e8 - Improve some log-messages.
- Set some optimization settings for gradle.
- Fix error-handling in "installAndRun.sh".
- Update dependencies.
1 year ago
Lampros Smyrnaios 6a03103b79 Update dependencies. 1 year ago
Lampros Smyrnaios 5e4fad2479 - Change the fileNames' structure in the S3-ObjectStore.
- Update dependencies.
2 years ago
Lampros Smyrnaios 48670f3399 - Show the percentage of the "NumFullTextsFound", in the logs.
- Update dependencies.
2 years ago
Lampros Smyrnaios 71f6b46130 - In case of an error when creating the "current_assignment" table (e.g out of memory in the backend database server), check for partial-creation and drop it. Also, in any case, before we drop this table, now check if it exists firsts (in general it should always exist, unless the creation results in an error and the table was not created at all).
- Fix an error-message.
- Update dependencies.
- Code cleanup.
2 years ago
Lampros Smyrnaios 82bf11b9b3 - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements.
- Update dependencies.
2 years ago
Lampros Smyrnaios 33ba3e8d91 - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches.
- Fix not updating the fileLocation with the s3Url for records which share the same full-text.
- Set only one delete-order for each assignments-batch-files, not one (or more, by mistake) per zip-batch.
- Set the HttpStatus to "204 - NO_CONTENT", when no assignments are available to be returned to the Worker.
- Fix not unlocking the "dataBaseLock" in case of a "dataBase-connection"-error, in "addWorkerReport()".
- Improve some log-messages.
- Change the log-level for the "S3-bucket already exists" message.
- Update Gradle.
- Optimize imports.
- Code cleanup.
2 years ago
Lampros Smyrnaios dea257b87f - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter".
- Fix a bug, which caused multiple workers to get assigned the same batch-counter, while the assignment-tasks where different.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the error-message returned from the Worker, when a getFullTexts-request fails.
- Improve some log-messages.
- Update dependencies.
- Code cleanup.
2 years ago
Lampros Smyrnaios 780ed15ce2 - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction.
- Fix the "null" representation of an "unknown" payload-size in the database.
- Remove the obsolete thread-locking for the "CreateDatabase" operation. This code is guaranteed to run BEFORE any other operation in the database.
- Implement the "handlePreparedStatementException" and "closeConnection" methods.
- Improve error-logs.
- Update dependencies.
- Code cleanup.
2 years ago
Lampros Smyrnaios 983b900da7 - Add the "installAndRun.sh" script.
- Update the README.
- Update the dependencies.
3 years ago
Lampros Smyrnaios 25c566bf68 - Change server's port.
- Update dependencies.
3 years ago
Lampros Smyrnaios 87044574b5 Update dependencies and add the "gradle-wrapper.properties" file which defines the gradle version. 3 years ago