michal.politowski
/
UrlsController
forked from lsmyrnaios/UrlsController
1
0
Fork 0
Code Pull Requests Activity
302 Commits 3 Branches 14 Tags 1.1 MiB
Commit Graph

83 Commits

Author SHA1 Message Date
Lampros Smyrnaios 7a8270c69f - Perform manual synchronization on "BulkImportReport.eventsMultimap", in order to avoid the "ConcurrentModificationException" when requesting a BulkImport-report. 
- Prepare app-version for next release.
 2024-07-04 01:35:31 +03:00
Lampros Smyrnaios 9e9f417f1f - Remove the unused "accessmode" column from the results returned by the "findAssignmentsQuery". 
- Update dependencies.
- Code polishing.
 2024-06-27 23:10:46 +03:00
Lampros Smyrnaios c45a172c21 - Add/improve log-messages. 
- Set new version.
- Update dependencies.
- Code polishing.
 2024-06-17 12:26:42 +03:00
Lampros Smyrnaios ed7bf09f9b - Replace all "json" usages, with "gson" ones, in order to remove the "org.json:json" dependency. 
- Add an extra check to verify that the remote parquet directories are directories indeed.
- Set new version.
 2024-06-06 14:40:39 +03:00
Lampros Smyrnaios 643e497826 - Fix a regression, in which the BulkImportReport was not returning the events in a time-ordered state. 
- Update dependencies.
 2024-06-03 13:07:17 +03:00
Lampros Smyrnaios a48f07f41d - Increase the "-Xmx" java argument to 6Gb. 
- Set the version to 2.7.3.
 2024-05-31 21:43:56 +03:00
Lampros Smyrnaios edf064616a Improve error-handling in "BulkImportReport.getJsonReport()" and "FileUtils.writeToFile()". 2024-05-30 11:52:04 +03:00
Lampros Smyrnaios d7697ef3f8 - Tighten the thread-safety protection on the "BulkImportReport.getJsonReport()" method. 
- Update dependencies.
- Code polishing.
 2024-05-27 10:40:05 +03:00
Lampros Smyrnaios e2e7ca72d5 - Fix not allowing the user to use the "shutdownAllWorkersGracefully" endpoint twice. 
- Code optimization.
- Update dependencies.
 2024-05-21 23:43:49 +03:00
Lampros Smyrnaios 64a1b7d4f0 - Comment-out the bucket-deletion process, in order to avoid any accidental deletion, even if it has to be explicitly allowed in the config. 
- Update dependencies.
 2024-04-26 12:54:00 +03:00
Lampros Smyrnaios 1d821ed803 - Prepare version for next release. 
- Fix typo of not using the "OpenAireID" in the S3 location of bulkImported files. Instead, the "fileNameID" was used, which in aggregation is the OpenAireID, but not in bulk-import.
- Update dependencies.
- Code polishing.
 2024-03-28 06:09:28 +02:00
Lampros Smyrnaios 724eae1514 - Optimize the placement of "DatabaseConnector.databaseLock.unlock()" statements. 
- Rename a maven-repository.
 2024-03-20 15:08:01 +02:00
Lampros Smyrnaios 785204419d Update/cleanup the repositories in "build.gradle". 2024-03-15 12:22:13 +02:00
Lampros Smyrnaios dd394f18a0 - Optimize the JOIN-order in the "findAssignmentsQuery". 
- Optimize the "DOC_URL_FILTER"-regex.
- Update dependencies.
 2024-03-11 11:35:38 +02:00
Lampros Smyrnaios 3563fd6e2a - Try to get the cause of the exception of the callable-tasks which handle parquet-files. 
- Update License.
- Update dependencies.
 2024-02-07 18:34:28 +02:00
Lampros Smyrnaios 5dadb8ad2f - Optimize the "DOC_URL_FILTER"-regex, by using a non-capturing group. 
- Remove an extra "File.separator" from the fulltexts-fullFilePath.
 2024-01-19 15:46:23 +02:00
Lampros Smyrnaios ee1ca8966b - Avoid continuing to request workerReport-batches when from the 1st batch, the base-directory of that assignments-counter is not found. 
- Update dependencies.
 2024-01-15 12:57:33 +02:00
Lampros Smyrnaios 2e60128084 - Allow to easily change the por used by workers. 
- Show the number of active background-tasks and bulkImportDirs, which delay the Service's shutdown.
- Update dependencies.
- Code polishing.
 2023-12-19 23:31:42 +02:00
Lampros Smyrnaios d20c9a7d2e - Show the original exception thrown by the background-job, not the one thrown in the main-thread, which is useless, except from its message. 
- Reduce the interval for deleting the unhandled assignments to once every 3 days.
- Set the upcoming version.
- Update dependencies.
 2023-11-27 18:19:53 +02:00
Lampros Smyrnaios 44c2fe7418 - Fix the "IndexOutOfBoundsException", when checking the futures' results. 
- Update dependencies.
 2023-10-20 14:25:05 +03:00
Lampros Smyrnaios a354da763d - Improve some log-messages. 
- Increase app's version.
- Code polishing.
 2023-10-06 17:28:54 +03:00
Lampros Smyrnaios 96c11ba4b8 - Add a missing change. 
- Code optimization and polishing.
- Update dependencies.
 2023-10-04 16:17:12 +03:00
Lampros Smyrnaios ede7ca5a89 - Add bulk-import support for non-Authoritative data-sources. 
- Update Spring Boot.
- Code polishing.
 2023-09-26 18:02:48 +03:00
Lampros Smyrnaios 6891c467d4 - Avoid displaying a warning for the "test" HDFS directory, when the Controller is running in PROD mode. 
- Add a missing change for the optimization of reading files.
- Update dependencies.
 2023-09-13 15:29:30 +03:00
Lampros Smyrnaios 98516498eb - Increase app's version. 
- Code polishing.
 2023-09-04 12:46:55 +03:00
Lampros Smyrnaios aa3f32f3da - Make sure the given number of threads, given by the user is above zero. 
- Adjust the number and size of log files.
- Update Spring Boot.
- Code polishing.
 2023-08-30 14:02:54 +03:00
Lampros Smyrnaios 44459c8681 - Rename "ImpalaConnector.java" to "DatabaseConnector.java". 
- Update dependencies.
- Code polishing.
 2023-08-23 16:55:23 +03:00
Lampros Smyrnaios 0699acc999 Make sure we use the latest version of the "zstd-jni" library, where the core code for the "ZStandard" compression algorithm is. The Apache's "commons-compress" package which wraps it in a file-managements code, updates the "zstd-jni" less often. 2023-07-27 17:42:57 +03:00
Lampros Smyrnaios cde6063d63 - Update dependencies. 
- Code polishing.
 2023-07-25 12:12:56 +03:00
Lampros Smyrnaios b94c35c66e - Fix double active "@Scheduled" annotation for the "ScheduledTasks.updatePrometheusMetrics()" method. 
- Code polishing.
 2023-07-13 18:32:45 +03:00
Lampros Smyrnaios e8644cb64f - Optimize the "insertAssignmentsQuery". 
- Add documentation about the Prometheus Metrics, in README.
- Update Dependencies.
- Code polishing.
 2023-07-05 17:10:30 +03:00
Lampros Smyrnaios 798fa09d68 - Identify and handle a possible Worker-crash, in "UrlsServiceImpl.postReportResultToWorker()". 
- Add/Improve some log messages.
- Update and cleanup dependencies.
- Code polishing.
 2023-06-15 23:19:36 +03:00
Lampros Smyrnaios 5d99a4be5d - Add the Shutdown Service API documentation. 
- Improve the BulkImport API documentation.
- Fix markdown in README.
- Update the app's version.
 2023-06-06 16:18:38 +03:00
Lampros Smyrnaios 03bf4294b8 - Add documentation about the "BulkImport API" in the README. 
- Fix a link in README.
- Update dependencies.
 2023-05-29 12:13:39 +03:00
Lampros Smyrnaios 5f75b48e95 - Increase the "read-timeout" when searching for the host's machine public-IP. 
- Update dependencies.
- Code polishing.
 2023-05-22 21:33:02 +03:00
Lampros Smyrnaios b6e8cd1889 New feature: BulkImport full-text files from compatible datasources. 2023-05-11 03:07:55 +03:00
Lampros Smyrnaios 68759e3023 Update dependencies. 2023-04-20 18:57:16 +03:00
Lampros Smyrnaios 495d5de19b - Automatically get the status of the docker containers after 30 secs of their initialization. 
- Add an error-handling in "installAndRun.sh"
- Update dependencies.
 2023-03-27 19:43:15 +03:00
Lampros Smyrnaios e975bec911 - Add Prometheus and Grafana which help measuring various metrics for the Controller's health and performance. 
- Fix Docker config still using the old (now removed) "application.properties" file.
- Simplify the process of building and running the docker image; Now we use docker compose to run the Controller, along with the Prometheus and Grafana. Also, now it is not requested from the user to login and push the image (this may change in the future).
 2023-03-21 16:46:33 +02:00
Lampros Smyrnaios 38643c76a3 - Code polishing. 
- Update Gradle.
 2023-03-07 16:55:41 +02:00
Lampros Smyrnaios 4af298a52a Revert the version of "libthrift"-dependency to "0.17.0", as the newer version is not compatible with Java 8. 2023-03-03 12:57:30 +02:00
Lampros Smyrnaios 62a4279e3b Update dependencies. 2023-03-02 17:40:16 +02:00
Lampros Smyrnaios c4670073ae - Add missing refactoring-change. 
- Code polishing.
- Update Spring.
 2023-02-24 23:49:04 +02:00
Lampros Smyrnaios dc8f0f2bd1 - Heavily reduce the maximum amount of space needed, by deleting the files of each full-texts batch, right after they are uploaded to the S3 Object Store. 
- Add a check for when the retrieved full-texts-batch is missing some requested files and show a warn-log.
- Update dependencies.
 2023-01-23 20:23:21 +02:00
Lampros Smyrnaios b0b00c8aed Update the minio dependency. 2023-01-11 15:46:34 +02:00
Lampros Smyrnaios c08ba1cc89 Revert the update of the "minio" dependency, as it introduces a bug, related to the "okhttp3.HttpUrl"-class. 2023-01-10 15:58:23 +02:00
Lampros Smyrnaios 8876089022 - Use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits on compression rate and speed. 
- Update the minIO dependency.
- Code polishing.
 2023-01-10 13:34:54 +02:00
Lampros Smyrnaios 9904ea5743 - Improve the stability of "UriBuilder.getPublicIP()", by using a "HttpURLConnection" to increase the connection and read timeouts and avoid timeout-exceptions. 
- Update Spring.
 2023-01-03 18:39:50 +02:00
Lampros Smyrnaios e51ee9dd27 - Add info about the Stats API usage in "README.md". 
- Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()".
- Handle the "EmptyResultDataAccessException" inside "StatsController".
- Optimize gradle's performance.
- Code polishing.
 2022-12-15 14:04:22 +02:00
Lampros Smyrnaios 8607594f6d - Improve exception handling. 
- Code polishing.
 2022-12-07 13:48:00 +02:00