lsmyrnaios
/
UrlsController
1
0
Fork 1
Code Issues Pull Requests Releases 9 Wiki Activity
277 Commits
2 Branches
9 Tags
4.1 MiB
master
springify_project
2.6.2
2.6.1
2.4
2.3
2.2
2.1
2.0
2.6
2.5
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
Commit Graph

73 Commits (master)

Author SHA1 Message Date
Lampros Smyrnaios 1d821ed803 - Prepare version for next release. 
- Fix typo of not using the "OpenAireID" in the S3 location of bulkImported files. Instead, the "fileNameID" was used, which in aggregation is the OpenAireID, but not in bulk-import.
- Update dependencies.
- Code polishing.
 4 weeks ago
Lampros Smyrnaios 724eae1514 - Optimize the placement of "DatabaseConnector.databaseLock.unlock()" statements. 
- Rename a maven-repository.
 1 month ago
Lampros Smyrnaios 785204419d Update/cleanup the repositories in "build.gradle". 1 month ago
Lampros Smyrnaios dd394f18a0 - Optimize the JOIN-order in the "findAssignmentsQuery". 
- Optimize the "DOC_URL_FILTER"-regex.
- Update dependencies.
 1 month ago
Lampros Smyrnaios 3563fd6e2a - Try to get the cause of the exception of the callable-tasks which handle parquet-files. 
- Update License.
- Update dependencies.
 3 months ago
Lampros Smyrnaios 5dadb8ad2f - Optimize the "DOC_URL_FILTER"-regex, by using a non-capturing group. 
- Remove an extra "File.separator" from the fulltexts-fullFilePath.
 3 months ago
Lampros Smyrnaios ee1ca8966b - Avoid continuing to request workerReport-batches when from the 1st batch, the base-directory of that assignments-counter is not found. 
- Update dependencies.
 3 months ago
Lampros Smyrnaios 2e60128084 - Allow to easily change the por used by workers. 
- Show the number of active background-tasks and bulkImportDirs, which delay the Service's shutdown.
- Update dependencies.
- Code polishing.
 4 months ago
Lampros Smyrnaios d20c9a7d2e - Show the original exception thrown by the background-job, not the one thrown in the main-thread, which is useless, except from its message. 
- Reduce the interval for deleting the unhandled assignments to once every 3 days.
- Set the upcoming version.
- Update dependencies.
 5 months ago
Lampros Smyrnaios 44c2fe7418 - Fix the "IndexOutOfBoundsException", when checking the futures' results. 
- Update dependencies.
 6 months ago
Lampros Smyrnaios a354da763d - Improve some log-messages. 
- Increase app's version.
- Code polishing.
 7 months ago
Lampros Smyrnaios 96c11ba4b8 - Add a missing change. 
- Code optimization and polishing.
- Update dependencies.
 7 months ago
Lampros Smyrnaios ede7ca5a89 - Add bulk-import support for non-Authoritative data-sources. 
- Update Spring Boot.
- Code polishing.
 7 months ago
Lampros Smyrnaios 6891c467d4 - Avoid displaying a warning for the "test" HDFS directory, when the Controller is running in PROD mode. 
- Add a missing change for the optimization of reading files.
- Update dependencies.
 7 months ago
Lampros Smyrnaios 98516498eb - Increase app's version. 
- Code polishing.
 8 months ago
Lampros Smyrnaios aa3f32f3da - Make sure the given number of threads, given by the user is above zero. 
- Adjust the number and size of log files.
- Update Spring Boot.
- Code polishing.
 8 months ago
Lampros Smyrnaios 44459c8681 - Rename "ImpalaConnector.java" to "DatabaseConnector.java". 
- Update dependencies.
- Code polishing.
 8 months ago
Lampros Smyrnaios 0699acc999 Make sure we use the latest version of the "zstd-jni" library, where the core code for the "ZStandard" compression algorithm is. The Apache's "commons-compress" package which wraps it in a file-managements code, updates the "zstd-jni" less often. 9 months ago
Lampros Smyrnaios cde6063d63 - Update dependencies. 
- Code polishing.
 9 months ago
Lampros Smyrnaios b94c35c66e - Fix double active "@Scheduled" annotation for the "ScheduledTasks.updatePrometheusMetrics()" method. 
- Code polishing.
 10 months ago
Lampros Smyrnaios e8644cb64f - Optimize the "insertAssignmentsQuery". 
- Add documentation about the Prometheus Metrics, in README.
- Update Dependencies.
- Code polishing.
 10 months ago
Lampros Smyrnaios 798fa09d68 - Identify and handle a possible Worker-crash, in "UrlsServiceImpl.postReportResultToWorker()". 
- Add/Improve some log messages.
- Update and cleanup dependencies.
- Code polishing.
 10 months ago
Lampros Smyrnaios 5d99a4be5d - Add the Shutdown Service API documentation. 
- Improve the BulkImport API documentation.
- Fix markdown in README.
- Update the app's version.
 11 months ago
Lampros Smyrnaios 03bf4294b8 - Add documentation about the "BulkImport API" in the README. 
- Fix a link in README.
- Update dependencies.
 11 months ago
Lampros Smyrnaios 5f75b48e95 - Increase the "read-timeout" when searching for the host's machine public-IP. 
- Update dependencies.
- Code polishing.
 11 months ago
Lampros Smyrnaios b6e8cd1889 New feature: BulkImport full-text files from compatible datasources. 12 months ago
Lampros Smyrnaios 68759e3023 Update dependencies. 1 year ago
Lampros Smyrnaios 495d5de19b - Automatically get the status of the docker containers after 30 secs of their initialization. 
- Add an error-handling in "installAndRun.sh"
- Update dependencies.
 1 year ago
Lampros Smyrnaios e975bec911 - Add Prometheus and Grafana which help measuring various metrics for the Controller's health and performance. 
- Fix Docker config still using the old (now removed) "application.properties" file.
- Simplify the process of building and running the docker image; Now we use docker compose to run the Controller, along with the Prometheus and Grafana. Also, now it is not requested from the user to login and push the image (this may change in the future).
 1 year ago
Lampros Smyrnaios 38643c76a3 - Code polishing. 
- Update Gradle.
 1 year ago
Lampros Smyrnaios 4af298a52a Revert the version of "libthrift"-dependency to "0.17.0", as the newer version is not compatible with Java 8. 1 year ago
Lampros Smyrnaios 62a4279e3b Update dependencies. 1 year ago
Lampros Smyrnaios c4670073ae - Add missing refactoring-change. 
- Code polishing.
- Update Spring.
 1 year ago
Lampros Smyrnaios dc8f0f2bd1 - Heavily reduce the maximum amount of space needed, by deleting the files of each full-texts batch, right after they are uploaded to the S3 Object Store. 
- Add a check for when the retrieved full-texts-batch is missing some requested files and show a warn-log.
- Update dependencies.
 1 year ago
Lampros Smyrnaios b0b00c8aed Update the minio dependency. 1 year ago
Lampros Smyrnaios c08ba1cc89 Revert the update of the "minio" dependency, as it introduces a bug, related to the "okhttp3.HttpUrl"-class. 1 year ago
Lampros Smyrnaios 8876089022 - Use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits on compression rate and speed. 
- Update the minIO dependency.
- Code polishing.
 1 year ago
Lampros Smyrnaios 9904ea5743 - Improve the stability of "UriBuilder.getPublicIP()", by using a "HttpURLConnection" to increase the connection and read timeouts and avoid timeout-exceptions. 
- Update Spring.
 1 year ago
Lampros Smyrnaios e51ee9dd27 - Add info about the Stats API usage in "README.md". 
- Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()".
- Handle the "EmptyResultDataAccessException" inside "StatsController".
- Optimize gradle's performance.
- Code polishing.
 1 year ago
Lampros Smyrnaios 8607594f6d - Improve exception handling. 
- Code polishing.
 1 year ago
Lampros Smyrnaios 577ea983e8 - Improve some log-messages. 
- Set some optimization settings for gradle.
- Fix error-handling in "installAndRun.sh".
- Update dependencies.
 1 year ago
Lampros Smyrnaios 6226e2298d - Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement. 
One side effect of using the parquet-files, is that the timestamps are now BIGDECIMAL numbers, instead of "Timestamp" objects, but, converting them to such objects is pretty easy, if we ever need to do it.
- Code polishing.
 1 year ago
Lampros Smyrnaios 6a03103b79 Update dependencies. 1 year ago
Lampros Smyrnaios a2cd02115f - Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated. 
- Update dependencies.
 2 years ago
Lampros Smyrnaios e3b374a32f - Optimize file-related tasks. 
- Update dependencies.
- Code cleanup.
 2 years ago
Lampros Smyrnaios 9b95eebb6c - Remove the obsolete "parenthesis" and "increasing duplicate-num" from the full-texts' names, before sending them to the S3-Object-Store. They now end with the "file-hash", so it is guaranteed that they will be unique. The Worker continues to produce the previous kind of names, without any disturbance. 
- Improve logging.
- Update MinIO dependency.
 2 years ago
Lampros Smyrnaios 33fc61a8d9 - Fix the fileName-ID not being directly related with the datasourceID, in the S3-ObjectStore name. Add explanatory comments. 
- Add missing error-logs.
 2 years ago
Lampros Smyrnaios 5e4fad2479 - Change the fileNames' structure in the S3-ObjectStore. 
- Update dependencies.
 2 years ago
Lampros Smyrnaios 48670f3399 - Show the percentage of the "NumFullTextsFound", in the logs. 
- Update dependencies.
 2 years ago
Lampros Smyrnaios e587b2ca6c Update Spring dependencies. 2 years ago