Lampros Smyrnaios
c7bfd75973
- Add the "getWorkersInfo" endpoint.
...
- Improve startup speed, by using a faster remote server to get the host's machine public IP. This also reduces the risk of not being able to get the public IP at all.
- Fix the detection of a different IP for a known worker.
- Improve documentation.
2023-05-23 14:57:15 +03:00
Lampros Smyrnaios
5f75b48e95
- Increase the "read-timeout" when searching for the host's machine public-IP.
...
- Update dependencies.
- Code polishing.
2023-05-22 21:33:02 +03:00
Lampros Smyrnaios
0ab6bae93a
- Optimize the json-conversion of the "BulkImportReport".
...
- Code polishing.
2023-05-18 17:30:40 +03:00
Lampros Smyrnaios
f7f919cee1
- Make sure we set the "hasShutdown" to "false", for each known worker which was restarted.
...
- Fix markdown of urls in prometheus' readme.
2023-05-16 12:24:14 +03:00
Lampros Smyrnaios
b499209ce3
- Move the Prometheus and grafana configuration in a dedicated directory and docker-compose file.
...
- Add documentation about setting-up prometheus and grafana.
2023-05-15 18:52:31 +03:00
Lampros Smyrnaios
a8eea1ccf4
Fix missing changes.
2023-05-15 13:13:24 +03:00
Lampros Smyrnaios
f51a34138f
- Use separate HDFS subdirectories for each worker in order to avoid seeing exceptions about "empty hdfs directory" when "loading" data to the database, because one worker has loaded data generated by multiple workers (since we use only 1 load operation for multiple parquet files).
...
- Store each worker's info in a hash-table, in order to efficiently know if we need to create new hdfs subdirectories. Also, this will help to issue "shutdown" requests to the workers in the future, as well as to know which worker has shutdown.
2023-05-15 13:12:20 +03:00
Lampros Smyrnaios
9412391903
- In test-environment mode, check for already existing file-hashes only in the "payload_aggregated" table, instead of the whole "payload" view. This way the investigation for false-positive docUrls is easier, as we avoid checking against the millions of "legacy" payloads.
...
- Improve performance in production, by not creating the string objects for "trace"-logs.
2023-05-15 12:44:16 +03:00
Lampros Smyrnaios
8381df70c6
- Improve performance of uploading parquet-files to HDFS.
...
- Add some logs.
- Code polishing.
2023-05-11 19:40:48 +03:00
Lampros Smyrnaios
992d4ffd5e
- Add the time-zone in the logs.
...
- Change some log-levels to "trace", although most of them are still disabled.
2023-05-11 03:10:53 +03:00
Lampros Smyrnaios
b6e8cd1889
New feature: BulkImport full-text files from compatible datasources.
2023-05-11 03:07:55 +03:00
Lampros Smyrnaios
42b93e9429
- Add the "getNumberOfAllDistinctFullTexts" stats-endpoint.
...
- Add TODOs for more stats endpoints.
- Code polishing.
2023-05-04 15:48:49 +03:00
Lampros Smyrnaios
b3196376eb
Fix a bug, which caused the full-text files to never close.
2023-05-04 13:03:28 +03:00
Lampros Smyrnaios
3473d91ce4
Add the "shutdownController.sh" script.
2023-05-03 20:43:44 +03:00
Lampros Smyrnaios
fd15372fd6
Add error-checks for retrieving the status-code from HttpUrlConnections.
2023-05-03 13:30:29 +03:00
Lampros Smyrnaios
49662319a1
- Simplify the creation of local directories.
...
- Improve exception messages.
2023-04-28 14:58:33 +03:00
Lampros Smyrnaios
55ea5118ac
- Update the "testDatabaseName" property.
...
- Code polishing.
2023-04-26 19:33:28 +03:00
Lampros Smyrnaios
d7797eaaf6
Add the "getNumberOfPayloadsForDatasource" endpoint.
2023-04-24 09:54:35 +03:00
Lampros Smyrnaios
1b14a7e554
- Add profiles to docker-services to selectively run the additional "Prometheus" and "Grafana" services or not.
...
- Update Gradle.
2023-04-22 16:50:33 +03:00
Lampros Smyrnaios
68759e3023
Update dependencies.
2023-04-20 18:57:16 +03:00
Lampros Smyrnaios
c2b17163cd
Automatically show the Controller's logs after the docker-container starts running and the status is shown.
2023-04-11 11:59:10 +03:00
Lampros Smyrnaios
4dc34429f8
- Increase the waiting-time before checking the docker containers' status, in order to catch configuration-crashes.
...
- Code polishing.
2023-04-10 22:28:53 +03:00
Lampros Smyrnaios
c39fef2654
Upgrade payload-table to payload-view which consists of three separate payload tables: "payload_legacy", "payload_aggregated" and "payload_bulk_import".
2023-04-10 15:55:50 +03:00
Lampros Smyrnaios
37363100fd
Prioritize most recent publications.
2023-04-10 15:00:23 +03:00
Lampros Smyrnaios
484cf5cefc
- Avoid requesting the remaining full-text batches in case the Worker returns a 5XX error in one of the batches.
...
- Add nullability-checks for "datasourceId" and "hash" before constructing the new filename and upload the full-text on S3.
- Improve a log-message.
2023-03-29 17:12:37 +03:00
Lampros Smyrnaios
495d5de19b
- Automatically get the status of the docker containers after 30 secs of their initialization.
...
- Add an error-handling in "installAndRun.sh"
- Update dependencies.
2023-03-27 19:43:15 +03:00
Lampros Smyrnaios
882c6f447b
Update the "testDatabaseName".
2023-03-21 23:10:21 +02:00
Lampros Smyrnaios
4280f89296
- Set the default value of the "isTestEnvironment" property to "true", in order to avoid undesired outcomes in the production db.
...
- Code polishing.
2023-03-21 17:04:28 +02:00
Lampros Smyrnaios
e975bec911
- Add Prometheus and Grafana which help measuring various metrics for the Controller's health and performance.
...
- Fix Docker config still using the old (now removed) "application.properties" file.
- Simplify the process of building and running the docker image; Now we use docker compose to run the Controller, along with the Prometheus and Grafana. Also, now it is not requested from the user to login and push the image (this may change in the future).
2023-03-21 16:46:33 +02:00
Lampros Smyrnaios
003c0bf179
- Add support for excluding specific datasources from being crawled. These datasources may be aggregated through bulk-imports, by other pieces of software. Such a datasource is "arXiv.org".
...
- Fix an issue, where the "datasource-type" was retrieved instead of the "datasource-name".
- Polish the "findAssignmentsQuery".
2023-03-21 07:19:35 +02:00
Lampros Smyrnaios
f835a752bf
Transform the "application.properties" file to "application.yml" and optimize the property-trees.
2023-03-20 15:23:00 +02:00
Lampros Smyrnaios
17a6c120dd
Improve logs for full-texts' metrics.
2023-03-14 20:57:01 +02:00
Lampros Smyrnaios
ff13af7abb
Use a StatsService interface.
2023-03-13 12:39:39 +02:00
Lampros Smyrnaios
38643c76a3
- Code polishing.
...
- Update Gradle.
2023-03-07 16:55:41 +02:00
Lampros Smyrnaios
4af298a52a
Revert the version of "libthrift"-dependency to "0.17.0", as the newer version is not compatible with Java 8.
2023-03-03 12:57:30 +02:00
Lampros Smyrnaios
7b217764e0
Improve performance when downloading and decompressing the full-texts archive.
2023-03-02 17:44:53 +02:00
Lampros Smyrnaios
62a4279e3b
Update dependencies.
2023-03-02 17:40:16 +02:00
Lampros Smyrnaios
c4670073ae
- Add missing refactoring-change.
...
- Code polishing.
- Update Spring.
2023-02-24 23:49:04 +02:00
Lampros Smyrnaios
c8485d472e
Code polishing.
2023-02-24 13:53:09 +02:00
Lampros Smyrnaios
b7f6056032
- Improve an error-message.
...
- Update Gradle.
2023-02-21 15:42:07 +02:00
Lampros Smyrnaios
8893662a81
Refactor the UrlsController: a) offload the business-logic to the dedicated "UrlsService" and b) move the "checkParquetFilesSuccess()"-method to "ParquetFileUtils".
2023-02-21 15:36:35 +02:00
Lampros Smyrnaios
a1c16ffc19
- Exclude empty and null urls in the assignments.
...
- Update the "getFullTextsImproved"-call to "getFullTexts", now that the "improved" version is stable.
- Update Gradle.
- Code polishing.
2023-02-16 14:24:47 +02:00
Lampros Smyrnaios
2253f05bf5
Refactor the "StatsController"-code, by offloading it to a dedicated "StatsService".
2023-02-09 19:25:48 +02:00
Lampros Smyrnaios
49fefefafd
- Refactor the payloads-statistics-code and provide two endpoints: "getNumberOfPayloadsAggregatedByService", which returns the number of payloads aggregated only by the PDF-Aggregation-Service, and the "getNumberOfAllPayloads", which returns the number of all payloads existing in the database, even the ones aggregated in the past, by other pieces of software.
...
- Update README.md.
- Make sure the docker image is clean-built, by avoiding the use of cache.
2023-02-02 17:58:47 +02:00
Lampros Smyrnaios
c9f33d3afa
Add an extra precaution-check to allow the emptying or deletion of an S3-Object-Store bucket, only when the app runs in "TestEnvironment".
2023-02-01 16:42:22 +02:00
Lampros Smyrnaios
f89730f196
Improve documentation.
2023-01-27 14:31:07 +02:00
Lampros Smyrnaios
dc8f0f2bd1
- Heavily reduce the maximum amount of space needed, by deleting the files of each full-texts batch, right after they are uploaded to the S3 Object Store.
...
- Add a check for when the retrieved full-texts-batch is missing some requested files and show a warn-log.
- Update dependencies.
2023-01-23 20:23:21 +02:00
Lampros Smyrnaios
d8773e6ebb
- Make sure the test-environment uses a dedicated hdfs-parquet-directory.
...
- Block app-execution in case the hdfs parquet directories failed to be created.
- Code polishing.
2023-01-18 13:38:05 +02:00
Lampros Smyrnaios
b0b00c8aed
Update the minio dependency.
2023-01-11 15:46:34 +02:00
Lampros Smyrnaios
c08ba1cc89
Revert the update of the "minio" dependency, as it introduces a bug, related to the "okhttp3.HttpUrl"-class.
2023-01-10 15:58:23 +02:00