Lampros Smyrnaios
cd1fb0af88
- Process the WorkerReports in background Jobs and post the reportResults to the Workers.
...
- Save the workerReports to json files, until they are processed successfully.
- Show some custom metrics in prometheus.
2023-05-24 13:52:28 +03:00
Lampros Smyrnaios
0ea3e2de24
Add the "shutdownService" and "cancelShutdownService" endpoints. The Controller sends the related requests to the Workers and shutdowns gracefully, after all workers have shutdown.
2023-05-24 13:42:29 +03:00
Lampros Smyrnaios
c2a1b96069
- Rename the mounted "mnt/bulkImport/" directory to "/mnt/bulk_import/".
...
- Increase the "awaitTermination" timeout for the ExecutorService to 2 minutes.
2023-05-23 21:09:34 +03:00
Lampros Smyrnaios
b6e8cd1889
New feature: BulkImport full-text files from compatible datasources.
2023-05-11 03:07:55 +03:00
Lampros Smyrnaios
55ea5118ac
- Update the "testDatabaseName" property.
...
- Code polishing.
2023-04-26 19:33:28 +03:00
Lampros Smyrnaios
c39fef2654
Upgrade payload-table to payload-view which consists of three separate payload tables: "payload_legacy", "payload_aggregated" and "payload_bulk_import".
2023-04-10 15:55:50 +03:00
Lampros Smyrnaios
882c6f447b
Update the "testDatabaseName".
2023-03-21 23:10:21 +02:00
Lampros Smyrnaios
4280f89296
- Set the default value of the "isTestEnvironment" property to "true", in order to avoid undesired outcomes in the production db.
...
- Code polishing.
2023-03-21 17:04:28 +02:00
Lampros Smyrnaios
e975bec911
- Add Prometheus and Grafana which help measuring various metrics for the Controller's health and performance.
...
- Fix Docker config still using the old (now removed) "application.properties" file.
- Simplify the process of building and running the docker image; Now we use docker compose to run the Controller, along with the Prometheus and Grafana. Also, now it is not requested from the user to login and push the image (this may change in the future).
2023-03-21 16:46:33 +02:00
Lampros Smyrnaios
003c0bf179
- Add support for excluding specific datasources from being crawled. These datasources may be aggregated through bulk-imports, by other pieces of software. Such a datasource is "arXiv.org".
...
- Fix an issue, where the "datasource-type" was retrieved instead of the "datasource-name".
- Polish the "findAssignmentsQuery".
2023-03-21 07:19:35 +02:00
Lampros Smyrnaios
f835a752bf
Transform the "application.properties" file to "application.yml" and optimize the property-trees.
2023-03-20 15:23:00 +02:00
Lampros Smyrnaios