Lampros Smyrnaios
43ea64758d
- Improve handling of the case when no fulltexts have been found or none of the found ones were requested from the worker, as they were already retrieved in the past.
...
- Show the number of files with problematic locations (if any of them exist).
- Code polishing.
2024-02-23 12:39:28 +02:00
Lampros Smyrnaios
2e60128084
- Allow to easily change the por used by workers.
...
- Show the number of active background-tasks and bulkImportDirs, which delay the Service's shutdown.
- Update dependencies.
- Code polishing.
2023-12-19 23:31:42 +02:00
Lampros Smyrnaios
ede7ca5a89
- Add bulk-import support for non-Authoritative data-sources.
...
- Update Spring Boot.
- Code polishing.
2023-09-26 18:02:48 +03:00
Lampros Smyrnaios
b4f91f188e
Fix the "retries-num" appearing in log-messages.
2023-09-14 12:08:33 +03:00
Lampros Smyrnaios
44459c8681
- Rename "ImpalaConnector.java" to "DatabaseConnector.java".
...
- Update dependencies.
- Code polishing.
2023-08-23 16:55:23 +03:00
Lampros Smyrnaios
860c73ea91
- Improve the "shutdownController.sh" script.
...
- Set names for the Prometheus and Grafana containers.
- Code polishing.
2023-07-27 18:27:48 +03:00
Lampros Smyrnaios
cde6063d63
- Update dependencies.
...
- Code polishing.
2023-07-25 12:12:56 +03:00
Lampros Smyrnaios
b73be6d8da
Fix the Stats API returning simple numbers as "application/json". Now they are returned as "text/plain".
2023-07-25 12:03:27 +03:00
Lampros Smyrnaios
d5c139c410
Handle the case where the "stats"-queries are executed while some table of the DB are in a "merge" state. In this case, the queries fail and the Controller retries up to 10 times.
2023-07-06 18:29:13 +03:00
Lampros Smyrnaios
0f4b63c4a9
Expose the following statistics as prometheus-metrics and create/update a stats-endpoint for each one:
...
- "numOfPayloadsAggregatedByServiceThroughCrawling"
- "numOfPayloadsAggregatedByServiceThroughBulkImport"
- "numOfPayloadsAggregatedByService"
- "numOfLegacyPayloads"
- "numOfRecordsInspectedByServiceThroughCrawling" (renamed from "numOfInspectedRecords")
2023-06-23 15:22:26 +03:00
Lampros Smyrnaios
b9712bed85
- Expose the "numOfAllPayloads" and "numOfInspectedRecords" DB-stats to Prometheus, by using a scheduling task to request the numbers from the DB, every 6 hours.
...
- Update the "StatsServiceImpl.getNumberOfPayloadsAggregatedByService()" to use the new table "payload_aggregated", instead of casting and checking the date of the records.
- Code polishing.
2023-06-19 14:42:00 +03:00
Lampros Smyrnaios
42b93e9429
- Add the "getNumberOfAllDistinctFullTexts" stats-endpoint.
...
- Add TODOs for more stats endpoints.
- Code polishing.
2023-05-04 15:48:49 +03:00
Lampros Smyrnaios
ff13af7abb
Use a StatsService interface.
2023-03-13 12:39:39 +02:00
Lampros Smyrnaios