Lampros Smyrnaios
d52601e819
- Add handling for the "EmptyResultDataAccessException" in "UrlsServiceImpl.getAssignments()", which is thrown when no assignments are returned from the query.
...
- Code polishing.
2023-06-22 12:39:11 +03:00
Lampros Smyrnaios
b9712bed85
- Expose the "numOfAllPayloads" and "numOfInspectedRecords" DB-stats to Prometheus, by using a scheduling task to request the numbers from the DB, every 6 hours.
...
- Update the "StatsServiceImpl.getNumberOfPayloadsAggregatedByService()" to use the new table "payload_aggregated", instead of casting and checking the date of the records.
- Code polishing.
2023-06-19 14:42:00 +03:00
Lampros Smyrnaios
798fa09d68
- Identify and handle a possible Worker-crash, in "UrlsServiceImpl.postReportResultToWorker()".
...
- Add/Improve some log messages.
- Update and cleanup dependencies.
- Code polishing.
2023-06-15 23:19:36 +03:00
Lampros Smyrnaios
6669dc61bf
- Increase the initialDelay for the "checkIfServiceIsReadyForShutdown" scheduled-task, in production, to 10 minutes.
...
- Code polishing.
2023-06-06 16:49:53 +03:00
Lampros Smyrnaios
54685bbe9a
- Avoid sending "cancelShutdown" requests to already shutDown Workers.
...
- Optimize performance of the code running right before the "postShutdownOrCancelRequestToWorker".
- Show which Workers have already shutdown and as a result a "postShutdownOrCancelRequestToWorker" will not be performed on them.
2023-05-29 13:41:37 +03:00
Lampros Smyrnaios
f9c6bad768
Do not send shutDownRequests to workers which have already shutdown.
2023-05-29 12:42:54 +03:00
Lampros Smyrnaios
a38d6ace79
Code polishing.
2023-05-29 12:21:48 +03:00
Lampros Smyrnaios
74ff31fc64
- Show the workerIPs in the logs.
...
- Rename the "FullTexts"-files to "BulkImport".
2023-05-29 12:12:08 +03:00
Lampros Smyrnaios
3988eb3a48
- Use a separate HDFS sub-dir for every assignments-batch, in order to avoid any disrruptancies from multiple threads moving parquet-files from the same sub-dir. Multiple batches from the same worker may be processed at the same time. These sub-dirs are deleted afterwards.
...
- Treat the "contains no visible files" situation as an error. In which case the assignments-data is presumed to not have been inserted to the database tables.
- Code polishing/cleanup.
2023-05-27 02:36:05 +03:00
Lampros Smyrnaios
164245cb53
- Automatically delete the unsuccessful WorkerReports, which are more than 7 days old.
...
- Optimize the Service's startup speed, by setting "initialDelays" to the scheduled tasks.
- Optimize documentation.
2023-05-24 16:59:42 +03:00
Lampros Smyrnaios
cd1fb0af88
- Process the WorkerReports in background Jobs and post the reportResults to the Workers.
...
- Save the workerReports to json files, until they are processed successfully.
- Show some custom metrics in prometheus.
2023-05-24 13:52:28 +03:00
Lampros Smyrnaios
0ea3e2de24
Add the "shutdownService" and "cancelShutdownService" endpoints. The Controller sends the related requests to the Workers and shutdowns gracefully, after all workers have shutdown.
2023-05-24 13:42:29 +03:00
Lampros Smyrnaios
c7bfd75973
- Add the "getWorkersInfo" endpoint.
...
- Improve startup speed, by using a faster remote server to get the host's machine public IP. This also reduces the risk of not being able to get the public IP at all.
- Fix the detection of a different IP for a known worker.
- Improve documentation.
2023-05-23 14:57:15 +03:00
Lampros Smyrnaios
5f75b48e95
- Increase the "read-timeout" when searching for the host's machine public-IP.
...
- Update dependencies.
- Code polishing.
2023-05-22 21:33:02 +03:00
Lampros Smyrnaios
0ab6bae93a
- Optimize the json-conversion of the "BulkImportReport".
...
- Code polishing.
2023-05-18 17:30:40 +03:00
Lampros Smyrnaios
f7f919cee1
- Make sure we set the "hasShutdown" to "false", for each known worker which was restarted.
...
- Fix markdown of urls in prometheus' readme.
2023-05-16 12:24:14 +03:00
Lampros Smyrnaios
f51a34138f
- Use separate HDFS subdirectories for each worker in order to avoid seeing exceptions about "empty hdfs directory" when "loading" data to the database, because one worker has loaded data generated by multiple workers (since we use only 1 load operation for multiple parquet files).
...
- Store each worker's info in a hash-table, in order to efficiently know if we need to create new hdfs subdirectories. Also, this will help to issue "shutdown" requests to the workers in the future, as well as to know which worker has shutdown.
2023-05-15 13:12:20 +03:00
Lampros Smyrnaios
9412391903
- In test-environment mode, check for already existing file-hashes only in the "payload_aggregated" table, instead of the whole "payload" view. This way the investigation for false-positive docUrls is easier, as we avoid checking against the millions of "legacy" payloads.
...
- Improve performance in production, by not creating the string objects for "trace"-logs.
2023-05-15 12:44:16 +03:00
Lampros Smyrnaios
b6e8cd1889
New feature: BulkImport full-text files from compatible datasources.
2023-05-11 03:07:55 +03:00
Lampros Smyrnaios
42b93e9429
- Add the "getNumberOfAllDistinctFullTexts" stats-endpoint.
...
- Add TODOs for more stats endpoints.
- Code polishing.
2023-05-04 15:48:49 +03:00
Lampros Smyrnaios
d7797eaaf6
Add the "getNumberOfPayloadsForDatasource" endpoint.
2023-04-24 09:54:35 +03:00
Lampros Smyrnaios
ff13af7abb
Use a StatsService interface.
2023-03-13 12:39:39 +02:00
Lampros Smyrnaios
38643c76a3
- Code polishing.
...
- Update Gradle.
2023-03-07 16:55:41 +02:00
Lampros Smyrnaios
8893662a81
Refactor the UrlsController: a) offload the business-logic to the dedicated "UrlsService" and b) move the "checkParquetFilesSuccess()"-method to "ParquetFileUtils".
2023-02-21 15:36:35 +02:00
Lampros Smyrnaios
a1c16ffc19
- Exclude empty and null urls in the assignments.
...
- Update the "getFullTextsImproved"-call to "getFullTexts", now that the "improved" version is stable.
- Update Gradle.
- Code polishing.
2023-02-16 14:24:47 +02:00
Lampros Smyrnaios
2253f05bf5
Refactor the "StatsController"-code, by offloading it to a dedicated "StatsService".
2023-02-09 19:25:48 +02:00
Lampros Smyrnaios
49fefefafd
- Refactor the payloads-statistics-code and provide two endpoints: "getNumberOfPayloadsAggregatedByService", which returns the number of payloads aggregated only by the PDF-Aggregation-Service, and the "getNumberOfAllPayloads", which returns the number of all payloads existing in the database, even the ones aggregated in the past, by other pieces of software.
...
- Update README.md.
- Make sure the docker image is clean-built, by avoiding the use of cache.
2023-02-02 17:58:47 +02:00
Lampros Smyrnaios
c9f33d3afa
Add an extra precaution-check to allow the emptying or deletion of an S3-Object-Store bucket, only when the app runs in "TestEnvironment".
2023-02-01 16:42:22 +02:00
Lampros Smyrnaios
8876089022
- Use Facebook's [**Zstandard**]( https://facebook.github.io/zstd/ ) compression algorithm, which brings very big benefits on compression rate and speed.
...
- Update the minIO dependency.
- Code polishing.
2023-01-10 13:34:54 +02:00
Lampros Smyrnaios
9cdbbdea67
Refactor the files' storage location.
2022-12-15 18:29:51 +02:00
Lampros Smyrnaios
e51ee9dd27
- Add info about the Stats API usage in "README.md".
...
- Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()".
- Handle the "EmptyResultDataAccessException" inside "StatsController".
- Optimize gradle's performance.
- Code polishing.
2022-12-15 14:04:22 +02:00
Lampros Smyrnaios
bfdf06bd09
- Apply error-checking on individual CallableTasks and in tasks-batches related to the creation and upload of all the data related to the "attempt" and "payload" table. So, if no data could be uploaded for one or both tables, no "load"-queries will be executed for that/those tables.
...
- Catch the more general "Exception", inside "FileUtils.mergeParquetFiles()", in order to be certain that the "SQLException" can also be caught.
- Code polishing.
2022-12-09 12:46:06 +02:00
Lampros Smyrnaios
0209d24068
- Change the parquet compression from "Snappy" to "Gzip", as there is an unhandleable exception when the app is running inside a Docker Container and uses the "Snappy" compression.
...
- Code polishing.
2022-12-08 16:28:41 +02:00
Lampros Smyrnaios
8607594f6d
- Improve exception handling.
...
- Code polishing.
2022-12-07 13:48:00 +02:00
Lampros Smyrnaios
b0c57d79a5
- When the Controller cannot retrieve any assignments from Impala (without an error), return an HTTP-"MULTI_STATUS" with an empty "AssignmentsResponse", instead of an "INTERNAL_SERVER_ERROR".
...
- Fix an error-message.
2022-12-05 16:44:00 +02:00
Lampros Smyrnaios
6226e2298d
- Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement.
...
One side effect of using the parquet-files, is that the timestamps are now BIGDECIMAL numbers, instead of "Timestamp" objects, but, converting them to such objects is pretty easy, if we ever need to do it.
- Code polishing.
2022-11-10 17:18:21 +02:00
Lampros Smyrnaios
aad37cd81e
Add the "StatsController", which brings the "getNumberOfPayloads" and "getNumberOfRecordsInspected" endpoints.
2022-10-18 15:00:26 +03:00
Lampros Smyrnaios
b6340066a7
- Improve handling of the case, where the full-texts were found, but the Controller could not acquire them from the Worker.
...
- Add/improve logs and comments.
- Code cleanup.
2022-09-28 22:34:33 +03:00
Lampros Smyrnaios
a22144bd51
- Refactor "FileUtils.getErrorMessageFromResponseBody(conn)" into "FileUtils.getMessageFromResponseBody(conn, isError)", in order to be able to either retrieve the "normal" or the "error" response.
...
- Add comments.
2022-09-15 23:12:05 +03:00
Lampros Smyrnaios
a81ed3c60f
- Add an "isTestEnvironment"-switch, which makes it easier to work with production and test databases.
...
- In case the Worker cannot be reached during a full-texts' batch request, abort the rest of the batches.
- Fix memory leaks when unzipping the batch-zip-file.
- Add explanatory comments for picking the database related to a full-text file.
2022-04-08 17:39:45 +03:00
Lampros Smyrnaios
a23c918a42
- Fix a "@JsonProperty" annotation inside "Payload.java".
...
- Fix a "@Value" annotation inside "FileUtils.java".
- Add a new database and show its name along with the initial's name in the logs.
- Code cleanup and improvement.
2022-04-05 00:01:44 +03:00
Lampros Smyrnaios
ad5dbdde9b
- Improve performance when inserting records into the "attempt" table, by splitting the records equally, across more threads.
...
- Bring back the "UriBuilder", which informs us in the logs, about the Controller's url (IP, PORT, API).
- Code cleanup.
2022-02-22 13:54:16 +02:00
Lampros Smyrnaios
dfd40cb105
Insert only the records with uploaded-to-S3 full-texts, in the "payload" table.
2022-02-17 16:27:40 +02:00
Lampros Smyrnaios
71f6b46130
- In case of an error when creating the "current_assignment" table (e.g out of memory in the backend database server), check for partial-creation and drop it. Also, in any case, before we drop this table, now check if it exists firsts (in general it should always exist, unless the creation results in an error and the table was not created at all).
...
- Fix an error-message.
- Update dependencies.
- Code cleanup.
2022-02-14 12:36:00 +02:00
Lampros Smyrnaios
d2ed9cd9ed
Improve efficiency and performance when processing the full-texts.
2022-02-08 15:02:13 +02:00
Lampros Smyrnaios
1111c850b9
- Add support for more than one full-text per id. Allow recognizing fileName additions: "id(1).pdf", "id(2).pdf", etc.
...
- Fix not giving the databaseName in the "ImpalaController.get10PublicationIdsTest()".
- Improve consistency in the "maxAttemptsPerRecord" value, among different threads. Also, reduce the value-increase by one.
- Check if the tableName string is empty, in the "mergeParquetFiles".
- Improve error-logging.
- Set some local variables to "final", optimizing code-execution by the JVM.
2022-02-07 13:57:09 +02:00
Lampros Smyrnaios
6aab1d242b
- Improve performance when handling WorkerReports' database insertions, by using parallelism to insert to two different tables in the same time. Also, pre-cache the query-argument-types.
...
- Update the error-message and counting system, on partial insertion event.
2022-02-04 14:48:22 +02:00
Lampros Smyrnaios
be4898e43e
Bug fixes and improvements:
...
- Fix an NPE, when the "getTestUrls"-endpoint is called. It was thrown because of an absent per-thread initialization of some thread-local variables.
- Fix JdbcTemplate error when querying the "getFileLocationForHashQuery".
- Fix the "S3ObjectStore.isLocationInStore" check.
- Fix not catching/handling some exceptions.
- Fix/improve log-messages.
- Optimize the "getFileLocationForHashQuery" to return only the first row. In the latest change, without this optimization, the query-result would cause non-handling the same-hash cases, because of an exception.
- Optimize the "ImpalaConnector.databaseLock.lock()" positioning.
- Update the "getTestUrls" api-path.
- Optimize list-allocation.
- Re-add the info-message about the successful emptying of the S3-bucket.
- Code cleanup.
2022-02-02 20:19:46 +02:00
Antonis Lempesis
35966b6f6e
finishing toucehs
2022-02-01 16:57:28 +02:00
Antonis Lempesis
e9bede5c45
more fixes
2022-02-01 02:08:02 +02:00