lsmyrnaios
/
UrlsController
1
0
Fork 1
Code Issues Pull Requests Releases 13 Wiki Activity
246 Commits 2 Branches 13 Tags 1.1 MiB
Commit Graph

121 Commits

Author SHA1 Message Date
Lampros Smyrnaios 7f789b8ad0 - If we receive an "UnknownHostException" when uploading to the S3ObjectStore, then skip the current full-texts' batch to leave some time for the network to get unstuck. 
- Code polishing.
 2023-11-22 15:29:18 +02:00
Lampros Smyrnaios 856c62887d - Make sure the "UTF_8" charset is used, when we get a message from the response-body. 
- Improve some log-messages.
 2023-10-26 11:44:23 +03:00
Lampros Smyrnaios 0c7bf6357b - Improve performance in "FileUtils.addUrlReportsByMatchingRecordsFromBacklog()". 
- Make sure we remove the assignments of all "not-successful", old, worker-reports, even for the ones which failed to be renamed to indicate success or failure, or failed to be executed by the background threads (and thus never reached the renaming stage).
 2023-10-23 12:21:42 +03:00
Lampros Smyrnaios a7581335f1 - Improve the "getDataForPayloadPrefillQuery". 
- Improve some error-messages.
 2023-10-21 11:31:31 +03:00
Lampros Smyrnaios df0ea62a5a - Handle the case when the "webHDFSBaseUrl" does not use HTTPS. 
- Improve error-reporting when uploading a file to HDFS.
 2023-10-19 11:59:37 +03:00
Lampros Smyrnaios 40729c6295 Move similar code into the new "ParquetFileUtils.getPayloadParquetRecord()" method. 2023-10-17 12:50:51 +03:00
Lampros Smyrnaios f05eee7569 Improve the names of some methods. 2023-10-16 23:39:43 +03:00
Lampros Smyrnaios a354da763d - Improve some log-messages. 
- Increase app's version.
- Code polishing.
 2023-10-06 17:28:54 +03:00
Lampros Smyrnaios 96c11ba4b8 - Add a missing change. 
- Code optimization and polishing.
- Update dependencies.
 2023-10-04 16:17:12 +03:00
Lampros Smyrnaios 7019f7c3c7 Improve aggregation speed, by generating additional "attempt" and "payload" records for the publications which are in the back-log and their url matches to one of the urls of the current payloads. 2023-10-04 15:43:31 +03:00
Lampros Smyrnaios c9626de120 Handle the case when the "upload-file-to-S3" operation fails with a "ConnectException". In this case, all remaining upload operations for the files of that particular batch or segment, are canceled. 2023-10-04 13:01:13 +03:00
Lampros Smyrnaios 865926fbc3 - Handle the case when some results have been found from the "getAssignmentsQuery", but no data could be extracted from them. 
- Code polishing.
 2023-10-02 15:46:55 +03:00
Lampros Smyrnaios ede7ca5a89 - Add bulk-import support for non-Authoritative data-sources. 
- Update Spring Boot.
- Code polishing.
 2023-09-26 18:02:48 +03:00
Lampros Smyrnaios 0f5d4dac78 Check and show warning/error message for failed payloads. 2023-09-20 17:38:22 +03:00
Lampros Smyrnaios 903c3e1ffc Add thread-safety when reading the bulkImportReport-files. 2023-09-15 11:54:32 +03:00
Lampros Smyrnaios 360731ba72 - Improve handling of the "NO_CONTENT" case, in "getAssignments"-endpoint. 
- Code optimization and polishing.
 2023-09-14 13:53:01 +03:00
Lampros Smyrnaios 6891c467d4 - Avoid displaying a warning for the "test" HDFS directory, when the Controller is running in PROD mode. 
- Add a missing change for the optimization of reading files.
- Update dependencies.
 2023-09-13 15:29:30 +03:00
Lampros Smyrnaios ee2df19ce1 - Allow "pretty-printing" the json response of the "getBulkImportReport" endpoint. 
- Add useful log-messages for various bulk-import stages and improve the current ones.
- Optimize reading and writing the reports.
 2023-09-11 17:24:39 +03:00
Lampros Smyrnaios 1c8f3765ca - Fix not acquiring the full workerReport when retrying it, with the scheduler. 
- Improve error-handling in the "inspectWorkerReportsAndTakeAction" process.
- Code polishing.
 2023-09-08 14:59:48 +03:00
Lampros Smyrnaios 4014d1eabb Code polishing. 2023-09-05 15:20:03 +03:00
Lampros Smyrnaios 98516498eb - Increase app's version. 
- Code polishing.
 2023-09-04 12:46:55 +03:00
Lampros Smyrnaios febe2b212c Upgrade management of failed workerReports: 
- Upon completing processing a workerReport, the name of the json-file will be appended with "successful" or "failed".
- Avoid deleting immediately the failed workerReports.
- Add a scheduling task to process leftover failed workerReports from previous executions of the service, only once, 12 hours after startup, in order for the workers to have participated and filled the "workersInfoMap".
- Add a scheduling task to process leftover failed workerReports from the current execution, regularly.
- Fix not iterating through the workers' subDirs when checking the last-access-time of workerReports.
- Fix not deleting the assignment records from the DB, when a failed leftover workerReport gets deleted.
- Code refactoring.
 2023-09-01 15:10:58 +03:00
Lampros Smyrnaios 601776e81c - Fix not handling for the case where the info about the worker in the WorkerReport, does not exist inside the "workersInfoMap", as that worker is not participating in the Service. (this case may appear in future code) 
- Code polishing.
 2023-08-30 17:07:51 +03:00
Lampros Smyrnaios aa3f32f3da - Make sure the given number of threads, given by the user is above zero. 
- Adjust the number and size of log files.
- Update Spring Boot.
- Code polishing.
 2023-08-30 14:02:54 +03:00
Lampros Smyrnaios 44459c8681 - Rename "ImpalaConnector.java" to "DatabaseConnector.java". 
- Update dependencies.
- Code polishing.
 2023-08-23 16:55:23 +03:00
Lampros Smyrnaios 66a5b3c7da Update Bulk-Import API: 
- Increase the "numOfThreadsPerBulkImportProcedure" to 6.
- Fix Bulk import not working from a second-level subdirectory; the report-subDirectory was not created.
- Fix not returning the bulk-import-report as "application/json".
- Add useful messages for missing parameters.
- Change the HTTP-method for the "bulkImportFullTexts" endpoint to "POST".
- Show a structured json-response for the "bulkImportFullTexts" endpoint.
- Fix uncommon date-format.
- Remove single quotes from json-report, since they are returned as bytes, not characters.
- Optimize the generation of the json-bulkImport-report.
 2023-07-25 11:59:47 +03:00
Lampros Smyrnaios fd1cf56863 - Avoid passing some duplicate publications to the Workers, by post-processing the assignments retrieved from Impala. The assignments may contain duplicate id-url pairs, which have different datasources, since one publication may be connected to multiple datasources. 
- Code polishing.
 2023-07-19 18:31:24 +03:00
Lampros Smyrnaios b94c35c66e - Fix double active "@Scheduled" annotation for the "ScheduledTasks.updatePrometheusMetrics()" method. 
- Code polishing.
 2023-07-13 18:32:45 +03:00
Lampros Smyrnaios 8dfb58ee63 Avoid assigning the same publications multiple times to the Workers, after the recent "parallelization enchantment". 
After that enchantment, each worker could request multiple assignment-batches, before its previous batches were processed by the Controller. This means that for each batch that was processed, the Controller was deleting from the "assignment" table, all the assignments (-batches) delivered to the Worker that brought that batch, even though the "attempt" and "payload" records for the rest of the batches were not inserted in the DB yet. So in a new assignments-batch request, the same publications that were already under processing, were delivered to the same or other Workers.
Now, for each finished batch, only the assignments of that batch are deleted from the "assignment" table.
 2023-07-11 17:27:23 +03:00
Lampros Smyrnaios 2d5643cb0a Fix missing spaces in some secondary "DROP"-queries. 2023-07-07 20:51:14 +03:00
Lampros Smyrnaios e8644cb64f - Optimize the "insertAssignmentsQuery". 
- Add documentation about the Prometheus Metrics, in README.
- Update Dependencies.
- Code polishing.
 2023-07-05 17:10:30 +03:00
Lampros Smyrnaios 4c3e2e6b6e - Fix not using actual the "currentAssignmentsBatch" of the workerReport itself, when creating the parquetFileNames and when reporting to the user the initialization of the "addition of the workerReport". 
- Code polishing.
 2023-06-27 16:08:01 +03:00
Lampros Smyrnaios 798fa09d68 - Identify and handle a possible Worker-crash, in "UrlsServiceImpl.postReportResultToWorker()". 
- Add/Improve some log messages.
- Update and cleanup dependencies.
- Code polishing.
 2023-06-15 23:19:36 +03:00
Lampros Smyrnaios c37f157f51 Split the full-texts-batch's main handling-code to two separate methods, which can be used in parallel by two threads, in the future. 2023-06-14 17:16:38 +03:00
Lampros Smyrnaios 0b1ab5b991 Attempt to recover from serious failures, by individualizing the error-handling for each of the "table-merging" queries. 2023-06-10 02:28:02 +03:00
Lampros Smyrnaios 6669dc61bf - Increase the initialDelay for the "checkIfServiceIsReadyForShutdown" scheduled-task, in production, to 10 minutes. 
- Code polishing.
 2023-06-06 16:49:53 +03:00
Lampros Smyrnaios 3988eb3a48 - Use a separate HDFS sub-dir for every assignments-batch, in order to avoid any disrruptancies from multiple threads moving parquet-files from the same sub-dir. Multiple batches from the same worker may be processed at the same time. These sub-dirs are deleted afterwards. 
- Treat the "contains no visible files" situation as an error. In which case the assignments-data is presumed to not have been inserted to the database tables.
- Code polishing/cleanup.
 2023-05-27 02:36:05 +03:00
Lampros Smyrnaios 2b50e08bf6 - Handle the case, were multiple threads may load the same HDFS directory to a database table, thus causing the "directory contains no visible files"-SQLException. 
- Improve the values of the delays for some scheduledTasks.
- Improve elapsed time precision for the "lastAccessedOn" metadata of the workerReports.
- Code polishing.
 2023-05-25 00:34:36 +03:00
Lampros Smyrnaios cd1fb0af88 - Process the WorkerReports in background Jobs and post the reportResults to the Workers. 
- Save the workerReports to json files, until they are processed successfully.
- Show some custom metrics in prometheus.
 2023-05-24 13:52:28 +03:00
Lampros Smyrnaios c7bfd75973 - Add the "getWorkersInfo" endpoint. 
- Improve startup speed, by using a faster remote server to get the host's machine public IP. This also reduces the risk of not being able to get the public IP at all.
- Fix the detection of a different IP for a known worker.
- Improve documentation.
 2023-05-23 14:57:15 +03:00
Lampros Smyrnaios 5f75b48e95 - Increase the "read-timeout" when searching for the host's machine public-IP. 
- Update dependencies.
- Code polishing.
 2023-05-22 21:33:02 +03:00
Lampros Smyrnaios 0ab6bae93a - Optimize the json-conversion of the "BulkImportReport". 
- Code polishing.
 2023-05-18 17:30:40 +03:00
Lampros Smyrnaios f51a34138f - Use separate HDFS subdirectories for each worker in order to avoid seeing exceptions about "empty hdfs directory" when "loading" data to the database, because one worker has loaded data generated by multiple workers (since we use only 1 load operation for multiple parquet files). 
- Store each worker's info in a hash-table, in order to efficiently know if we need to create new hdfs subdirectories. Also, this will help to issue "shutdown" requests to the workers in the future, as well as to know which worker has shutdown.
 2023-05-15 13:12:20 +03:00
Lampros Smyrnaios 9412391903 - In test-environment mode, check for already existing file-hashes only in the "payload_aggregated" table, instead of the whole "payload" view. This way the investigation for false-positive docUrls is easier, as we avoid checking against the millions of "legacy" payloads. 
- Improve performance in production, by not creating the string objects for "trace"-logs.
 2023-05-15 12:44:16 +03:00
Lampros Smyrnaios 8381df70c6 - Improve performance of uploading parquet-files to HDFS. 
- Add some logs.
- Code polishing.
 2023-05-11 19:40:48 +03:00
Lampros Smyrnaios 992d4ffd5e - Add the time-zone in the logs. 
- Change some log-levels to "trace", although most of them are still disabled.
 2023-05-11 03:10:53 +03:00
Lampros Smyrnaios b6e8cd1889 New feature: BulkImport full-text files from compatible datasources. 2023-05-11 03:07:55 +03:00
Lampros Smyrnaios b3196376eb Fix a bug, which caused the full-text files to never close. 2023-05-04 13:03:28 +03:00
Lampros Smyrnaios fd15372fd6 Add error-checks for retrieving the status-code from HttpUrlConnections. 2023-05-03 13:30:29 +03:00
Lampros Smyrnaios 49662319a1 - Simplify the creation of local directories. 
- Improve exception messages.
 2023-04-28 14:58:33 +03:00