michal.politowski
/
UrlsController
forked from lsmyrnaios/UrlsController
1
0
Fork 0
Code Pull Requests Activity
292 Commits 3 Branches 14 Tags 1.1 MiB
Commit Graph

292 Commits

Author SHA1 Message Date
Lampros Smyrnaios 3417e5c68c Add handling for "null" values in "BulkImport.BulkImportSource". 
When setting an "application.yml" property to "null", then its value is registered as "empty". So, specifically for "BulkImport", where we do not want empty values, only nulls or valid values, convert the empty values to null, upon initialization.
 2024-06-14 12:32:39 +03:00
Lampros Smyrnaios fc258e2e26 - Rename the "pdfUrlPrefix" config-field to "fulltextUrlPrefix", as it may point to different file-formats in the future. 
- Code polishing.
 2024-06-07 13:21:27 +03:00
Lampros Smyrnaios ed7bf09f9b - Replace all "json" usages, with "gson" ones, in order to remove the "org.json:json" dependency. 
- Add an extra check to verify that the remote parquet directories are directories indeed.
- Set new version.
 2024-06-06 14:40:39 +03:00
Lampros Smyrnaios 9610b77b2b Update "gradlew" script. 2024-06-03 13:15:13 +03:00
Lampros Smyrnaios 643e497826 - Fix a regression, in which the BulkImportReport was not returning the events in a time-ordered state. 
- Update dependencies.
 2024-06-03 13:07:17 +03:00
Lampros Smyrnaios a48f07f41d - Increase the "-Xmx" java argument to 6Gb. 
- Set the version to 2.7.3.
 2024-05-31 21:43:56 +03:00
Lampros Smyrnaios 2241a89452 - Limit the depth of subdirectories to process in BulkImport. 
- Code polishing.
 2024-05-31 21:36:03 +03:00
Lampros Smyrnaios edf064616a Improve error-handling in "BulkImportReport.getJsonReport()" and "FileUtils.writeToFile()". 2024-05-30 11:52:04 +03:00
Lampros Smyrnaios d7697ef3f8 - Tighten the thread-safety protection on the "BulkImportReport.getJsonReport()" method. 
- Update dependencies.
- Code polishing.
 2024-05-27 10:40:05 +03:00
Lampros Smyrnaios b6ad2af48b - Reduce occupying space at any given time, by deleting the archives right after decompression and files-extraction. 
- Code refactoring.
 2024-05-22 12:11:39 +03:00
Lampros Smyrnaios e2e7ca72d5 - Fix not allowing the user to use the "shutdownAllWorkersGracefully" endpoint twice. 
- Code optimization.
- Update dependencies.
 2024-05-21 23:43:49 +03:00
Lampros Smyrnaios 39c36f9e66 - Resolve a concurrency issue, by enforcing synchronization on the "BulkImportReport.getJsonReport()" method. 
- Increase the number of stacktrace-lines to 20, for bulkImport-segment-failures.
- Improve "GenericUtils.getSelectiveStackTrace()".
 2024-05-01 01:29:25 +03:00
Lampros Smyrnaios 8e14d4dbe0 - Fix not counting the files from the bulkImport-segment, which failed due to an exception. 
- Write segment-exception-messages to the bulkImport-report.
 2024-04-30 23:43:52 +03:00
Lampros Smyrnaios 0d117743c2 - Code-optimization. 
- Upload the updated gradle-wrapper and set using the latest Gradle version in "installAndRun.sh" script.
 2024-04-30 02:13:08 +03:00
Lampros Smyrnaios 64a1b7d4f0 - Comment-out the bucket-deletion process, in order to avoid any accidental deletion, even if it has to be explicitly allowed in the config. 
- Update dependencies.
 2024-04-26 12:54:00 +03:00
Lampros Smyrnaios e2d43a9af0 Upgrade the "processBulkImportedFilesSegment" code: 
1) Pre-calculate the file-hashes for all files of the segment and perform a single "getHashLocationsQuery", instead of thousands
2) Write some important events to the bulkImportReport file, as soon as they are added in the list.
 2024-04-02 14:35:19 +03:00
Lampros Smyrnaios bd323ad69a Avoid a very rare case, where we might get an "IllegalArgumentException" from "Lists.partition()", in case the "sizeOfUrlReports" is <= 3. 2024-03-29 18:12:52 +02:00
Lampros Smyrnaios 08de530f03 Various improvements: 
- Handle the case when "fileUtils.constructS3FilenameAndUploadToS3()" returns "null", in "processBulkImportedFile()".
- Avoid an "IllegalArgumentException" in "Lists.partition()" when the number of files to bulkImport are fewer than the number of threads available to handle them.
- Include the last directory's "/" divider in the fileDIR group of "FILEPATH_ID_EXTENSION" regex (renamed from "FILENAME_ID_EXTENSION").
- Fix an incomplete log-message.
- Provide the "fileLocation" argument in the "DocFileData" constructor, in "processBulkImportedFile()", even though it's not used after.
 2024-03-29 17:23:01 +02:00
Lampros Smyrnaios 1d821ed803 - Prepare version for next release. 
- Fix typo of not using the "OpenAireID" in the S3 location of bulkImported files. Instead, the "fileNameID" was used, which in aggregation is the OpenAireID, but not in bulk-import.
- Update dependencies.
- Code polishing.
 2024-03-28 06:09:28 +02:00
Lampros Smyrnaios 8bc5cc35e2 - Optimize writing to the Bulk-import-report file. 
- Show the IP of the worker which posts a "workerShutdownReport".
- Code polishing.
 2024-03-22 17:50:55 +02:00
Lampros Smyrnaios b9b29dd51c Move some code from "FileUtils.getAndUploadFullTexts()" to two separate methods. 2024-03-20 16:53:03 +02:00
Lampros Smyrnaios 56d233d38e - Move the "FileUtils.mergeParquetFiles()" method to "ParquetFileUtils.mergeParquetFilesOfTable()". 
- Fix a typo.
 2024-03-20 15:25:19 +02:00
Lampros Smyrnaios 724eae1514 - Optimize the placement of "DatabaseConnector.databaseLock.unlock()" statements. 
- Rename a maven-repository.
 2024-03-20 15:08:01 +02:00
Lampros Smyrnaios 785204419d Update/cleanup the repositories in "build.gradle". 2024-03-15 12:22:13 +02:00
Lampros Smyrnaios 9b0818b535 - Add handling for additional/specific exceptions, when checking the "futures". 
- Move common "ExecutionException" handling-code into its own method: "GenericUtils.getSelectedStackTraceForCausedException()".
- Avoid a double log.
- Code polishing.
 2024-03-14 13:59:23 +02:00
Lampros Smyrnaios b34417dc45 Optimize the test-DB creation process: 
- Use views of the "initialDatabase" view and tables to a) reduce the amount of space used by test-DBs and b) improve test-db creation performance.
- Avoid possible failures from outdated metadata.
 2024-03-14 13:10:54 +02:00
Lampros Smyrnaios f61cae41a1 - Try to get the cause of the exception of the callable-tasks which handle the bulk-import of fileSegments. 
- Fix not counting the failedSegments when an exception was thrown.
- Code polishing.
 2024-03-13 12:15:59 +02:00
Lampros Smyrnaios 8f9786de09 Upgrade the algorithm for finding the previously-found fulltexts, based on their md5hash: 
- Use a single query with a list of the fileHashes, instead of thousands of singe-md5hash-check queries (run at most 6 in parallel) which require a lot of I/O.
- Avoid checking multiple times the same fileHash, in case it is related with multiple payloads.
- In case of a database-error, avoid completely losing the full-texts of that worker, instead, continue processing the full-texts.
 2024-03-13 11:28:37 +02:00
Lampros Smyrnaios e4540e7f3c Handle the case when a urlReports-sublist does not have any payloads inside. 2024-03-12 14:25:00 +02:00
Lampros Smyrnaios e20c5d2146 - Add error-handling for the case when no payloads could be associated with a specific url which should have been in the hashMultiMap in "addUrlReportsByMatchingRecordsFromBacklog". 
- Fix not cloning the payload, before changing it and adding it in the "prefilledPayloads"-list; instead, an object-reference was used.
 2024-03-11 19:48:04 +02:00
Lampros Smyrnaios 1048463ca0 - Improve error-handling in "S3ObjectStore.emptyBucket()". 
- Change some log-levels.
- Code polishing.
 2024-03-11 16:17:32 +02:00
Lampros Smyrnaios 8f18008001 Avoid performing payload-related operations in case no fulltext was received from the worker, due to en error. 2024-03-11 14:57:13 +02:00
Lampros Smyrnaios ce3e149a95 Improve the "emptying/deleting" process of the S3-bucket. 2024-03-11 13:34:38 +02:00
Lampros Smyrnaios dd394f18a0 - Optimize the JOIN-order in the "findAssignmentsQuery". 
- Optimize the "DOC_URL_FILTER"-regex.
- Update dependencies.
 2024-03-11 11:35:38 +02:00
Lampros Smyrnaios 43ea64758d - Improve handling of the case when no fulltexts have been found or none of the found ones were requested from the worker, as they were already retrieved in the past. 
- Show the number of files with problematic locations (if any of them exist).
- Code polishing.
 2024-02-23 12:39:28 +02:00
Lampros Smyrnaios 749172edd8 Add the Jenkins' build-status badge in README. 2024-02-08 19:49:58 +02:00
Lampros Smyrnaios b72996c9a9 - Configure the destination of the logs in the "application.properties" file. 
- Add some gradle files to be used by Jenkins.
 2024-02-08 19:47:34 +02:00
Lampros Smyrnaios 3563fd6e2a - Try to get the cause of the exception of the callable-tasks which handle parquet-files. 
- Update License.
- Update dependencies.
 2024-02-07 18:34:28 +02:00
Lampros Smyrnaios 34d7a143e7 Add/improve documentation. 2024-02-01 14:37:29 +02:00
Lampros Smyrnaios 5dadb8ad2f - Optimize the "DOC_URL_FILTER"-regex, by using a non-capturing group. 
- Remove an extra "File.separator" from the fulltexts-fullFilePath.
 2024-01-19 15:46:23 +02:00
Lampros Smyrnaios bdc61c2cda When at least one worker is still active and have to wait for service-shutdown, show a log-message to inform the user, including that worker's IP. 2024-01-15 13:35:22 +02:00
Lampros Smyrnaios 3a70b57146 Prioritize the full-text urls over the landing-page ones. 2024-01-15 12:59:50 +02:00
Lampros Smyrnaios ee1ca8966b - Avoid continuing to request workerReport-batches when from the 1st batch, the base-directory of that assignments-counter is not found. 
- Update dependencies.
 2024-01-15 12:57:33 +02:00
Lampros Smyrnaios 2e60128084 - Allow to easily change the por used by workers. 
- Show the number of active background-tasks and bulkImportDirs, which delay the Service's shutdown.
- Update dependencies.
- Code polishing.
 2023-12-19 23:31:42 +02:00
Lampros Smyrnaios d90ad51609 Add the "shutdownAllWorkersGracefully" and "cancelShutdownAllWorkersGracefully" endpoints, in order to be able to shut them down at once and update them, without shutting down the whole Service. So in this case the bulk-import procedures will continue to work. 2023-11-29 16:45:58 +02:00
Lampros Smyrnaios d20c9a7d2e - Show the original exception thrown by the background-job, not the one thrown in the main-thread, which is useless, except from its message. 
- Reduce the interval for deleting the unhandled assignments to once every 3 days.
- Set the upcoming version.
- Update dependencies.
 2023-11-27 18:19:53 +02:00
Lampros Smyrnaios 7f789b8ad0 - If we receive an "UnknownHostException" when uploading to the S3ObjectStore, then skip the current full-texts' batch to leave some time for the network to get unstuck. 
- Code polishing.
 2023-11-22 15:29:18 +02:00
Lampros Smyrnaios 9b1f2c4931 Improve performance and reduce memory usage of the "findAssignmentsQuery": 
- Reorder JOINs and predicates to reduce the computational cost.
- Remove the memory-costly "pu.url" predicates from the "where" clause, as the DB has no empty urls anymore.
 2023-10-31 15:59:48 +02:00
Lampros Smyrnaios db929d8931 - Add a scheduling job to delete assignments older than 7 days. These may be left behind when the worker throws a "SocketTimeoutException" before it can receive the assignments and process them. No workerReport gets created for those assignments. 
- Improve some log-messages.
- Code polishing.
 2023-10-30 12:29:54 +02:00
Lampros Smyrnaios 856c62887d - Make sure the "UTF_8" charset is used, when we get a message from the response-body. 
- Improve some log-messages.
 2023-10-26 11:44:23 +03:00