lsmyrnaios
/
UrlsController
1
0
Fork 1
Code Issues Pull Requests Releases 13 Wiki Activity
93 Commits 2 Branches 13 Tags 1.1 MiB
Commit Graph

73 Commits

Author SHA1 Message Date
Lampros Smyrnaios e51ee9dd27 - Add info about the Stats API usage in "README.md". 
- Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()".
- Handle the "EmptyResultDataAccessException" inside "StatsController".
- Optimize gradle's performance.
- Code polishing.
 2022-12-15 14:04:22 +02:00
Lampros Smyrnaios bfdf06bd09 - Apply error-checking on individual CallableTasks and in tasks-batches related to the creation and upload of all the data related to the "attempt" and "payload" table. So, if no data could be uploaded for one or both tables, no "load"-queries will be executed for that/those tables. 
- Catch the more general "Exception", inside "FileUtils.mergeParquetFiles()", in order to be certain that the "SQLException" can also be caught.
- Code polishing.
 2022-12-09 12:46:06 +02:00
Lampros Smyrnaios 0209d24068 - Change the parquet compression from "Snappy" to "Gzip", as there is an unhandleable exception when the app is running inside a Docker Container and uses the "Snappy" compression. 
- Code polishing.
 2022-12-08 16:28:41 +02:00
Lampros Smyrnaios c8baf5a5fc - Fix not finding the parquet-schema files when the app was run inside a Docker Container. 
- Update the "namespaces" and the "names" inside the parquet schemas.
- Code polishing.
 2022-12-08 12:16:05 +02:00
Lampros Smyrnaios 95c38c4a24 - Fix creating the "assignment" table, always in the testDatabase. 
- Code polishing.
 2022-12-07 14:58:38 +02:00
Lampros Smyrnaios 3c5f4c6464 Fix bytes to MB conversion. 2022-12-07 14:32:18 +02:00
Lampros Smyrnaios 8607594f6d - Improve exception handling. 
- Code polishing.
 2022-12-07 13:48:00 +02:00
Lampros Smyrnaios f183df276b - Move the "uploadFullTexts"-code in its own method. 
- Code polishing.
 2022-12-06 12:24:34 +02:00
Lampros Smyrnaios b0c57d79a5 - When the Controller cannot retrieve any assignments from Impala (without an error), return an HTTP-"MULTI_STATUS" with an empty "AssignmentsResponse", instead of an "INTERNAL_SERVER_ERROR". 
- Fix an error-message.
 2022-12-05 16:44:00 +02:00
Lampros Smyrnaios 577ea983e8 - Improve some log-messages. 
- Set some optimization settings for gradle.
- Fix error-handling in "installAndRun.sh".
- Update dependencies.
 2022-11-30 16:28:39 +02:00
Lampros Smyrnaios 6226e2298d - Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement. 
One side effect of using the parquet-files, is that the timestamps are now BIGDECIMAL numbers, instead of "Timestamp" objects, but, converting them to such objects is pretty easy, if we ever need to do it.
- Code polishing.
 2022-11-10 17:18:21 +02:00
Lampros Smyrnaios aad37cd81e Add the "StatsController", which brings the "getNumberOfPayloads" and "getNumberOfRecordsInspected" endpoints. 2022-10-18 15:00:26 +03:00
Lampros Smyrnaios e2d53105d1 Fix not creating the "assignment" table in a new production database, which contains only the "publication" and "datasource" data. 2022-10-07 15:51:31 +03:00
Lampros Smyrnaios b6340066a7 - Improve handling of the case, where the full-texts were found, but the Controller could not acquire them from the Worker. 
- Add/improve logs and comments.
- Code cleanup.
 2022-09-28 22:34:33 +03:00
Lampros Smyrnaios a22144bd51 - Refactor "FileUtils.getErrorMessageFromResponseBody(conn)" into "FileUtils.getMessageFromResponseBody(conn, isError)", in order to be able to either retrieve the "normal" or the "error" response. 
- Add comments.
 2022-09-15 23:12:05 +03:00
Lampros Smyrnaios 3e8f9c6074 Update the "UriBuilder.java" to be able to acquire the running port of the server, in case the port-number was initially set to "random" (0). Also make sure we get the "localHostAddress" and not the "localHostName", in case the public IP is not retrievable. 2022-09-12 17:04:05 +03:00
Lampros Smyrnaios a2cd02115f - Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated. 
- Update dependencies.
 2022-06-27 21:41:32 +03:00
Lampros Smyrnaios e3b374a32f - Optimize file-related tasks. 
- Update dependencies.
- Code cleanup.
 2022-05-26 15:43:59 +03:00
Lampros Smyrnaios 9b95eebb6c - Remove the obsolete "parenthesis" and "increasing duplicate-num" from the full-texts' names, before sending them to the S3-Object-Store. They now end with the "file-hash", so it is guaranteed that they will be unique. The Worker continues to produce the previous kind of names, without any disturbance. 
- Improve logging.
- Update MinIO dependency.
 2022-04-11 21:15:22 +03:00
Lampros Smyrnaios a81ed3c60f - Add an "isTestEnvironment"-switch, which makes it easier to work with production and test databases. 
- In case the Worker cannot be reached during a full-texts' batch request, abort the rest of the batches.
- Fix memory leaks when unzipping the batch-zip-file.
- Add explanatory comments for picking the database related to a full-text file.
 2022-04-08 17:39:45 +03:00
Lampros Smyrnaios 33fc61a8d9 - Fix the fileName-ID not being directly related with the datasourceID, in the S3-ObjectStore name. Add explanatory comments. 
- Add missing error-logs.
 2022-04-05 16:22:02 +03:00
Lampros Smyrnaios a23c918a42 - Fix a "@JsonProperty" annotation inside "Payload.java". 
- Fix a "@Value" annotation inside "FileUtils.java".
- Add a new database and show its name along with the initial's name in the logs.
- Code cleanup and improvement.
 2022-04-05 00:01:44 +03:00
Lampros Smyrnaios 5e4fad2479 - Change the fileNames' structure in the S3-ObjectStore. 
- Update dependencies.
 2022-04-01 19:24:04 +03:00
Lampros Smyrnaios 48670f3399 - Show the percentage of the "NumFullTextsFound", in the logs. 
- Update dependencies.
 2022-03-28 14:29:31 +03:00
Lampros Smyrnaios 88acaae20f - Replace the "numFullTextUrlsFound"-counter with "numFullTextsFound"-counter to reflect the end result of the actually available full-texts (which were downloaded by the Worker). 
- Optimize the gather-fileNames loop.
- Improve a message in "installAndRun.sh"
 2022-02-23 17:40:06 +02:00
Lampros Smyrnaios ad5dbdde9b - Improve performance when inserting records into the "attempt" table, by splitting the records equally, across more threads. 
- Bring back the "UriBuilder", which informs us in the logs, about the Controller's url (IP, PORT, API).
- Code cleanup.
 2022-02-22 13:54:16 +02:00
Lampros Smyrnaios dfd40cb105 Insert only the records with uploaded-to-S3 full-texts, in the "payload" table. 2022-02-17 16:27:40 +02:00
Lampros Smyrnaios 71f6b46130 - In case of an error when creating the "current_assignment" table (e.g out of memory in the backend database server), check for partial-creation and drop it. Also, in any case, before we drop this table, now check if it exists firsts (in general it should always exist, unless the creation results in an error and the table was not created at all). 
- Fix an error-message.
- Update dependencies.
- Code cleanup.
 2022-02-14 12:36:00 +02:00
Lampros Smyrnaios d2ed9cd9ed Improve efficiency and performance when processing the full-texts. 2022-02-08 15:02:13 +02:00
Lampros Smyrnaios 1111c850b9 - Add support for more than one full-text per id. Allow recognizing fileName additions: "id(1).pdf", "id(2).pdf", etc. 
- Fix not giving the databaseName in the "ImpalaController.get10PublicationIdsTest()".
- Improve consistency in the "maxAttemptsPerRecord" value, among different threads. Also, reduce the value-increase by one.
- Check if the tableName string is empty, in the "mergeParquetFiles".
- Improve error-logging.
- Set some local variables to "final", optimizing code-execution by the JVM.
 2022-02-07 13:57:09 +02:00
Lampros Smyrnaios 6aab1d242b - Improve performance when handling WorkerReports' database insertions, by using parallelism to insert to two different tables in the same time. Also, pre-cache the query-argument-types. 
- Update the error-message and counting system, on partial insertion event.
 2022-02-04 14:48:22 +02:00
Lampros Smyrnaios be4898e43e Bug fixes and improvements: 
- Fix an NPE, when the "getTestUrls"-endpoint is called. It was thrown because of an absent per-thread initialization of some thread-local variables.
- Fix JdbcTemplate error when querying the "getFileLocationForHashQuery".
- Fix the "S3ObjectStore.isLocationInStore" check.
- Fix not catching/handling some exceptions.
- Fix/improve log-messages.
- Optimize the "getFileLocationForHashQuery" to return only the first row. In the latest change, without this optimization, the query-result would cause non-handling the same-hash cases, because of an exception.
- Optimize the "ImpalaConnector.databaseLock.lock()" positioning.
- Update the "getTestUrls" api-path.
- Optimize list-allocation.
- Re-add the info-message about the successful emptying of the S3-bucket.
- Code cleanup.
 2022-02-02 20:19:46 +02:00
Antonis Lempesis 35966b6f6e finishing toucehs 2022-02-01 16:57:28 +02:00
Antonis Lempesis e9bede5c45 more fixes 2022-02-01 02:08:02 +02:00
Antonis Lempesis 9ac10fc4b3 fixed Value annotations 2022-01-31 14:01:26 +02:00
Antonis Lempesis 1c82088a7c fixed Value annotations 2022-01-31 13:49:14 +02:00
Antonis Lempesis 6dde8c0faa finished merge 2022-01-31 04:17:16 +02:00
Antonis Lempesis bf26bf955f springified project 2022-01-30 22:14:52 +02:00
Lampros Smyrnaios d0ab42e4fa - Change the scheme of the file-location URI. 
- Move the old and the current database names in the "application.properties" file.
- Improve logging.
 2022-01-28 07:24:42 +02:00
Lampros Smyrnaios 92b11baf93 - Update the repository for the Impala JDBC Driver. 
- Code cleanup.
 2022-01-28 00:59:19 +02:00
Lampros Smyrnaios a01e11eef0 When all the data is processed, increase the number of "max-attempts" to retry some very old records, in the next requests. 2022-01-27 01:18:26 +02:00
Lampros Smyrnaios 3c9f8870d1 - Change the repository for the Impala JDBC Driver, as the previous one had networking issues. 
- Optimize the "findAssignmentsQuery".
 2022-01-26 19:52:46 +02:00
Lampros Smyrnaios ab99bc6168 - Make sure the temp table "current_assignment" from a cancelled previous execution, is dropped and purged on startup. 
- Improve logging.
- Code cleanup.
 2022-01-19 01:37:47 +02:00
Lampros Smyrnaios 83f40a23d9 Bring back the prepared-statements for the insert-queries. After the fix of the "broken pipe"-error, they now work. Bringing them back, increases security and solves the "SQL syntax errors" caused by the values of some URLs. 2022-01-13 00:54:21 +02:00
Lampros Smyrnaios 2cf25b0d26 - Fix Impala "broken pipe" error, by closing the connection when not in need. The connection is reopened later with minimal overhead, as a connection pool is used. 
- Fix not closing the database-connection in case of a specific error (also in a commented error-case).
 2022-01-13 00:47:15 +02:00
Lampros Smyrnaios 82bf11b9b3 - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements. 
- Update dependencies.
 2021-12-24 00:25:50 +02:00
Lampros Smyrnaios 33ba3e8d91 - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 
- Fix not updating the fileLocation with the s3Url for records which share the same full-text.
- Set only one delete-order for each assignments-batch-files, not one (or more, by mistake) per zip-batch.
- Set the HttpStatus to "204 - NO_CONTENT", when no assignments are available to be returned to the Worker.
- Fix not unlocking the "dataBaseLock" in case of a "dataBase-connection"-error, in "addWorkerReport()".
- Improve some log-messages.
- Change the log-level for the "S3-bucket already exists" message.
- Update Gradle.
- Optimize imports.
- Code cleanup.
 2021-12-21 15:55:27 +02:00
Lampros Smyrnaios 0178e44574 - Increase security by sanitizing the value of the "workerId" before use it in sql-statements. Impala has bugs with some types of PreparedStatements. 
- Improve reliability, by dropping the "current_assignment" table in case of an error, thus the next "getUrls"-request will not fail.
- Fix the "databaseLock" not being unlocked when the "addWorkerReport()" method returned early on some error-cases.
- Delete the "assignment"-data after inserting the related payloads and attempts in the database.
 2021-12-10 21:47:58 +02:00
Lampros Smyrnaios a46ab84f10 - Increase the lower and upper limits for the Java Heap Size. 
- Update the "ServerBaseURL" to the Public IP Address of the machine which is running the app.
 2021-12-06 20:27:39 +02:00
Lampros Smyrnaios dea257b87f - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter". 
- Fix a bug, which caused multiple workers to get assigned the same batch-counter, while the assignment-tasks where different.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the error-message returned from the Worker, when a getFullTexts-request fails.
- Improve some log-messages.
- Update dependencies.
- Code cleanup.
 2021-12-06 20:18:30 +02:00