lsmyrnaios
/
UrlsController
1
0
Fork 1
Code Issues Pull Requests Releases 10 Wiki Activity
281 Commits 2 Branches 10 Tags 799 KiB
Commit Graph

247 Commits

Author SHA1 Message Date
Lampros Smyrnaios 4c3e2e6b6e - Fix not using actual the "currentAssignmentsBatch" of the workerReport itself, when creating the parquetFileNames and when reporting to the user the initialization of the "addition of the workerReport". 
- Code polishing.
 2023-06-27 16:08:01 +03:00
Lampros Smyrnaios 0f4b63c4a9 Expose the following statistics as prometheus-metrics and create/update a stats-endpoint for each one: 
- "numOfPayloadsAggregatedByServiceThroughCrawling"
 - "numOfPayloadsAggregatedByServiceThroughBulkImport"
 - "numOfPayloadsAggregatedByService"
 - "numOfLegacyPayloads"
 - "numOfRecordsInspectedByServiceThroughCrawling" (renamed from "numOfInspectedRecords")
 2023-06-23 15:22:26 +03:00
Lampros Smyrnaios d52601e819 - Add handling for the "EmptyResultDataAccessException" in "UrlsServiceImpl.getAssignments()", which is thrown when no assignments are returned from the query. 
- Code polishing.
 2023-06-22 12:39:11 +03:00
Lampros Smyrnaios b9712bed85 - Expose the "numOfAllPayloads" and "numOfInspectedRecords" DB-stats to Prometheus, by using a scheduling task to request the numbers from the DB, every 6 hours. 
- Update the "StatsServiceImpl.getNumberOfPayloadsAggregatedByService()" to use the new table "payload_aggregated", instead of casting and checking the date of the records.
- Code polishing.
 2023-06-19 14:42:00 +03:00
Lampros Smyrnaios 798fa09d68 - Identify and handle a possible Worker-crash, in "UrlsServiceImpl.postReportResultToWorker()". 
- Add/Improve some log messages.
- Update and cleanup dependencies.
- Code polishing.
 2023-06-15 23:19:36 +03:00
Lampros Smyrnaios 88a74b2c41 Add support for all private addresses, defined in "RFC 1918" standard. This fixes the issue of discarding some "shutdownService" requests due to coming from different local private addresses, when the Controller was run inside a docker container. 2023-06-15 13:26:27 +03:00
Lampros Smyrnaios c37f157f51 Split the full-texts-batch's main handling-code to two separate methods, which can be used in parallel by two threads, in the future. 2023-06-14 17:16:38 +03:00
Lampros Smyrnaios e2776c50d0 - Optimize the "WorkerReportResult" and the "ShutdownWorker" requests. 
- Improve documentation.
 2023-06-10 02:31:57 +03:00
Lampros Smyrnaios 0b1ab5b991 Attempt to recover from serious failures, by individualizing the error-handling for each of the "table-merging" queries. 2023-06-10 02:28:02 +03:00
Lampros Smyrnaios 6669dc61bf - Increase the initialDelay for the "checkIfServiceIsReadyForShutdown" scheduled-task, in production, to 10 minutes. 
- Code polishing.
 2023-06-06 16:49:53 +03:00
Lampros Smyrnaios 54685bbe9a - Avoid sending "cancelShutdown" requests to already shutDown Workers. 
- Optimize performance of the code running right before the "postShutdownOrCancelRequestToWorker".
- Show which Workers have already shutdown and as a result a "postShutdownOrCancelRequestToWorker" will not be performed on them.
 2023-05-29 13:41:37 +03:00
Lampros Smyrnaios f9c6bad768 Do not send shutDownRequests to workers which have already shutdown. 2023-05-29 12:42:54 +03:00
Lampros Smyrnaios a38d6ace79 Code polishing. 2023-05-29 12:21:48 +03:00
Lampros Smyrnaios 74ff31fc64 - Show the workerIPs in the logs. 
- Rename the "FullTexts"-files to "BulkImport".
 2023-05-29 12:12:08 +03:00
Lampros Smyrnaios 3988eb3a48 - Use a separate HDFS sub-dir for every assignments-batch, in order to avoid any disrruptancies from multiple threads moving parquet-files from the same sub-dir. Multiple batches from the same worker may be processed at the same time. These sub-dirs are deleted afterwards. 
- Treat the "contains no visible files" situation as an error. In which case the assignments-data is presumed to not have been inserted to the database tables.
- Code polishing/cleanup.
 2023-05-27 02:36:05 +03:00
Lampros Smyrnaios 02cee097d4 Fix an issue, which could cause some background jobs to be executed more than 1 times. The previously executed jobs were not deleted from the global list fast enough, and they would be selected again, in case they were not finished before the scheduler started again. 2023-05-26 13:08:00 +03:00
Lampros Smyrnaios 2b50e08bf6 - Handle the case, were multiple threads may load the same HDFS directory to a database table, thus causing the "directory contains no visible files"-SQLException. 
- Improve the values of the delays for some scheduledTasks.
- Improve elapsed time precision for the "lastAccessedOn" metadata of the workerReports.
- Code polishing.
 2023-05-25 00:34:36 +03:00
Lampros Smyrnaios 164245cb53 - Automatically delete the unsuccessful WorkerReports, which are more than 7 days old. 
- Optimize the Service's startup speed, by setting "initialDelays" to the scheduled tasks.
- Optimize documentation.
 2023-05-24 16:59:42 +03:00
Lampros Smyrnaios 551c4acef5 Fix property naming missmatch. 2023-05-24 14:49:29 +03:00
Lampros Smyrnaios 8b5f143b0a Place the "workerReports" and the "bulkImportReports" dirs inside the "reports" parent-directory. 2023-05-24 14:10:57 +03:00
Lampros Smyrnaios cd1fb0af88 - Process the WorkerReports in background Jobs and post the reportResults to the Workers. 
- Save the workerReports to json files, until they are processed successfully.
- Show some custom metrics in prometheus.
 2023-05-24 13:52:28 +03:00
Lampros Smyrnaios 0ea3e2de24 Add the "shutdownService" and "cancelShutdownService" endpoints. The Controller sends the related requests to the Workers and shutdowns gracefully, after all workers have shutdown. 2023-05-24 13:42:29 +03:00
Lampros Smyrnaios c2a1b96069 - Rename the mounted "mnt/bulkImport/" directory to "/mnt/bulk_import/". 
- Increase the "awaitTermination" timeout for the ExecutorService to 2 minutes.
 2023-05-23 21:09:34 +03:00
Lampros Smyrnaios c7bfd75973 - Add the "getWorkersInfo" endpoint. 
- Improve startup speed, by using a faster remote server to get the host's machine public IP. This also reduces the risk of not being able to get the public IP at all.
- Fix the detection of a different IP for a known worker.
- Improve documentation.
 2023-05-23 14:57:15 +03:00
Lampros Smyrnaios 5f75b48e95 - Increase the "read-timeout" when searching for the host's machine public-IP. 
- Update dependencies.
- Code polishing.
 2023-05-22 21:33:02 +03:00
Lampros Smyrnaios 0ab6bae93a - Optimize the json-conversion of the "BulkImportReport". 
- Code polishing.
 2023-05-18 17:30:40 +03:00
Lampros Smyrnaios f7f919cee1 - Make sure we set the "hasShutdown" to "false", for each known worker which was restarted. 
- Fix markdown of urls in prometheus' readme.
 2023-05-16 12:24:14 +03:00
Lampros Smyrnaios b499209ce3 - Move the Prometheus and grafana configuration in a dedicated directory and docker-compose file. 
- Add documentation about setting-up prometheus and grafana.
 2023-05-15 18:52:31 +03:00
Lampros Smyrnaios a8eea1ccf4 Fix missing changes. 2023-05-15 13:13:24 +03:00
Lampros Smyrnaios f51a34138f - Use separate HDFS subdirectories for each worker in order to avoid seeing exceptions about "empty hdfs directory" when "loading" data to the database, because one worker has loaded data generated by multiple workers (since we use only 1 load operation for multiple parquet files). 
- Store each worker's info in a hash-table, in order to efficiently know if we need to create new hdfs subdirectories. Also, this will help to issue "shutdown" requests to the workers in the future, as well as to know which worker has shutdown.
 2023-05-15 13:12:20 +03:00
Lampros Smyrnaios 9412391903 - In test-environment mode, check for already existing file-hashes only in the "payload_aggregated" table, instead of the whole "payload" view. This way the investigation for false-positive docUrls is easier, as we avoid checking against the millions of "legacy" payloads. 
- Improve performance in production, by not creating the string objects for "trace"-logs.
 2023-05-15 12:44:16 +03:00
Lampros Smyrnaios 8381df70c6 - Improve performance of uploading parquet-files to HDFS. 
- Add some logs.
- Code polishing.
 2023-05-11 19:40:48 +03:00
Lampros Smyrnaios 992d4ffd5e - Add the time-zone in the logs. 
- Change some log-levels to "trace", although most of them are still disabled.
 2023-05-11 03:10:53 +03:00
Lampros Smyrnaios b6e8cd1889 New feature: BulkImport full-text files from compatible datasources. 2023-05-11 03:07:55 +03:00
Lampros Smyrnaios 42b93e9429 - Add the "getNumberOfAllDistinctFullTexts" stats-endpoint. 
- Add TODOs for more stats endpoints.
- Code polishing.
 2023-05-04 15:48:49 +03:00
Lampros Smyrnaios b3196376eb Fix a bug, which caused the full-text files to never close. 2023-05-04 13:03:28 +03:00
Lampros Smyrnaios fd15372fd6 Add error-checks for retrieving the status-code from HttpUrlConnections. 2023-05-03 13:30:29 +03:00
Lampros Smyrnaios 49662319a1 - Simplify the creation of local directories. 
- Improve exception messages.
 2023-04-28 14:58:33 +03:00
Lampros Smyrnaios 55ea5118ac - Update the "testDatabaseName" property. 
- Code polishing.
 2023-04-26 19:33:28 +03:00
Lampros Smyrnaios d7797eaaf6 Add the "getNumberOfPayloadsForDatasource" endpoint. 2023-04-24 09:54:35 +03:00
Lampros Smyrnaios 4dc34429f8 - Increase the waiting-time before checking the docker containers' status, in order to catch configuration-crashes. 
- Code polishing.
 2023-04-10 22:28:53 +03:00
Lampros Smyrnaios c39fef2654 Upgrade payload-table to payload-view which consists of three separate payload tables: "payload_legacy", "payload_aggregated" and "payload_bulk_import". 2023-04-10 15:55:50 +03:00
Lampros Smyrnaios 37363100fd Prioritize most recent publications. 2023-04-10 15:00:23 +03:00
Lampros Smyrnaios 484cf5cefc - Avoid requesting the remaining full-text batches in case the Worker returns a 5XX error in one of the batches. 
- Add nullability-checks for "datasourceId" and "hash" before constructing the new filename and upload the full-text on S3.
- Improve a log-message.
 2023-03-29 17:12:37 +03:00
Lampros Smyrnaios 882c6f447b Update the "testDatabaseName". 2023-03-21 23:10:21 +02:00
Lampros Smyrnaios 4280f89296 - Set the default value of the "isTestEnvironment" property to "true", in order to avoid undesired outcomes in the production db. 
- Code polishing.
 2023-03-21 17:04:28 +02:00
Lampros Smyrnaios e975bec911 - Add Prometheus and Grafana which help measuring various metrics for the Controller's health and performance. 
- Fix Docker config still using the old (now removed) "application.properties" file.
- Simplify the process of building and running the docker image; Now we use docker compose to run the Controller, along with the Prometheus and Grafana. Also, now it is not requested from the user to login and push the image (this may change in the future).
 2023-03-21 16:46:33 +02:00
Lampros Smyrnaios 003c0bf179 - Add support for excluding specific datasources from being crawled. These datasources may be aggregated through bulk-imports, by other pieces of software. Such a datasource is "arXiv.org". 
- Fix an issue, where the "datasource-type" was retrieved instead of the "datasource-name".
- Polish the "findAssignmentsQuery".
 2023-03-21 07:19:35 +02:00
Lampros Smyrnaios f835a752bf Transform the "application.properties" file to "application.yml" and optimize the property-trees. 2023-03-20 15:23:00 +02:00
Lampros Smyrnaios 17a6c120dd Improve logs for full-texts' metrics. 2023-03-14 20:57:01 +02:00
Lampros Smyrnaios ff13af7abb Use a StatsService interface. 2023-03-13 12:39:39 +02:00
Lampros Smyrnaios 38643c76a3 - Code polishing. 
- Update Gradle.
 2023-03-07 16:55:41 +02:00
Lampros Smyrnaios 7b217764e0 Improve performance when downloading and decompressing the full-texts archive. 2023-03-02 17:44:53 +02:00
Lampros Smyrnaios c4670073ae - Add missing refactoring-change. 
- Code polishing.
- Update Spring.
 2023-02-24 23:49:04 +02:00
Lampros Smyrnaios c8485d472e Code polishing. 2023-02-24 13:53:09 +02:00
Lampros Smyrnaios b7f6056032 - Improve an error-message. 
- Update Gradle.
 2023-02-21 15:42:07 +02:00
Lampros Smyrnaios 8893662a81 Refactor the UrlsController: a) offload the business-logic to the dedicated "UrlsService" and b) move the "checkParquetFilesSuccess()"-method to "ParquetFileUtils". 2023-02-21 15:36:35 +02:00
Lampros Smyrnaios a1c16ffc19 - Exclude empty and null urls in the assignments. 
- Update the "getFullTextsImproved"-call to "getFullTexts", now that the "improved" version is stable.
- Update Gradle.
- Code polishing.
 2023-02-16 14:24:47 +02:00
Lampros Smyrnaios 2253f05bf5 Refactor the "StatsController"-code, by offloading it to a dedicated "StatsService". 2023-02-09 19:25:48 +02:00
Lampros Smyrnaios 49fefefafd - Refactor the payloads-statistics-code and provide two endpoints: "getNumberOfPayloadsAggregatedByService", which returns the number of payloads aggregated only by the PDF-Aggregation-Service, and the "getNumberOfAllPayloads", which returns the number of all payloads existing in the database, even the ones aggregated in the past, by other pieces of software. 
- Update README.md.
- Make sure the docker image is clean-built, by avoiding the use of cache.
 2023-02-02 17:58:47 +02:00
Lampros Smyrnaios c9f33d3afa Add an extra precaution-check to allow the emptying or deletion of an S3-Object-Store bucket, only when the app runs in "TestEnvironment". 2023-02-01 16:42:22 +02:00
Lampros Smyrnaios f89730f196 Improve documentation. 2023-01-27 14:31:07 +02:00
Lampros Smyrnaios dc8f0f2bd1 - Heavily reduce the maximum amount of space needed, by deleting the files of each full-texts batch, right after they are uploaded to the S3 Object Store. 
- Add a check for when the retrieved full-texts-batch is missing some requested files and show a warn-log.
- Update dependencies.
 2023-01-23 20:23:21 +02:00
Lampros Smyrnaios d8773e6ebb - Make sure the test-environment uses a dedicated hdfs-parquet-directory. 
- Block app-execution in case the hdfs parquet directories failed to be created.
- Code polishing.
 2023-01-18 13:38:05 +02:00
Lampros Smyrnaios 8876089022 - Use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits on compression rate and speed. 
- Update the minIO dependency.
- Code polishing.
 2023-01-10 13:34:54 +02:00
Lampros Smyrnaios d1a4c84289 - Make sure the fullPath of the baseFilesLocation is available when the user specifies a non-root directory. 
- Improve error-checking and exception-handling in some "S3ObjectStore"-methods.
- Make sure the "responseCode" is "200-OK", before trying to get the InputStream in "UriBuilder.getPublicIP()".
 2023-01-09 15:44:53 +02:00
Lampros Smyrnaios 9904ea5743 - Improve the stability of "UriBuilder.getPublicIP()", by using a "HttpURLConnection" to increase the connection and read timeouts and avoid timeout-exceptions. 
- Update Spring.
 2023-01-03 18:39:50 +02:00
Lampros Smyrnaios 4528d1f9be - Fix the "baseFilesLocation" being null (there was no serious problem, but multiple directories were spawned in the project's directory). 
- Make sure the given "baseFilesLocation" ends with a file-separator, before using it.
- Optimize the process of unzipping-files.
 2022-12-20 18:38:11 +02:00
Lampros Smyrnaios e11afe5ab2 Improve performance of the hash-checking algorithm by using multithreading. 2022-12-15 18:34:28 +02:00
Lampros Smyrnaios 9cdbbdea67 Refactor the files' storage location. 2022-12-15 18:29:51 +02:00
Lampros Smyrnaios e51ee9dd27 - Add info about the Stats API usage in "README.md". 
- Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()".
- Handle the "EmptyResultDataAccessException" inside "StatsController".
- Optimize gradle's performance.
- Code polishing.
 2022-12-15 14:04:22 +02:00
Lampros Smyrnaios bfdf06bd09 - Apply error-checking on individual CallableTasks and in tasks-batches related to the creation and upload of all the data related to the "attempt" and "payload" table. So, if no data could be uploaded for one or both tables, no "load"-queries will be executed for that/those tables. 
- Catch the more general "Exception", inside "FileUtils.mergeParquetFiles()", in order to be certain that the "SQLException" can also be caught.
- Code polishing.
 2022-12-09 12:46:06 +02:00
Lampros Smyrnaios 0209d24068 - Change the parquet compression from "Snappy" to "Gzip", as there is an unhandleable exception when the app is running inside a Docker Container and uses the "Snappy" compression. 
- Code polishing.
 2022-12-08 16:28:41 +02:00
Lampros Smyrnaios c8baf5a5fc - Fix not finding the parquet-schema files when the app was run inside a Docker Container. 
- Update the "namespaces" and the "names" inside the parquet schemas.
- Code polishing.
 2022-12-08 12:16:05 +02:00
Lampros Smyrnaios 95c38c4a24 - Fix creating the "assignment" table, always in the testDatabase. 
- Code polishing.
 2022-12-07 14:58:38 +02:00
Lampros Smyrnaios 3c5f4c6464 Fix bytes to MB conversion. 2022-12-07 14:32:18 +02:00
Lampros Smyrnaios 8607594f6d - Improve exception handling. 
- Code polishing.
 2022-12-07 13:48:00 +02:00
Lampros Smyrnaios f183df276b - Move the "uploadFullTexts"-code in its own method. 
- Code polishing.
 2022-12-06 12:24:34 +02:00
Lampros Smyrnaios b0c57d79a5 - When the Controller cannot retrieve any assignments from Impala (without an error), return an HTTP-"MULTI_STATUS" with an empty "AssignmentsResponse", instead of an "INTERNAL_SERVER_ERROR". 
- Fix an error-message.
 2022-12-05 16:44:00 +02:00
Lampros Smyrnaios 577ea983e8 - Improve some log-messages. 
- Set some optimization settings for gradle.
- Fix error-handling in "installAndRun.sh".
- Update dependencies.
 2022-11-30 16:28:39 +02:00
Lampros Smyrnaios 6226e2298d - Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement. 
One side effect of using the parquet-files, is that the timestamps are now BIGDECIMAL numbers, instead of "Timestamp" objects, but, converting them to such objects is pretty easy, if we ever need to do it.
- Code polishing.
 2022-11-10 17:18:21 +02:00
Lampros Smyrnaios aad37cd81e Add the "StatsController", which brings the "getNumberOfPayloads" and "getNumberOfRecordsInspected" endpoints. 2022-10-18 15:00:26 +03:00
Lampros Smyrnaios e2d53105d1 Fix not creating the "assignment" table in a new production database, which contains only the "publication" and "datasource" data. 2022-10-07 15:51:31 +03:00
Lampros Smyrnaios b6340066a7 - Improve handling of the case, where the full-texts were found, but the Controller could not acquire them from the Worker. 
- Add/improve logs and comments.
- Code cleanup.
 2022-09-28 22:34:33 +03:00
Lampros Smyrnaios a22144bd51 - Refactor "FileUtils.getErrorMessageFromResponseBody(conn)" into "FileUtils.getMessageFromResponseBody(conn, isError)", in order to be able to either retrieve the "normal" or the "error" response. 
- Add comments.
 2022-09-15 23:12:05 +03:00
Lampros Smyrnaios 3e8f9c6074 Update the "UriBuilder.java" to be able to acquire the running port of the server, in case the port-number was initially set to "random" (0). Also make sure we get the "localHostAddress" and not the "localHostName", in case the public IP is not retrievable. 2022-09-12 17:04:05 +03:00
Lampros Smyrnaios a2cd02115f - Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated. 
- Update dependencies.
 2022-06-27 21:41:32 +03:00
Lampros Smyrnaios e3b374a32f - Optimize file-related tasks. 
- Update dependencies.
- Code cleanup.
 2022-05-26 15:43:59 +03:00
Lampros Smyrnaios 9096137008 Update documentation. 2022-04-14 14:42:36 +03:00
Lampros Smyrnaios 9b95eebb6c - Remove the obsolete "parenthesis" and "increasing duplicate-num" from the full-texts' names, before sending them to the S3-Object-Store. They now end with the "file-hash", so it is guaranteed that they will be unique. The Worker continues to produce the previous kind of names, without any disturbance. 
- Improve logging.
- Update MinIO dependency.
 2022-04-11 21:15:22 +03:00
Lampros Smyrnaios a81ed3c60f - Add an "isTestEnvironment"-switch, which makes it easier to work with production and test databases. 
- In case the Worker cannot be reached during a full-texts' batch request, abort the rest of the batches.
- Fix memory leaks when unzipping the batch-zip-file.
- Add explanatory comments for picking the database related to a full-text file.
 2022-04-08 17:39:45 +03:00
Lampros Smyrnaios 33fc61a8d9 - Fix the fileName-ID not being directly related with the datasourceID, in the S3-ObjectStore name. Add explanatory comments. 
- Add missing error-logs.
 2022-04-05 16:22:02 +03:00
Lampros Smyrnaios a23c918a42 - Fix a "@JsonProperty" annotation inside "Payload.java". 
- Fix a "@Value" annotation inside "FileUtils.java".
- Add a new database and show its name along with the initial's name in the logs.
- Code cleanup and improvement.
 2022-04-05 00:01:44 +03:00
Lampros Smyrnaios 5e4fad2479 - Change the fileNames' structure in the S3-ObjectStore. 
- Update dependencies.
 2022-04-01 19:24:04 +03:00
Lampros Smyrnaios 48670f3399 - Show the percentage of the "NumFullTextsFound", in the logs. 
- Update dependencies.
 2022-03-28 14:29:31 +03:00
Lampros Smyrnaios 88acaae20f - Replace the "numFullTextUrlsFound"-counter with "numFullTextsFound"-counter to reflect the end result of the actually available full-texts (which were downloaded by the Worker). 
- Optimize the gather-fileNames loop.
- Improve a message in "installAndRun.sh"
 2022-02-23 17:40:06 +02:00
Lampros Smyrnaios ad5dbdde9b - Improve performance when inserting records into the "attempt" table, by splitting the records equally, across more threads. 
- Bring back the "UriBuilder", which informs us in the logs, about the Controller's url (IP, PORT, API).
- Code cleanup.
 2022-02-22 13:54:16 +02:00
Lampros Smyrnaios dfd40cb105 Insert only the records with uploaded-to-S3 full-texts, in the "payload" table. 2022-02-17 16:27:40 +02:00
Lampros Smyrnaios 71f6b46130 - In case of an error when creating the "current_assignment" table (e.g out of memory in the backend database server), check for partial-creation and drop it. Also, in any case, before we drop this table, now check if it exists firsts (in general it should always exist, unless the creation results in an error and the table was not created at all). 
- Fix an error-message.
- Update dependencies.
- Code cleanup.
 2022-02-14 12:36:00 +02:00
Lampros Smyrnaios d2ed9cd9ed Improve efficiency and performance when processing the full-texts. 2022-02-08 15:02:13 +02:00
Lampros Smyrnaios 1111c850b9 - Add support for more than one full-text per id. Allow recognizing fileName additions: "id(1).pdf", "id(2).pdf", etc. 
- Fix not giving the databaseName in the "ImpalaController.get10PublicationIdsTest()".
- Improve consistency in the "maxAttemptsPerRecord" value, among different threads. Also, reduce the value-increase by one.
- Check if the tableName string is empty, in the "mergeParquetFiles".
- Improve error-logging.
- Set some local variables to "final", optimizing code-execution by the JVM.
 2022-02-07 13:57:09 +02:00
Lampros Smyrnaios b206114144 - Allow the user to build, push and run the App in Docker, straight though the "installAndRun.sh" script. 
- Re-add the logback-spring configuration.
- Change the docker-app name.
 2022-02-04 15:49:56 +02:00
Lampros Smyrnaios 6aab1d242b - Improve performance when handling WorkerReports' database insertions, by using parallelism to insert to two different tables in the same time. Also, pre-cache the query-argument-types. 
- Update the error-message and counting system, on partial insertion event.
 2022-02-04 14:48:22 +02:00
Lampros Smyrnaios be4898e43e Bug fixes and improvements: 
- Fix an NPE, when the "getTestUrls"-endpoint is called. It was thrown because of an absent per-thread initialization of some thread-local variables.
- Fix JdbcTemplate error when querying the "getFileLocationForHashQuery".
- Fix the "S3ObjectStore.isLocationInStore" check.
- Fix not catching/handling some exceptions.
- Fix/improve log-messages.
- Optimize the "getFileLocationForHashQuery" to return only the first row. In the latest change, without this optimization, the query-result would cause non-handling the same-hash cases, because of an exception.
- Optimize the "ImpalaConnector.databaseLock.lock()" positioning.
- Update the "getTestUrls" api-path.
- Optimize list-allocation.
- Re-add the info-message about the successful emptying of the S3-bucket.
- Code cleanup.
 2022-02-02 20:19:46 +02:00
Antonis Lempesis 35966b6f6e finishing toucehs 2022-02-01 16:57:28 +02:00
Antonis Lempesis e9bede5c45 more fixes 2022-02-01 02:08:02 +02:00
Antonis Lempesis 9ac10fc4b3 fixed Value annotations 2022-01-31 14:01:26 +02:00
Antonis Lempesis 1c82088a7c fixed Value annotations 2022-01-31 13:49:14 +02:00
Antonis Lempesis 6dde8c0faa finished merge 2022-01-31 04:17:16 +02:00
Antonis Lempesis bf26bf955f springified project 2022-01-30 22:14:52 +02:00
Lampros Smyrnaios d0ab42e4fa - Change the scheme of the file-location URI. 
- Move the old and the current database names in the "application.properties" file.
- Improve logging.
 2022-01-28 07:24:42 +02:00
Lampros Smyrnaios 92b11baf93 - Update the repository for the Impala JDBC Driver. 
- Code cleanup.
 2022-01-28 00:59:19 +02:00
Lampros Smyrnaios a01e11eef0 When all the data is processed, increase the number of "max-attempts" to retry some very old records, in the next requests. 2022-01-27 01:18:26 +02:00
Lampros Smyrnaios 3c9f8870d1 - Change the repository for the Impala JDBC Driver, as the previous one had networking issues. 
- Optimize the "findAssignmentsQuery".
 2022-01-26 19:52:46 +02:00
Lampros Smyrnaios ab99bc6168 - Make sure the temp table "current_assignment" from a cancelled previous execution, is dropped and purged on startup. 
- Improve logging.
- Code cleanup.
 2022-01-19 01:37:47 +02:00
Lampros Smyrnaios 83f40a23d9 Bring back the prepared-statements for the insert-queries. After the fix of the "broken pipe"-error, they now work. Bringing them back, increases security and solves the "SQL syntax errors" caused by the values of some URLs. 2022-01-13 00:54:21 +02:00
Lampros Smyrnaios 2cf25b0d26 - Fix Impala "broken pipe" error, by closing the connection when not in need. The connection is reopened later with minimal overhead, as a connection pool is used. 
- Fix not closing the database-connection in case of a specific error (also in a commented error-case).
 2022-01-13 00:47:15 +02:00
Lampros Smyrnaios 82bf11b9b3 - Workaround a bug of Impala-JDBC-Driver, when creating insert-prepared-statements. 
- Update dependencies.
 2021-12-24 00:25:50 +02:00
Lampros Smyrnaios 33ba3e8d91 - Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches. 
- Fix not updating the fileLocation with the s3Url for records which share the same full-text.
- Set only one delete-order for each assignments-batch-files, not one (or more, by mistake) per zip-batch.
- Set the HttpStatus to "204 - NO_CONTENT", when no assignments are available to be returned to the Worker.
- Fix not unlocking the "dataBaseLock" in case of a "dataBase-connection"-error, in "addWorkerReport()".
- Improve some log-messages.
- Change the log-level for the "S3-bucket already exists" message.
- Update Gradle.
- Optimize imports.
- Code cleanup.
 2021-12-21 15:55:27 +02:00
Lampros Smyrnaios 0178e44574 - Increase security by sanitizing the value of the "workerId" before use it in sql-statements. Impala has bugs with some types of PreparedStatements. 
- Improve reliability, by dropping the "current_assignment" table in case of an error, thus the next "getUrls"-request will not fail.
- Fix the "databaseLock" not being unlocked when the "addWorkerReport()" method returned early on some error-cases.
- Delete the "assignment"-data after inserting the related payloads and attempts in the database.
 2021-12-10 21:47:58 +02:00
Lampros Smyrnaios a46ab84f10 - Increase the lower and upper limits for the Java Heap Size. 
- Update the "ServerBaseURL" to the Public IP Address of the machine which is running the app.
 2021-12-06 20:27:39 +02:00
Lampros Smyrnaios dea257b87f - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter". 
- Fix a bug, which caused multiple workers to get assigned the same batch-counter, while the assignment-tasks where different.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the error-message returned from the Worker, when a getFullTexts-request fails.
- Improve some log-messages.
- Update dependencies.
- Code cleanup.
 2021-12-06 20:18:30 +02:00
Lampros Smyrnaios 15224c6468 Improve performance in the "getUrls"-endpoint, and more: 
- Optimize the "findAssignmentsQuery" by using an inner limit (larger than the outer).
- Save a ton of time from inserting the assignments into the database, by using a temporal table to hold the new assignments, in order for them to be easily accessible both from the Controller (which processes them and send them to the Worker) and the database itself, in order to "import" them into the "assignment"-table.
- Replace the "Date" with "Timestamp", in order to hold more detailed information.
- Code cleanup.
 2021-11-30 19:59:46 +02:00
Lampros Smyrnaios 48eed20dd8 - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 
- Move the "mergeParquetFiles" and "getCutBatchExceptionMessage" methods inside the "FileUtils" class.
- Code cleanup.
 2021-11-30 18:23:27 +02:00
Lampros Smyrnaios 780ed15ce2 - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction. 
- Fix the "null" representation of an "unknown" payload-size in the database.
- Remove the obsolete thread-locking for the "CreateDatabase" operation. This code is guaranteed to run BEFORE any other operation in the database.
- Implement the "handlePreparedStatementException" and "closeConnection" methods.
- Improve error-logs.
- Update dependencies.
- Code cleanup.
 2021-11-30 13:26:19 +02:00
Lampros Smyrnaios d100af35d0 - Implement the "getUrls" and "addWorkerReport" endpoints with full database-handling. 
- Add connectivity with an Impala-database and create a dedicated Controller for future statistics-requests.
- Optimize the "getTestUrls"-endpoint.
- Disable the "reportCurrentTime()" scheduled-task.
- Update dependencies and bump project's version to '1.0.0-SNAPSHOT'.
- Set the logging-appender to "File".
- Code cleanup.
 2021-11-09 23:59:27 +02:00
Lampros Smyrnaios 0d47c33a08 - Improve logging configurations. 
- Fix the "AssignmentResponse.toString()" method.
- Update SpringBoot to v.2.5.6
- Code cleanup.
 2021-11-04 11:57:19 +02:00
Lampros Smyrnaios 0540820817 -Update "installAndRun.sh": 
-- Check if a gradle installation with the given version already exists, before downloading and installing gradle.
-- Make sure the "unzip" package is installed, before trying unzipping the gradle package.
- Update the logs' fileName.
 2021-10-14 02:46:33 +03:00
Lampros Smyrnaios d931315ced - Add the "isControllerAlive"-endpoint. 
- Change the data-type of the "UrlReport.status" to be "enum StatusType", in order to increase consistency and comparability.
- Change the "Date" datatype in "Payload" to have the SQL's version.
- Fix the project's name inside "settings.gradle".
- Code cleanup.
 2021-09-23 15:08:52 +03:00
Lampros Smyrnaios d56e988518 - Process the Error of PDF-aggregation. Distinguish between "couldRetry" and "noRetry" cases. 
- Update the "RequestParam" for the getUrls endpoints.
- Fix the "assignmentCounter".
- Code cleanup.
 2021-08-05 15:43:37 +03:00
Lampros Smyrnaios 25c566bf68 - Change server's port. 
- Update dependencies.
 2021-07-29 08:44:36 +03:00
Lampros Smyrnaios 27375b9396 - Refactor the Assignment-creation. 
In order to match the database, now we have a list of Assignments sent through the AssignmentResponse, instead of a single Assignment having a list of tasks.
- Cleanup the members of the "Payload" model (also prepare for database integration).
 2021-07-05 14:04:39 +03:00
Lampros Smyrnaios 5e7ccbd8c6 Add the "addWorkerReport" endpoint. 2021-06-22 05:38:48 +03:00
Lampros Smyrnaios 40763ec146 Update the "WorkerReport" request and the "UrlReport" and "Payload" models. 2021-06-19 07:07:36 +03:00
Lampros Smyrnaios c194af167f Allow handling of concurrent requests to the "getTestUrls"-endpoint. 2021-06-10 20:24:51 +03:00
Lampros Smyrnaios 308cab5ecd - Return an HTTP-500-error when the server cannot find the resourceFile requested by the "getTestUrls"-endpoint. 
- Close the "inputScanner" after each use when retrieving the test-tasks.
- Show info-logs when sending an assignment to a worker.
- Code cleanup.
 2021-06-10 14:21:39 +03:00
Lampros Smyrnaios 6729f51b03 Add an "assignmentId" field in the "Assignment"-class. 2021-06-09 05:48:54 +03:00
Lampros Smyrnaios 787299b5b7 Add the "Datasource" inside the "Task" class and include it in the Assignment. 2021-05-20 02:50:50 +03:00
Lampros Smyrnaios d20fcf9cce - Update the "getUrls" and "getTestUrls" endpoints to take the data from the workers, in http-request-parameters instead of the http-body. 
- Fix tasksLimit-check.
- Code cleanup.
 2021-05-19 02:32:46 +03:00
Lampros Smyrnaios e2cc320baf - Add the "getTestUrls"-endpoint which returns an "Assignment" with data retrieved from the added resource-file. 
- Update the "getUrls"-endpoint to be ready to retrieve data from the database, once it's added.
- Update the dependencies.
- Code cleanup.
 2021-05-18 17:23:20 +03:00
Lampros Smyrnaios d3588ea36b Add the "DownloadAttempt" class. 2021-04-24 21:44:51 +03:00
Lampros Smyrnaios a6ab810ad3 Update classes: "Publication" and "Payload". 2021-04-24 21:40:10 +03:00
Lampros Smyrnaios 85ecc4a36b Add classes: "AssignmentResponse", "WorkerReport", "WorkerRequest", "UrlReport". 2021-04-24 21:06:52 +03:00
Lampros Smyrnaios c2ea8a69de Update classes: "Assignment", "Task", "Error", "Payload", "UrlsRequest". 2021-04-24 21:05:21 +03:00
Lampros Smyrnaios 89c6a73a30 Add "Assignment", "Task" and "Error" classes. 2021-04-15 03:36:08 +03:00
Lampros Smyrnaios c6e12d3e95 - Update "addResults"-endpoint. 
- Add "UrlsRequest.java".
- Some minor updates in "build.gradle" and "application.properties".
 2021-03-16 18:07:30 +02:00
Lampros Smyrnaios 8a4376da9c Initial commit of UrlsController. 2021-03-16 15:25:15 +02:00