Commit Graph

65 Commits

Author SHA1 Message Date
Lampros Smyrnaios 4b85b092fe Handle the new "HttpStatus.MULTI_STATUS"-response from the Controller, inside "AssignmentsHandler.postWorkerReport()". 2022-09-28 22:41:43 +03:00
Lampros Smyrnaios b051e10fd3 - Fix a bug, causing the domainAndPath-tracking data to be deleted after every batch, after the initial threshold was reached. Now the thresholds increase, along the processed id-urls, in order to clear data, e.g. every 300_000 processed id-urls, as intended.
- Use different thresholds for clearing just the "domainAndPath"-blocking-data and all-tracking-data.
2022-09-28 19:10:01 +03:00
Lampros Smyrnaios 373bfa810b - Apply a "shouldShutdownWorker"-check in "ScheduledTasks.handleNewAssignments()", when there was a "connection-error" in the previous request. This makes sure that the Worker will honor the user's shut down request, even if it's "stuck" in a connection-error loop.
- Optimize the input-streams creation in the "FullTextsController".
2022-09-12 16:48:44 +03:00
Lampros Smyrnaios d73a99b1c0 - Increase the security of "shutdownWorker" and "cancelShutdownWorker" endpoints, by only allowing the requests, which come from the same machine.
- Update the "UriBuilder.java" to be able to take the running port of the server, in case the port-number was initially set to "random" (0).
2022-09-12 16:38:44 +03:00
Lampros Smyrnaios 25070d7aba - Lower the thresholds for how often to clear the data-structures.
- Clear the "ConnSupportUtils.domainsWithConnectionData" data-structure, after each batch.
- Move the code for handling the "CookieStore" inside the "PublicationsRetrieverPlugin", as it is more related to that.
2022-07-04 18:42:05 +03:00
Lampros Smyrnaios 5035094e44 - Move the "shutdownOrCancelCode" input in the "inputDataFile" provided by the user, for convenience and to be able to make this "auth-code" mandatory. Previously, it was optional and the app could not be made to stop in a normal-manner, if this code was not provided.
- Improve the instructions and the error-messages for the "inputDataFile".
2022-06-28 16:00:11 +03:00
Lampros Smyrnaios d91732bc16 - Add deletion, of the cookies in the newly-supported CookieManager, after each batch.
- Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated.
- Update dependencies.
- Code cleanup.
2022-06-27 17:58:02 +03:00
Lampros Smyrnaios 26cbb83b51 - Add the "shutdownWorker"-endpoint to accept requests for shutting-down the Worker, gracefully, after it completes its current work (including sending the publications-files to the Controller). A user-defined "auth-code" is required.
- Add the "cancelShutdownWorker"-endpoint to cancel a previous "shutdownWorker"-request. A user-defined "auth-code" is required.
2022-06-22 18:53:27 +03:00
Lampros Smyrnaios d6e94912a4 - Optimize zip-file creation.
- Update dependencies.
2022-05-26 15:24:36 +03:00
Lampros Smyrnaios a1f750a0aa - Handle the case, where, from a group of related records, the initial record which led to a publication-url, failed to have its full-text downloaded. Now we make sure the file-related data for all those related records is kept "null" and a special error is written.
- Code optimization.
2022-04-05 17:51:45 +03:00
Lampros Smyrnaios d682298850 Improve assignment of "PublicationsRetriever.threadsMultiplier", depending on the total available threads on the system. The previous assignment was not scaling well. 2022-04-05 00:13:52 +03:00
Lampros Smyrnaios 4976afa829 Fix a "@JsonProperty" annotation inside "Payload.java". 2022-04-01 23:43:43 +03:00
Lampros Smyrnaios 31af0a81eb - Update the Worker's report to include the datasourceID for each record. It is used by the Controller inside the S3-fileNames.
- Update dependencies.
2022-04-01 19:42:32 +03:00
Lampros Smyrnaios 760e0ef7e2 Increase "PublicationsRetriever.threadsMultiplier" to 10, which in turn, increases performance by 41%. 2022-02-23 17:31:32 +02:00
Lampros Smyrnaios 377b98d677 Increase the "requestReadTimeoutDuration" from 1 hour to 3. This way, each worker will handle saturation without aborting the connection, when multiple workers are waiting for the "databaseLock" in the Controller. 2022-02-22 13:29:02 +02:00
Lampros Smyrnaios edbf6461d5 - Refactor the scheduling of the "handleNewAssignments()" task. Spring already waits for the last task to get finished, before running the new one (unless Async is specifically enabled), so the "isAvailableForWork" didn't do anything (thus the bug described in a previous commit was never going to appear). Also, now we set to request the new assignments-batch immediately after the last one is finished (not after 15 mins), while dealing with potential continuous connection-errors.
- Avoid running the "deleteHandledAssignmentsFullTexts()" scheduled task on application's start.
- Optimize assignment of "requestUrl".
- Add clarity in the scheduled tasks, by using "fixedDelay" instead of "fixedRate", to signify that the time specified is counted right from the time the last task is finished (even though without enabling the "Async" there is no "danger" of running them in parallel).
- Code cleanup.
2022-02-21 12:48:21 +02:00
Lampros Smyrnaios 0d2f0b8b01 Code cleanup. 2022-02-19 17:21:51 +02:00
Lampros Smyrnaios b63ad87d00 Bug fixes:
- Fix a bug, where, in case it took too long to get the assignments from the Controller (possible when there are too many workers requesting at the same time or if the database is responding slowly), the Worker's scheduler would request for new assignments, in the meantime.
- Fix a bug, where, if the "maxAssignmentsBatchesToHandleBeforeRestart" was set, the Worker's scheduler could request another batch, right before the Worker was about to shut down.
- Fix a bug, where the condition of when to clear the over-sized data-structures was based on the "assignmentRequestCounter" send by the Controller (which is increased on each request by any worker and not for each individual one), and not on the "numHandledAssignmentsBatches" kept by each individual worker. This would result in much earlier cleanup, relative to the number of the Workers.
2022-02-19 17:09:02 +02:00
Lampros Smyrnaios 3d1faf4a8a - Reduce memory-consumption in the long-run, by clearing some underlying data-structures after a threshold.
- Update Gradle.
2022-02-18 20:02:34 +02:00
Lampros Smyrnaios 73552ce079 - Handle the latest download-errors provided by the "PublicationsRetriever" program.
- Update the "test" requestUrl.
2022-02-07 14:40:33 +02:00
Lampros Smyrnaios 2e4c1323a3 - Add check for null or empty id or url.
- Code cleanup.
2022-01-28 03:47:46 +02:00
Lampros Smyrnaios 8912bb1cf9 Fix adding an invalid error-message in case of an "alreadyDownloaded" full-text being discovered inside the "FileUtils.dataToBeLoggedList". 2022-01-17 23:46:15 +02:00
Lampros Smyrnaios 0032a8018f - Improve search-accuracy of "alreadyDownloaded" full-texts.
- Handle the potential error-case of an "alreadyDownloaded" full-text not being discovered inside the "FileUtils.dataToBeLoggedList".
2022-01-17 10:12:48 +02:00
Lampros Smyrnaios d61ff4b6dd Integrate some changes from the "PublicationsRetrieverPlugin". 2022-01-14 15:13:00 +02:00
Lampros Smyrnaios 8abb260d60 - In case of an unknown (non-documented) exception inside "LoaderAndChecker.invokeAllTasksAndWait", now it will be logged and the app will gently shut down with an error-message in the Error-stream.
- Avoid double-checking for handledAssignments -in order to delete their full-texts- when the app is about to shut down, in case the "maxAssignmentsBatchesToHandleBeforeRestart" is set above Zero.
2022-01-04 00:23:45 +02:00
Lampros Smyrnaios 92d011e8a0 - Make sure the handled assignments - full-texts are deleted before the application exits.
- When the user sets the "maxAssignmentsBatchesToHandleBeforeRestart" above zero, shutdown immediately after the last assignments-batch. Do not wait for the next scheduled check.
- Allow the user to set the "maxAssignmentsBatchesToHandleBeforeRestart" in the "installAndRun.sh" script.
- Increase the "fixedRate" for the "ScheduledTasks.deleteHandledAssignmentsFullTexts()" method to 12 hours.
- Update README.md
2021-12-31 04:09:05 +02:00
Lampros Smyrnaios 1ddfd34236 - Allow the user to set a maximum number of assignments-batches for the Worker to handle. After handling those batches, the Worker will shut down. A number of < 0 > indicates an infinite number of batches.
- Avoid converting the zero fileSize to < null >. Now, the default value is < null >, so the zero-value will indicate a zero-byte file.
- Update dependencies.
- Code cleanup.
2021-12-24 00:12:34 +02:00
Lampros Smyrnaios a8e2ddcf54 - Reduce the "PublicationsRetriever.threadsMultiplier" to 4.
- Eliminate a possible NPE.
2021-12-20 22:25:27 +02:00
Lampros Smyrnaios c46c8c448a - Upgrade the zip-file delivery by using the "InputStreamResource". This way is more reliable, have better performance and uses less memory.
- Use the "InputStreamResource" also in "get(single)FullText"-endpoint, in order to avoid loading a big full-text file in memory.
- Decrease the system-reserved memory by 128 MB.
- Fix path-variable regexes for "getFullText"-endpoint.
- Optimize imports.
- Code cleanup.
2021-12-17 08:25:54 +02:00
Lampros Smyrnaios 4fb5becace - Increase the system-reserved memory, in "installAndRun.sh".
- Fix not closing the zip-entry in case of an error.
2021-12-17 00:26:47 +02:00
Lampros Smyrnaios 82d69f3bf5 - Calculate and set the max heap size with respect to the system resources, in "installAndRun.sh".
- Fix not setting the right "Error"-members when the docUrl was found, but the full-text was not retrieved.
- Set a "couldRetry"-indication in the "Error"-class, when the full-text was retrieved, as, in general, it could be retried to give the same successful result.
- Update the "docFileNotRetrieved"-check to use the standardized string.
- Eliminate some possible NPEs.
- Update Gradle.
2021-12-16 02:04:05 +02:00
Lampros Smyrnaios 0db35a83e7 - Reduce memory consumption and fix a potential issue, where many "already-retrieved" full-texts would be already deleted (or in different directories, for a short time), as they belonged to a previous assignments-batch (this case is now possible, after the following fix).
- Fix a bug, causing a missing character in the "alreadyDownloaded" full-text fileName, which in turn caused the file-data of that record to not get updated with the file-data of the record for which the same file was initially downloaded for.
2021-12-13 21:16:30 +02:00
Lampros Smyrnaios 859f850f56 - Improve Zip-file delivery and significantly decrease memory consumption, by streaming it, instead of loading the whole file in memory before sending it to the Controller.
- Fix not closing the inputStream of the zip-file.
- Count and show the number of files which were zipped in each batch.
2021-12-13 15:29:03 +02:00
Lampros Smyrnaios ab5e04698c - Fix a bug, causing the "fileLocation" to be set to the value of the "errorCause", when the docFile was not retrieved.
- Leave the "fileLocation" to be NULL, when the DocFile was not retrieved. Previously, the value "File not retrieved" was assigned (only in theory, because the bug above caused the related check to always fail).
- Verify that the "ControllerBaseUrl" given by the user is not malformed.
2021-12-07 19:33:10 +02:00
Lampros Smyrnaios fd5b56e3c6 - Allow the user to set the "maxAssignmentsLimitPerBatch" value.
- Set increased lower and upper limits for the Java Heap Size.
- Update the "ServerBaseURL" to the Public IP Address of the machine which is running the app.
- Improve two log-messages.
2021-12-07 00:52:40 +02:00
Lampros Smyrnaios ce49bff50e - Reduce memory consumption when loading a zipFile.
- Check whether the "zipBatchCounter" is larger than the "totalZipBatches".
- Improve the "failed tasks" log-message.
2021-12-03 16:29:16 +02:00
Lampros Smyrnaios 018326eedd - Optimize the "FileZipper.zipMultipleFilesAndGetZip()" and "FileZipper.zipAFile()" methods.
- Improve the "getMultipleFullTexts"-endpoint. Check if the "fileNamesWithExtensions"-list is empty. Check if the baseDir for the fullTexts of a given assignments-counter is missing.
- Optimize the "PublicationsRetrieverPlugin.processAssignments()" method.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the heap size, in the beginning.
- Update Gradle.
- Code cleanup.
2021-12-03 04:09:40 +02:00
Lampros Smyrnaios 212f8f377d - Set the "ConnSupportUtils.shouldBlockMost5XXDomains" to "false" and call the "LoaderAndChecker.setCouldRetryRegex()" method. The above, make sure that for HTTP-5XX-errors, only the 511-domains get blocked and only the 511-urls get labeled with "noRetry".
- Improve performance and reduce memory consumption, by calling the "ConnSupportUtils.setKnownMimeTypes()" method only once, in the constructor-method.
- Code cleanup.
2021-11-30 06:57:51 +02:00
Lampros Smyrnaios 6355b3e397 - Increase the "PublicationsRetriever.threadsMultiplier" to "6", as the threads are mostly network-blocked.
- Make sure the "maven" package is installed before compiling the "PublicationsRetriever" library.
- Update dependencies.
2021-11-30 01:02:06 +02:00
Lampros Smyrnaios 045788c728 - Use the "Timestamp" data-type instead of the "Date", in order to include more information.
- Code cleanup.
2021-11-27 02:37:33 +02:00
Lampros Smyrnaios 20b71164d5 - The worker will store the files in its local file-system and will send them to the controller in batches, after the latter requests them. When all files from a given assignments-num are sent, the files will be deleted from the Worker, in a scheduled-job.
- Implement the "getFullTexts"-endpoint, which returns the requested full-texts in a zip file.
- Implement the "getFullText"-endpoint, which returns the requested full-text.
- Implement the "getHandledAssignmentsCounts"-endpoint which returns the assignments-numbers, which were handled by that worker.
- Make sure each urlReport has the same "Date" for a given assignments-number. Also, make sure the "size" and "hash" have a "null" value, in case the full-text was not found.
- Check and log thread-pool shutdown errors.
- Add the stack-trace in the error-logs, instead of the Stderr.
- Update SpringBoot dependency.
- Change log levels.
- Code cleanup.
2021-11-26 17:04:31 +02:00
Lampros Smyrnaios 3220c97373 - Improve performance when requesting, processing and posting requests.
- Fix a bug, causing degraded performance when processing more than 3000 assignments.
- Fix the progress percentage shown in the logs.
- Avoid a potential NPE when processing a broken "Assignment" object.
- Update Spring to v.2.5.6.
- Code cleanup.
2021-10-30 17:14:18 +03:00
Lampros Smyrnaios 0f12a9305c - Decrease the time interval for the scheduled task "handleNewAssignments". This helps to reduce the "dead-time" between reporting the current assignments and requesting the new ones.
- Avoid a potential NPE when giving information about the received AssignmentRequest.
- Log and return, when the received assignments-list is empty.
- Improve some logging-messages.
- Update the logs' fileName and change the preferred appender to "File".
- Code cleanup.
2021-10-14 03:03:47 +03:00
Lampros Smyrnaios 380137fbff - Add an HTTP-error-handler in "AssignmentHandler.requestAssignments()".
- Increase the "requestConnectTimeoutDuration" and the "requestReadTimeoutDuration".
- Increase project's version to "1.0.0-SNAPSHOT".
- Update dependencies.
- Code cleanup.
2021-10-11 13:27:40 +03:00
Lampros Smyrnaios 5386035397 - Add timeDurationLimits to wait for the requested Assignments to come from the Controller.
- Make sure that the test-Results do not get posted to the Controller and written to the database.
- Improve error-handling in "AssignmentHandler.requestAssignments()".
2021-09-23 16:23:49 +03:00
Lampros Smyrnaios 2ffb44a615 - Update the "installAndRun.sh":
--Ask the user to give the "workerId" and the "controllerBaseUrl".
--Make sure the "libs" directory is created, if not exists.
--Make sure the "unzip" package is installed.
- Change the data-type of the "UrlReport.status" to be "enum StatusType", in order to increase consistency and comparability.
- Update the guidelines in the README.
2021-09-22 16:36:48 +03:00
Lampros Smyrnaios 61597d1627 - Read the Controller's url from a file, when starting the Application.
- Switch the "AssignmentsHandler.askForTest" to "false".
- Get the size and the hash of a docFile which is previously downloaded by another ID in that batch.
- Reset the "AssignmentHandler.urlReports" list after posting the results to the Controller.
- Enhance logging and comments.
- Add more guidelines in the README.
- Disable the scheduled test-live job.
- Code cleanup.
2021-09-21 16:21:39 +03:00
Lampros Smyrnaios 32aff8c44a - Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want.
- Fix a missing "mimeType"-assignment.
- Add gitignore.
2021-09-09 16:28:58 +03:00
Lampros Smyrnaios b2788d31a9 - Integrate the latest changes from the "PublicationsRetriever"-plugin. The fileSize and the fileHash are computed inside the plugin now.
- Make the "mimeType" "null", when no docFile was retrieved.
- Signal the scheduler that the worker is ready for work, when it has finished processing but not yet posted the previous data.
- Fix a minor bug; now return "false" when there is any problem with the url of a specific task.
- Avoid memory re-allocations for "callableTasks".
2021-09-08 05:02:14 +03:00
Lampros Smyrnaios b6d66653f7 - Integrate the latest changes from the "PublicationsRetriever"-plugin.
- Update dependencies.
2021-09-01 19:42:32 +03:00