Commit Graph

17 Commits

Author SHA1 Message Date
Lampros Smyrnaios 92d011e8a0 - Make sure the handled assignments - full-texts are deleted before the application exits.
- When the user sets the "maxAssignmentsBatchesToHandleBeforeRestart" above zero, shutdown immediately after the last assignments-batch. Do not wait for the next scheduled check.
- Allow the user to set the "maxAssignmentsBatchesToHandleBeforeRestart" in the "installAndRun.sh" script.
- Increase the "fixedRate" for the "ScheduledTasks.deleteHandledAssignmentsFullTexts()" method to 12 hours.
- Update README.md
2021-12-31 04:09:05 +02:00
Lampros Smyrnaios 1ddfd34236 - Allow the user to set a maximum number of assignments-batches for the Worker to handle. After handling those batches, the Worker will shut down. A number of < 0 > indicates an infinite number of batches.
- Avoid converting the zero fileSize to < null >. Now, the default value is < null >, so the zero-value will indicate a zero-byte file.
- Update dependencies.
- Code cleanup.
2021-12-24 00:12:34 +02:00
Lampros Smyrnaios c46c8c448a - Upgrade the zip-file delivery by using the "InputStreamResource". This way is more reliable, have better performance and uses less memory.
- Use the "InputStreamResource" also in "get(single)FullText"-endpoint, in order to avoid loading a big full-text file in memory.
- Decrease the system-reserved memory by 128 MB.
- Fix path-variable regexes for "getFullText"-endpoint.
- Optimize imports.
- Code cleanup.
2021-12-17 08:25:54 +02:00
Lampros Smyrnaios 4fb5becace - Increase the system-reserved memory, in "installAndRun.sh".
- Fix not closing the zip-entry in case of an error.
2021-12-17 00:26:47 +02:00
Lampros Smyrnaios 82d69f3bf5 - Calculate and set the max heap size with respect to the system resources, in "installAndRun.sh".
- Fix not setting the right "Error"-members when the docUrl was found, but the full-text was not retrieved.
- Set a "couldRetry"-indication in the "Error"-class, when the full-text was retrieved, as, in general, it could be retried to give the same successful result.
- Update the "docFileNotRetrieved"-check to use the standardized string.
- Eliminate some possible NPEs.
- Update Gradle.
2021-12-16 02:04:05 +02:00
Lampros Smyrnaios fd5b56e3c6 - Allow the user to set the "maxAssignmentsLimitPerBatch" value.
- Set increased lower and upper limits for the Java Heap Size.
- Update the "ServerBaseURL" to the Public IP Address of the machine which is running the app.
- Improve two log-messages.
2021-12-07 00:52:40 +02:00
Lampros Smyrnaios 018326eedd - Optimize the "FileZipper.zipMultipleFilesAndGetZip()" and "FileZipper.zipAFile()" methods.
- Improve the "getMultipleFullTexts"-endpoint. Check if the "fileNamesWithExtensions"-list is empty. Check if the baseDir for the fullTexts of a given assignments-counter is missing.
- Optimize the "PublicationsRetrieverPlugin.processAssignments()" method.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the heap size, in the beginning.
- Update Gradle.
- Code cleanup.
2021-12-03 04:09:40 +02:00
Lampros Smyrnaios 212f8f377d - Set the "ConnSupportUtils.shouldBlockMost5XXDomains" to "false" and call the "LoaderAndChecker.setCouldRetryRegex()" method. The above, make sure that for HTTP-5XX-errors, only the 511-domains get blocked and only the 511-urls get labeled with "noRetry".
- Improve performance and reduce memory consumption, by calling the "ConnSupportUtils.setKnownMimeTypes()" method only once, in the constructor-method.
- Code cleanup.
2021-11-30 06:57:51 +02:00
Lampros Smyrnaios 6355b3e397 - Increase the "PublicationsRetriever.threadsMultiplier" to "6", as the threads are mostly network-blocked.
- Make sure the "maven" package is installed before compiling the "PublicationsRetriever" library.
- Update dependencies.
2021-11-30 01:02:06 +02:00
Lampros Smyrnaios 380137fbff - Add an HTTP-error-handler in "AssignmentHandler.requestAssignments()".
- Increase the "requestConnectTimeoutDuration" and the "requestReadTimeoutDuration".
- Increase project's version to "1.0.0-SNAPSHOT".
- Update dependencies.
- Code cleanup.
2021-10-11 13:27:40 +03:00
Lampros Smyrnaios 42f8cb769d Update "installAndRun.sh": check if a gradle installation with the given version already exists, before downloading and installing gradle. 2021-10-11 11:19:52 +03:00
Lampros Smyrnaios e091a029a8 - Fix the project's name inside "settings.gradle".
- Fix the "change-dir" to the "libs"-directory in "installAndRun.sh"
2021-09-22 17:06:30 +03:00
Lampros Smyrnaios 2ffb44a615 - Update the "installAndRun.sh":
--Ask the user to give the "workerId" and the "controllerBaseUrl".
--Make sure the "libs" directory is created, if not exists.
--Make sure the "unzip" package is installed.
- Change the data-type of the "UrlReport.status" to be "enum StatusType", in order to increase consistency and comparability.
- Update the guidelines in the README.
2021-09-22 16:36:48 +03:00
Lampros Smyrnaios 32aff8c44a - Update the "installAndRun.sh" script to be able to just run the app (without re-installing), if you want.
- Fix a missing "mimeType"-assignment.
- Add gitignore.
2021-09-09 16:28:58 +03:00
Lampros Smyrnaios 6fd9eed1ec - Eliminate some warnings, by excluding an inner dependency.
- Comment-out some debugging gradle commands.
2021-09-02 18:35:47 +03:00
Lampros Smyrnaios b6d66653f7 - Integrate the latest changes from the "PublicationsRetriever"-plugin.
- Update dependencies.
2021-09-01 19:42:32 +03:00
Lampros Smyrnaios 62ce7ee4a5 - Process the Error of PDF-aggregation. Distinguish between "couldRetry" and "noRetry" cases.
- Add a "test"-switch in order to easily switch between test and normal mode.
- Fix an NPE, when requesting for the "AssignmentRequest".
- Upgrade the "installPublicationsRetriever.sh" to "installAndRun.sh", which takes care of everything.
- Define the newest SpringBoot-version in "build.gradle".
- Code cleanup.
2021-08-05 15:09:28 +03:00