lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 17 Wiki Activity
155 Commits 1 Branch 17 Tags 1.5 MiB
Commit Graph

23 Commits

Author SHA1 Message Date
Lampros Smyrnaios 066d6f665f - Take into account the new "errorMsg" value returned by "LoaderAndChecker.getWasValidAndCouldRetry()". 
- Update dependencies.
 2023-12-18 15:17:51 +02:00
Lampros Smyrnaios 1b45f384a7 - In case a faulty "assignmentsCounter" was given to the "addReportResultToWorker"-endpoint, then return an explanatory error-message along with the HTTP-404 error. 
- Update Gradle.
 2023-10-06 15:45:53 +03:00
Lampros Smyrnaios 2895668417 - Add LICENSE. 
- Code polishing.
 2023-09-14 16:09:20 +03:00
Lampros Smyrnaios b579296ada - Code optimization and polishing. 
- Update dependencies.
 2023-08-28 16:11:26 +03:00
Lampros Smyrnaios 088cf73b30 - Update dependencies. 
- Code optimization and polishing.
 2023-07-27 17:46:17 +03:00
Lampros Smyrnaios 952bf7c035 - Update dependencies. 
- Code polishing.
 2023-07-06 13:22:09 +03:00
Lampros Smyrnaios 344bc46e08 Update Gradle. 2023-04-22 06:44:04 +03:00
Lampros Smyrnaios 0997558347 Update dependencies. 2023-04-20 15:39:15 +03:00
Lampros Smyrnaios 4da54e7a7d - Show a warning, in case the number of archived files is different from the number of requested files. 
- Code polishing.
- Update Gradle.
 2023-03-07 16:25:10 +02:00
Lampros Smyrnaios 9888349bef Update Gradle. 2023-02-20 19:14:16 +02:00
Lampros Smyrnaios 0626e85894 Update dependencies. 2023-02-15 16:18:33 +02:00
Lampros Smyrnaios 182d6153d4 - Set some optimization settings for gradle. 
- Fix error-handling in "installAndRun.sh".
- Update dependencies.
 2022-11-30 16:25:57 +02:00
Lampros Smyrnaios 01f12e2fe2 - Align with "PublicationsRetriever's" updated "couldRetry" and "wasValid" logic. 
- Update dependencies.
 2022-11-11 16:02:20 +02:00
Lampros Smyrnaios 31af0a81eb - Update the Worker's report to include the datasourceID for each record. It is used by the Controller inside the S3-fileNames. 
- Update dependencies.
 2022-04-01 19:42:32 +03:00
Lampros Smyrnaios 5fee05e994 Update dependencies. 2022-03-28 14:29:54 +03:00
Lampros Smyrnaios 3d1faf4a8a - Reduce memory-consumption in the long-run, by clearing some underlying data-structures after a threshold. 
- Update Gradle.
 2022-02-18 20:02:34 +02:00
Lampros Smyrnaios 1ddfd34236 - Allow the user to set a maximum number of assignments-batches for the Worker to handle. After handling those batches, the Worker will shut down. A number of < 0 > indicates an infinite number of batches. 
- Avoid converting the zero fileSize to < null >. Now, the default value is < null >, so the zero-value will indicate a zero-byte file.
- Update dependencies.
- Code cleanup.
 2021-12-24 00:12:34 +02:00
Lampros Smyrnaios 82d69f3bf5 - Calculate and set the max heap size with respect to the system resources, in "installAndRun.sh". 
- Fix not setting the right "Error"-members when the docUrl was found, but the full-text was not retrieved.
- Set a "couldRetry"-indication in the "Error"-class, when the full-text was retrieved, as, in general, it could be retried to give the same successful result.
- Update the "docFileNotRetrieved"-check to use the standardized string.
- Eliminate some possible NPEs.
- Update Gradle.
 2021-12-16 02:04:05 +02:00
Lampros Smyrnaios 018326eedd - Optimize the "FileZipper.zipMultipleFilesAndGetZip()" and "FileZipper.zipAFile()" methods. 
- Improve the "getMultipleFullTexts"-endpoint. Check if the "fileNamesWithExtensions"-list is empty. Check if the baseDir for the fullTexts of a given assignments-counter is missing.
- Optimize the "PublicationsRetrieverPlugin.processAssignments()" method.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the heap size, in the beginning.
- Update Gradle.
- Code cleanup.
 2021-12-03 04:09:40 +02:00
Lampros Smyrnaios 6355b3e397 - Increase the "PublicationsRetriever.threadsMultiplier" to "6", as the threads are mostly network-blocked. 
- Make sure the "maven" package is installed before compiling the "PublicationsRetriever" library.
- Update dependencies.
 2021-11-30 01:02:06 +02:00
Lampros Smyrnaios b6d66653f7 - Integrate the latest changes from the "PublicationsRetriever"-plugin. 
- Update dependencies.
 2021-09-01 19:42:32 +03:00
Lampros Smyrnaios 6cc2673fca - Add the ability to upload the files on an S3-ObjectStore. 
- Change the server's port and the port of the controller-api.
- Update dependencies.
 2021-07-29 09:01:53 +03:00
Lampros Smyrnaios 5f3409e072 - Update Spring and add the "gradle-wrapper.properties" file which defines the gradle version. 
- Improve an info-logging message and cleanup the code.
 2021-06-10 14:29:20 +03:00