lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 10 Wiki Activity
160 Commits
1 Branch
10 Tags
1.3 MiB
master
2.1.8
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1
2.0
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
Commit Graph

160 Commits (master)
 

Author SHA1 Message Date
Lampros Smyrnaios 736d0f8526 Add a missing change in logback-spring.xml. 3 months ago
Lampros Smyrnaios cb736a8d66 Add the Jenkins' build-status badge in README. 3 months ago
Lampros Smyrnaios 5d7465df3c Add some gradle files to be used by Jenkins. 3 months ago
Lampros Smyrnaios 24c4a75acf - Use the "RollingFile" logs-appender by default. 
- Set the next version.
 3 months ago
Lampros Smyrnaios 50d756d582 - Automatically use the latest version of "publications_retriever" software from the Nexus maven-repository. 
- Update Gradle.
- Update License.
- Configure the destination of the logs in the "application.properties" file.
 3 months ago
Lampros Smyrnaios 3909104a1b - Update a dependency. 
- Set new version.
 3 months ago
Lampros Smyrnaios c4770ee716 Set new version. 4 months ago
Lampros Smyrnaios 066d6f665f - Take into account the new "errorMsg" value returned by "LoaderAndChecker.getWasValidAndCouldRetry()". 
- Update dependencies.
 4 months ago
Lampros Smyrnaios bad9544c58 - Improve error-handling. 
- Improve a log-message.
 5 months ago
Lampros Smyrnaios 5a9e7228ae - Set the upcoming version. 
- Update dependencies.
 5 months ago
Lampros Smyrnaios 9073f56227 Revert the "read-timeout" value back to 1 hour, as there is no that big of a problem with the load of either server, it's a frequent network-"lag" that causes the issue, which is not solved even with 2 hours of waiting. 6 months ago
Lampros Smyrnaios 69ea5b6d19 - Increase the "ReadTimeout" to 2 hours, as the Worker struggles to get the assignments-data in time. 
- Revert the change about special handling of the "RestClientException". The exMsg was appearing in a different line, in the logs, and was a "SocketTimeoutException".
 6 months ago
Lampros Smyrnaios bfa76e9484 - Show the full stacktrace in the weird case of a "RestClientException" without an exception-message. Also, in this case, retry immediately, as there is no long-lasting network problem that requires some time between requests, but most probably a random interruption. 
- Code polishing.
 6 months ago
Lampros Smyrnaios 10e39d79a4 - Improve a log-message. 
- Update dependencies.
 6 months ago
Lampros Smyrnaios 1b45f384a7 - In case a faulty "assignmentsCounter" was given to the "addReportResultToWorker"-endpoint, then return an explanatory error-message along with the HTTP-404 error. 
- Update Gradle.
 7 months ago
Lampros Smyrnaios 01e378ea66 - Add progress-report-log for assignments-processing. 
- Code polishing.
 7 months ago
Lampros Smyrnaios 18cc9e0e68 - Improve error-handling in file-compression. 
- Update dependencies.
 7 months ago
Lampros Smyrnaios 2895668417 - Add LICENSE. 
- Code polishing.
 7 months ago
Lampros Smyrnaios 49cd0c19c2 - Increase the "hoursToWaitBeforeDeletion" to 48. 
- Adjust the number and size of log files.
 8 months ago
Lampros Smyrnaios e85282d35b Update the "addReportResultToWorker"-endpoint to check if the given "assignmentsCounter" was handled by that worker, without considering the related full-texts directory, since that may have been deleted in the meantime. 8 months ago
Lampros Smyrnaios b579296ada - Code optimization and polishing. 
- Update dependencies.
 8 months ago
Lampros Smyrnaios dc97b323c9 - Show a warning, if the "numOfUnretrievedFiles" is over 50. 
- Delete gradle .zip file after installation.
- Code polishing.
 9 months ago
Lampros Smyrnaios 088cf73b30 - Update dependencies. 
- Code optimization and polishing.
 9 months ago
Lampros Smyrnaios 952bf7c035 - Update dependencies. 
- Code polishing.
 10 months ago
Lampros Smyrnaios 33df46f6f5 - Improve README. 
- Update and cleanup dependencies.
- Code polishing.
 10 months ago
Lampros Smyrnaios 9c897b8bf4 - Make use of the new Normalizer utilized by the PublicationRetriever plugin. 
- Code polishing.
 11 months ago
Lampros Smyrnaios 2aedae2367 - In case a serious error happened while processing the assignments, instead of shutting down immediately, now the Worker shuts down the executor service, registers that it will shut down soon and waits for the Controller to retrieve the already downloaded full-text files. 
- In case the full-texts' subdirectory could not be created, then terminate the "handleAssignment" method immediately. No posting of a faulty workerReport to the Controller should happen.
- Code polishing.
 11 months ago
Lampros Smyrnaios 4a95826f58 - Avoid processing the assignments, for which the assignments_full-texts subdirectory cannot be created. 
- Avoid a double-log.
 11 months ago
Lampros Smyrnaios 7f3ca80959 Bypass url-canonicalization for urls containing certain uncommon characters which cause the urls to get rejected. 11 months ago
Lampros Smyrnaios a9b1b20a51 - Prevent running out of space, by checking the available free space and stalling the acquisition of new assignments until more free space becomes available. 
- Fix missing change.
 11 months ago
Lampros Smyrnaios 84f29ea7e0 Update versioning. 11 months ago
Lampros Smyrnaios 0908dcab8a Use a single "restTemplate" object, with the same timeouts (a bit increased from the old requestRestTemplate, to account for a possible overloaded Controller), since we no longer need to wait for hours until the workerReport is processed by the Controller. 11 months ago
Lampros Smyrnaios 2b69733912 - Increase the test-delays of the scheduled tasks. 
- Update dependencies.
 11 months ago
Lampros Smyrnaios f57314908a - Improve elapsed time precision for the "lastModified" metadata of the assignments-fulltext subDirectories. 
- Code polishing.
 11 months ago
Lampros Smyrnaios 1bf27a5a4e - Fix a bug, which caused the old full-text files to not be deleted. 
- Reduce the "InitialDelay" for the "checkIfShouldShutdown" scheduler.
 11 months ago
Lampros Smyrnaios 0ca02f3587 Change the delay values of scheduledTasks to production ones. 11 months ago
Lampros Smyrnaios bfa569685a - Use the "POST" method for shutdown and cancelShutdown requests. 
- Polish some messages.
 11 months ago
Lampros Smyrnaios 9fdaa9503b - Delete any left-over full-texts after 36 hours. 
- Upon shutting down, post a "shutdownReport" to the Controller.
 11 months ago
Lampros Smyrnaios 903032f454 - After a WorkerReport has been sent, ask for new assignments immediately. So, the Worker does not have to wait for hours for the Controller to check for duplicate files in the DB, retrieve and upload the full-texts and insert the records to the DB. 
- Special care is taken to delete the delivered full-texts as soon as possible.
- Write the workerReport to a json-file, in case something goes wrong, and keep it until the Controller notifies the Worker that the processing was successful.
 11 months ago
Lampros Smyrnaios 9cb43b3d94 - Improve startup speed, by using a faster remote server to get the host's machine public IP. This also reduces the risk of not being able to get the public IP at all. 
- Set the App to gracefully shut down the WebServer and wait up to 2 minutes.
- Increase the waiting time for the "PublicationsRetriever.executor" to shut down, to 2 minutes.
 11 months ago
Lampros Smyrnaios 4d90846261 - In case the specified "controllerIP" is actually a domain-name, find its IP-address, so that a proper IP-to-IP comparison can be performed and the "securityChecks" can pass. 
- Increase the "read-timeout" when searching for the host's machine public-IP.
- Update dependencies.
- Code polishing.
 11 months ago
Lampros Smyrnaios bd0ead816d Make the value of time-out for "restTemplateForReport", to scale along the "maxAssignmentsLimitPerBatch". 11 months ago
Lampros Smyrnaios 93d1aa9588 - Fix a missing change. 
- Add todo.
 12 months ago
Lampros Smyrnaios cc55354e73 Show the worker-id when the worker starts. 12 months ago
Lampros Smyrnaios 714938531b - Add the time-zone in the logs. 
- Code polishing.
 12 months ago
Lampros Smyrnaios 29a54f0b30 Remove the "shutDownOrCancelCode" from security checks, since we have an IP whitelisting mechanism in place. 12 months ago
Lampros Smyrnaios 4eac7c5c66 Fix typo in property's name. 1 year ago
Lampros Smyrnaios 0ea7bccadb Leave the Max-Heap-Size to 8Gb, we assume that enough swap space will be available on the host. 
We can still override the max-heap-size if desired.
 1 year ago
Lampros Smyrnaios d5a997ad3d Use restTemplates with different read timeouts depending on the operation. For the assignments-request we need a shorter read timeout, than the one we need for the worker-report. This guarantees that the connection does not hungs for so long, when the Controller crashes before sending the assignments. 1 year ago
Lampros Smyrnaios 53ab51922a Allow shutdown requests from the Controller. 1 year ago