Commit Graph

11 Commits

Author SHA1 Message Date
Lampros Smyrnaios 26cbb83b51 - Add the "shutdownWorker"-endpoint to accept requests for shutting-down the Worker, gracefully, after it completes its current work (including sending the publications-files to the Controller). A user-defined "auth-code" is required.
- Add the "cancelShutdownWorker"-endpoint to cancel a previous "shutdownWorker"-request. A user-defined "auth-code" is required.
2022-06-22 18:53:27 +03:00
Lampros Smyrnaios edbf6461d5 - Refactor the scheduling of the "handleNewAssignments()" task. Spring already waits for the last task to get finished, before running the new one (unless Async is specifically enabled), so the "isAvailableForWork" didn't do anything (thus the bug described in a previous commit was never going to appear). Also, now we set to request the new assignments-batch immediately after the last one is finished (not after 15 mins), while dealing with potential continuous connection-errors.
- Avoid running the "deleteHandledAssignmentsFullTexts()" scheduled task on application's start.
- Optimize assignment of "requestUrl".
- Add clarity in the scheduled tasks, by using "fixedDelay" instead of "fixedRate", to signify that the time specified is counted right from the time the last task is finished (even though without enabling the "Async" there is no "danger" of running them in parallel).
- Code cleanup.
2022-02-21 12:48:21 +02:00
Lampros Smyrnaios c46c8c448a - Upgrade the zip-file delivery by using the "InputStreamResource". This way is more reliable, have better performance and uses less memory.
- Use the "InputStreamResource" also in "get(single)FullText"-endpoint, in order to avoid loading a big full-text file in memory.
- Decrease the system-reserved memory by 128 MB.
- Fix path-variable regexes for "getFullText"-endpoint.
- Optimize imports.
- Code cleanup.
2021-12-17 08:25:54 +02:00
Lampros Smyrnaios fd5b56e3c6 - Allow the user to set the "maxAssignmentsLimitPerBatch" value.
- Set increased lower and upper limits for the Java Heap Size.
- Update the "ServerBaseURL" to the Public IP Address of the machine which is running the app.
- Improve two log-messages.
2021-12-07 00:52:40 +02:00
Lampros Smyrnaios 045788c728 - Use the "Timestamp" data-type instead of the "Date", in order to include more information.
- Code cleanup.
2021-11-27 02:37:33 +02:00
Lampros Smyrnaios 20b71164d5 - The worker will store the files in its local file-system and will send them to the controller in batches, after the latter requests them. When all files from a given assignments-num are sent, the files will be deleted from the Worker, in a scheduled-job.
- Implement the "getFullTexts"-endpoint, which returns the requested full-texts in a zip file.
- Implement the "getFullText"-endpoint, which returns the requested full-text.
- Implement the "getHandledAssignmentsCounts"-endpoint which returns the assignments-numbers, which were handled by that worker.
- Make sure each urlReport has the same "Date" for a given assignments-number. Also, make sure the "size" and "hash" have a "null" value, in case the full-text was not found.
- Check and log thread-pool shutdown errors.
- Add the stack-trace in the error-logs, instead of the Stderr.
- Update SpringBoot dependency.
- Change log levels.
- Code cleanup.
2021-11-26 17:04:31 +02:00
Lampros Smyrnaios 2ffb44a615 - Update the "installAndRun.sh":
--Ask the user to give the "workerId" and the "controllerBaseUrl".
--Make sure the "libs" directory is created, if not exists.
--Make sure the "unzip" package is installed.
- Change the data-type of the "UrlReport.status" to be "enum StatusType", in order to increase consistency and comparability.
- Update the guidelines in the README.
2021-09-22 16:36:48 +03:00
Lampros Smyrnaios 61597d1627 - Read the Controller's url from a file, when starting the Application.
- Switch the "AssignmentsHandler.askForTest" to "false".
- Get the size and the hash of a docFile which is previously downloaded by another ID in that batch.
- Reset the "AssignmentHandler.urlReports" list after posting the results to the Controller.
- Enhance logging and comments.
- Add more guidelines in the README.
- Disable the scheduled test-live job.
- Code cleanup.
2021-09-21 16:21:39 +03:00
Lampros Smyrnaios 6307cda23a - Refactor the assignments-handling. In order to match with the database schema, now the AssignmentRequest returns a list of Assignments instead of a single assignment having a list of Tasks.
- Cleanup the members of the "Payload" model.
2021-07-05 15:00:29 +03:00
Lampros Smyrnaios 82e12655e7 - Add an "AssignmentHandler", which retrieves the assignment from the controller and categorises the tasks using their datasource. In the future, it will execute the tasks of the assignment, using different plugins. It runs upon the Application start and also every 30 mins (if no other job is in execution).
- Add the "isWorkerAvailableForWork-endpoint.
2021-05-20 03:28:48 +03:00
Lampros Smyrnaios 08eabe6f08 Initial commit of UrlsWorker. 2021-03-16 18:38:53 +02:00