Commit Graph

12 Commits

Author SHA1 Message Date
Lampros Smyrnaios 20b71164d5 - The worker will store the files in its local file-system and will send them to the controller in batches, after the latter requests them. When all files from a given assignments-num are sent, the files will be deleted from the Worker, in a scheduled-job.
- Implement the "getFullTexts"-endpoint, which returns the requested full-texts in a zip file.
- Implement the "getFullText"-endpoint, which returns the requested full-text.
- Implement the "getHandledAssignmentsCounts"-endpoint which returns the assignments-numbers, which were handled by that worker.
- Make sure each urlReport has the same "Date" for a given assignments-number. Also, make sure the "size" and "hash" have a "null" value, in case the full-text was not found.
- Check and log thread-pool shutdown errors.
- Add the stack-trace in the error-logs, instead of the Stderr.
- Update SpringBoot dependency.
- Change log levels.
- Code cleanup.
2021-11-26 17:04:31 +02:00
Lampros Smyrnaios 3220c97373 - Improve performance when requesting, processing and posting requests.
- Fix a bug, causing degraded performance when processing more than 3000 assignments.
- Fix the progress percentage shown in the logs.
- Avoid a potential NPE when processing a broken "Assignment" object.
- Update Spring to v.2.5.6.
- Code cleanup.
2021-10-30 17:14:18 +03:00
Lampros Smyrnaios 380137fbff - Add an HTTP-error-handler in "AssignmentHandler.requestAssignments()".
- Increase the "requestConnectTimeoutDuration" and the "requestReadTimeoutDuration".
- Increase project's version to "1.0.0-SNAPSHOT".
- Update dependencies.
- Code cleanup.
2021-10-11 13:27:40 +03:00
Lampros Smyrnaios 6fd9eed1ec - Eliminate some warnings, by excluding an inner dependency.
- Comment-out some debugging gradle commands.
2021-09-02 18:35:47 +03:00
Lampros Smyrnaios b6d66653f7 - Integrate the latest changes from the "PublicationsRetriever"-plugin.
- Update dependencies.
2021-09-01 19:42:32 +03:00
Lampros Smyrnaios 62ce7ee4a5 - Process the Error of PDF-aggregation. Distinguish between "couldRetry" and "noRetry" cases.
- Add a "test"-switch in order to easily switch between test and normal mode.
- Fix an NPE, when requesting for the "AssignmentRequest".
- Upgrade the "installPublicationsRetriever.sh" to "installAndRun.sh", which takes care of everything.
- Define the newest SpringBoot-version in "build.gradle".
- Code cleanup.
2021-08-05 15:09:28 +03:00
Lampros Smyrnaios 6cc2673fca - Add the ability to upload the files on an S3-ObjectStore.
- Change the server's port and the port of the controller-api.
- Update dependencies.
2021-07-29 09:01:53 +03:00
Lampros Smyrnaios f6e53ca289 Integrate the "PublicationsRetriever" program as a plugin, which downloads the full-texts of the publications. Afterwards, the retrieved data info is transferred to the Controller.
The "PublicationsRetriever" can be installed locally as a library, using the "installPublicationsRetriever.sh" script.
2021-06-22 05:58:07 +03:00
Lampros Smyrnaios 5f3409e072 - Update Spring and add the "gradle-wrapper.properties" file which defines the gradle version.
- Improve an info-logging message and cleanup the code.
2021-06-10 14:29:20 +03:00
Lampros Smyrnaios 82e12655e7 - Add an "AssignmentHandler", which retrieves the assignment from the controller and categorises the tasks using their datasource. In the future, it will execute the tasks of the assignment, using different plugins. It runs upon the Application start and also every 30 mins (if no other job is in execution).
- Add the "isWorkerAvailableForWork-endpoint.
2021-05-20 03:28:48 +03:00
Lampros Smyrnaios a4c97dffbf - Add the "Datasource" class in the "Task" class and include it in the Assignment that the worker retrieves.
- Update dependencies.
2021-05-20 02:58:08 +03:00
Lampros Smyrnaios 08eabe6f08 Initial commit of UrlsWorker. 2021-03-16 18:38:53 +02:00