lsmyrnaios
/
UrlsWorker
1
0
Fork 1
Code Issues Pull Requests Releases 13 Wiki Activity
47 Commits 1 Branch 13 Tags 1.3 MiB
Commit Graph

4 Commits

Author SHA1 Message Date
Lampros Smyrnaios c46c8c448a - Upgrade the zip-file delivery by using the "InputStreamResource". This way is more reliable, have better performance and uses less memory. 
- Use the "InputStreamResource" also in "get(single)FullText"-endpoint, in order to avoid loading a big full-text file in memory.
- Decrease the system-reserved memory by 128 MB.
- Fix path-variable regexes for "getFullText"-endpoint.
- Optimize imports.
- Code cleanup.
 2021-12-17 08:25:54 +02:00
Lampros Smyrnaios 859f850f56 - Improve Zip-file delivery and significantly decrease memory consumption, by streaming it, instead of loading the whole file in memory before sending it to the Controller. 
- Fix not closing the inputStream of the zip-file.
- Count and show the number of files which were zipped in each batch.
 2021-12-13 15:29:03 +02:00
Lampros Smyrnaios ce49bff50e - Reduce memory consumption when loading a zipFile. 
- Check whether the "zipBatchCounter" is larger than the "totalZipBatches".
- Improve the "failed tasks" log-message.
 2021-12-03 16:29:16 +02:00
Lampros Smyrnaios 20b71164d5 - The worker will store the files in its local file-system and will send them to the controller in batches, after the latter requests them. When all files from a given assignments-num are sent, the files will be deleted from the Worker, in a scheduled-job. 
- Implement the "getFullTexts"-endpoint, which returns the requested full-texts in a zip file.
- Implement the "getFullText"-endpoint, which returns the requested full-text.
- Implement the "getHandledAssignmentsCounts"-endpoint which returns the assignments-numbers, which were handled by that worker.
- Make sure each urlReport has the same "Date" for a given assignments-number. Also, make sure the "size" and "hash" have a "null" value, in case the full-text was not found.
- Check and log thread-pool shutdown errors.
- Add the stack-trace in the error-logs, instead of the Stderr.
- Update SpringBoot dependency.
- Change log levels.
- Code cleanup.
 2021-11-26 17:04:31 +02:00