Lampros Smyrnaios
8bc5cc35e2
- Optimize writing to the Bulk-import-report file.
...
- Show the IP of the worker which posts a "workerShutdownReport".
- Code polishing.
2024-03-22 17:50:55 +02:00
Lampros Smyrnaios
1048463ca0
- Improve error-handling in "S3ObjectStore.emptyBucket()".
...
- Change some log-levels.
- Code polishing.
2024-03-11 16:17:32 +02:00
Lampros Smyrnaios
d90ad51609
Add the "shutdownAllWorkersGracefully" and "cancelShutdownAllWorkersGracefully" endpoints, in order to be able to shut them down at once and update them, without shutting down the whole Service. So in this case the bulk-import procedures will continue to work.
2023-11-29 16:45:58 +02:00
Lampros Smyrnaios
bdf834c439
- Refactor the "shutdown" script to do an orderly-shutdown, by default, by calling the "shutdownService" endpoint. In case a "force-shutdown" is needed, that can be requested with a cmd-argument.
...
- Fix not updating the "UrlsController.numOfWorkers" correctly.
- Code polishing.
2023-10-23 17:19:29 +03:00
Lampros Smyrnaios
96c11ba4b8
- Add a missing change.
...
- Code optimization and polishing.
- Update dependencies.
2023-10-04 16:17:12 +03:00
Lampros Smyrnaios
aa3f32f3da
- Make sure the given number of threads, given by the user is above zero.
...
- Adjust the number and size of log files.
- Update Spring Boot.
- Code polishing.
2023-08-30 14:02:54 +03:00
Lampros Smyrnaios
798fa09d68
- Identify and handle a possible Worker-crash, in "UrlsServiceImpl.postReportResultToWorker()".
...
- Add/Improve some log messages.
- Update and cleanup dependencies.
- Code polishing.
2023-06-15 23:19:36 +03:00
Lampros Smyrnaios
6669dc61bf
- Increase the initialDelay for the "checkIfServiceIsReadyForShutdown" scheduled-task, in production, to 10 minutes.
...
- Code polishing.
2023-06-06 16:49:53 +03:00
Lampros Smyrnaios
54685bbe9a
- Avoid sending "cancelShutdown" requests to already shutDown Workers.
...
- Optimize performance of the code running right before the "postShutdownOrCancelRequestToWorker".
- Show which Workers have already shutdown and as a result a "postShutdownOrCancelRequestToWorker" will not be performed on them.
2023-05-29 13:41:37 +03:00
Lampros Smyrnaios
f9c6bad768
Do not send shutDownRequests to workers which have already shutdown.
2023-05-29 12:42:54 +03:00
Lampros Smyrnaios
3988eb3a48
- Use a separate HDFS sub-dir for every assignments-batch, in order to avoid any disrruptancies from multiple threads moving parquet-files from the same sub-dir. Multiple batches from the same worker may be processed at the same time. These sub-dirs are deleted afterwards.
...
- Treat the "contains no visible files" situation as an error. In which case the assignments-data is presumed to not have been inserted to the database tables.
- Code polishing/cleanup.
2023-05-27 02:36:05 +03:00
Lampros Smyrnaios
0ea3e2de24
Add the "shutdownService" and "cancelShutdownService" endpoints. The Controller sends the related requests to the Workers and shutdowns gracefully, after all workers have shutdown.
2023-05-24 13:42:29 +03:00
Lampros Smyrnaios