Lampros Smyrnaios 3988eb3a48 - Use a separate HDFS sub-dir for every assignments-batch, in order to avoid any disrruptancies from multiple threads moving parquet-files from the same sub-dir. Multiple batches from the same worker may be processed at the same time. These sub-dirs are deleted afterwards. ... - Treat the "contains no visible files" situation as an error. In which case the assignments-data is presumed to not have been inserted to the database tables. - Code polishing/cleanup.		2023-05-27 02:36:05 +03:00
..
urls_controller	- Use a separate HDFS sub-dir for every assignments-batch, in order to avoid any disrruptancies from multiple threads moving parquet-files from the same sub-dir. Multiple batches from the same worker may be processed at the same time. These sub-dirs are deleted afterwards.	2023-05-27 02:36:05 +03:00