Various fixes in the stats wf #430

Merged
claudio.atzori merged 12 commits from antonis.lempesis/dnet-hadoop:beta into beta 2024-05-08 13:41:03 +02:00
No description provided.
antonis.lempesis added 10 commits 2024-04-24 11:18:08 +02:00
49af2e5740 Miscellaneous updates to the copying operation to Impala Cluster:
- Update the algorithm for creating views that depend on other views; overcome some bash-instabilities.
- Upon any error, fail the whole process, not just the current DB-creation, as those errors usually indicate a bug in the initial DB-creation, that should be fixed immediately.
- Enhance parallel-copy of large files by "hadoop distcp" command.
- Reduce the "invalidate metadata" commands to just the current DB's tables, in order to eliminate the general overhead on Impala.
- Show the number of tables and views in the logs.
- Fix some log-messages.
antonis.lempesis added 1 commit 2024-05-08 12:18:10 +02:00
antonis.lempesis added 1 commit 2024-05-08 12:43:13 +02:00
claudio.atzori merged commit 908ed9da7a into beta 2024-05-08 13:41:03 +02:00
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#430
No description provided.