Various fixes in the stats wf #430

antonis.lempesis · 2024-04-24T11:18:07+02:00

antonis.lempesis commented

2024-04-24 11:18:07 +02:00

No description provided.

antonis.lempesis added 10 commits 2024-04-24 11:18:08 +02:00

1f5aba12fa slight optimization in indi_pub_gold_oa definition

27d22bd8f9 slight optimization in indi_pub_gold_oa definition

308ae580a9 slight optimization in indi_pub_gold_oa definition

e728a0897c fixed the definition of indi_pub_bronze_oa

43d05dbebb fixed the definition of result_country

0c71c58df6 fixed the definition of gold_oa

c3fe9662b2 all indicator tables are now stored as parquet

b52a5a753b Merge remote-tracking branch 'upstream/beta' into beta

d2649a1429 increased the jvm ram

49af2e5740 Miscellaneous updates to the copying operation to Impala Cluster:

- Update the algorithm for creating views that depend on other views; overcome some bash-instabilities.
- Upon any error, fail the whole process, not just the current DB-creation, as those errors usually indicate a bug in the initial DB-creation, that should be fixed immediately.
- Enhance parallel-copy of large files by "hadoop distcp" command.
- Reduce the "invalidate metadata" commands to just the current DB's tables, in order to eliminate the general overhead on Impala.
- Show the number of tables and views in the logs.
- Fix some log-messages.

antonis.lempesis added 1 commit 2024-05-08 12:18:10 +02:00

90a4fb3547 fixed typos

antonis.lempesis added 1 commit 2024-05-08 12:43:13 +02:00

0cada3cc8f every step is run in the analytics queue. Hardcoded for now, will make a parameter later

claudio.atzori merged commit 908ed9da7a into beta

2024-05-08 13:41:03 +02:00

claudio.atzori referenced this issue from a commit

2024-05-08 13:41:04 +02:00

Merge pull request 'Various fixes in the stats wf' (#430) from antonis.lempesis/dnet-hadoop:beta into beta

claudio.atzori referenced this pull request

2024-05-21 14:10:55 +02:00

beta to master May 2024 #437