5 Commits

Author	SHA1	Message	Date
Lampros Smyrnaios	abf0b69f29	Upgrade the copying operation to Impala Cluster: ... - Use only hive commands in the Ocean Cluster, as the "impala-shell" will be removed from there to free-up resources. - Hugely improve the performance in every aspect of the copying process: a) speedup file-transferring and DB-deletion, b) eliminate permissions-assignment, "load" operations and "use $db" queries, c) retry only the "create view" statements and only as long as they depend on other non-created views, instead of trying to recreate all tables and views 5 consecutive times. - Add error-checks for the creation of tables and views.	2024-04-11 17:12:12 +03:00
Lampros Smyrnaios	b7c8acc563	- Update the code which acquires the "IMPALA_HDFS_NODE", to test the "tmp"-dir, instead of the base-dir and introduce retries, to overcome potential file-system failures. This change was suggested by "Sebastian Tymkow" and "Grzegorz Bakalarski". ... - Fix typos.	2024-04-03 13:15:37 +03:00
Lampros Smyrnaios	bc8c97182d	Automatically select the ACTIVE HDFS NODE for Impala cluster, in all "copyDataToImpalaCluster.sh" scripts.	2024-03-26 13:01:12 +02:00
dimitrispie	8b2cbb611e	Changes to beta db names	2024-01-09 00:40:56 +02:00
dimitrispie	75bfde043c	Historical Snapshots Workflow ... Create historical snapshots db with parameters: hist_db_name=openaire_beta_historical_snapshots_xxx hist_db_name_prev=openaire_beta_historical_snapshots_xxx (previous run of wf) stats_db_name=openaire_beta_stats_xxx stats_irish_db_name=openaire_beta_stats_monitor_ie_xxx monitor_db_name=openaire_beta_stats_monitor_xxx monitor_db_prod_name=openaire_beta_stats_monitor monitor_irish_db_name=openaire_beta_stats_monitor_ie_xxx monitor_irish_db_prod_name=openaire_beta_stats_monitor_ie hist_db_prod_name=openaire_beta_historical_snapshots hist_db_shadow_name=openaire_beta_historical_snapshots_shadow hist_date=122023 hive_timeout=150000 hadoop_user_name=xxx resumeFrom=CreateDB	2024-01-04 15:11:04 +02:00