D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 12 Projects 2 Releases 2 Wiki Activity

changes in copy script - beta2master #439

Merged
claudio.atzori merged 5 commits from antonis.lempesis/dnet-hadoop:beta into beta_to_master_may2024 2024-05-29 14:29:27 +02:00
Conversation 0 Commits 5 Files Changed 5 +243 -133
claudio.atzori commented 2024-05-27 11:53:56 +02:00
Owner
Copy Link
No description provided.
claudio.atzori added 4 commits 2024-05-27 11:53:56 +02:00
c7b32bbacc Update CopyDataToImpalaCluster: 
Update the code of acquiring the entities from Ocean cluster, through hive, in order to optimize the process and account for additional reserved keywords in Impala.

Co-authored-by: Antonis Lempesis <antleb@di.uoa.gr>
68322843e2 Small updates to the copy-operation to Impala Cluster: 
- Add a configuration-"switch" to control whether the script exits upon an error or not.
- Allow the script to exit when a table could not be created.
- Show the elapsed time for processing each database.
b48ed6e617 Change configuration in the copy-operation to Impala Cluster: 
Set the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" parameter to "false".
lsmyrnaios added 1 commit 2024-05-28 16:51:51 +02:00
e3f28338c1 Miscellaneous updates to the copying operation to Impala Cluster: 
- Assign the WRITE and EXECUTE permissions to the DBs' HDFS-directories, in order to be able to create tables on top of them, in the Impala Cluster.
- Make sure the "copydb" function returns early, when it encounters a fatal error, while respecting the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" config.
claudio.atzori merged commit b703f94f09 into beta_to_master_may2024 2024-05-29 14:29:27 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#439
No description provided.
Delete Branch "antonis.lempesis/dnet-hadoop:beta"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?