D-Net
/
dnet-hadoop
13
4
Fork 10
Code Issues 14 Pull Requests 8 Projects 2 Releases 1 Wiki Activity

Support for the PromoteAction strategy [master] #391

Merged
claudio.atzori merged 2 commits from promote_actions_join_type_master into master 2024-02-08 15:12:17 +01:00
Conversation 0 Commits 2 Files Changed 13 +84 -10
claudio.atzori commented 2024-02-08 15:12:07 +01:00
Owner
Copy Link

This PR introduces support for two strategies the workflow can use to promote the actionset contents in the graph.

So far the only strategy assumed to upsert each record against the corresponding entity table, merging with the records matched by OpenAIRE id and inserting those that do not match. This turned out to be introducing noisy or stable records often contained in the actionset, which pollute the graph and may cause failures along the graph processing pipeline.

The upsert strategy continues to be the default one, but this modification allows the workflow caller to speficy a different one, named ENRICH, which only updates the matching records, while discarding the non-matching ones.

Hence, the oozie workflow definition now suppoerts a new optional parameter named promoteActionStrategy that can assume two values

  • UPSERT (default)
  • ENRICH
This PR introduces support for two strategies the workflow can use to promote the actionset contents in the graph. So far the only strategy assumed to _upsert_ each record against the corresponding entity table, merging with the records matched by OpenAIRE id and inserting those that do not match. This turned out to be introducing noisy or stable records often contained in the actionset, which pollute the graph and may cause failures along the graph processing pipeline. The upsert strategy continues to be the default one, but this modification allows the workflow caller to speficy a different one, named `ENRICH`, which only updates the matching records, while discarding the non-matching ones. Hence, the oozie workflow definition now suppoerts a new optional parameter named `promoteActionStrategy` that can assume two values - `UPSERT` (default) - `ENRICH`
claudio.atzori added 2 commits 2024-02-08 15:12:08 +01:00
claudio.atzori merged commit f21133229a into master 2024-02-08 15:12:17 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#391
No description provided.
Delete Branch "promote_actions_join_type_master"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?