D-Net
/
dnet-hadoop
15
4
Fork 10
Code Issues 14 Pull Requests 10 Projects 2 Releases 1 Wiki Activity

DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag #329

Merged
miriam.baglioni merged 2 commits from dispatch_filter_invisible_entities into beta 2023-08-10 12:56:19 +02:00
Conversation 0 Commits 2 Files Changed 6 +142 -456
giambattista.bloisi commented 2023-08-09 15:54:23 +02:00
Member
Copy Link

This PR implements point 5 of 8898:

  • Get rid of invisible records after deduplication. They already contributed so we don't need them anymore after the deduplication workflow (graph consistency)

DispatchEntitiesSparkJob has been extended with a "--filterInvisible" flag to control whether invisible records have to be filtered out or not.
Also, DispatchEntitiesSparkJob has been refactored to dispatch all entities as parallel Spark jobs in the same application instead of parallel oozie jobs.

Note: graph/group and dedup/consistency workflows now require a mandatory filterInvisible parameters to be set.

This PR implements point 5 of [8898](https://support.openaire.eu/issues/8898): - Get rid of invisible records after deduplication. They already contributed so we don't need them anymore after the deduplication workflow (graph consistency) DispatchEntitiesSparkJob has been extended with a "--filterInvisible" flag to control whether invisible records have to be filtered out or not. Also, DispatchEntitiesSparkJob has been refactored to dispatch all entities as parallel Spark jobs in the same application instead of parallel oozie jobs. Note: graph/group and dedup/consistency workflows now require a mandatory filterInvisible parameters to be set.
giambattista.bloisi added 1 commit 2023-08-09 15:54:24 +02:00
giambattista.bloisi requested review from claudio.atzori 2023-08-09 15:54:36 +02:00
giambattista.bloisi requested review from miriam.baglioni 2023-08-09 15:54:44 +02:00
giambattista.bloisi added 1 commit 2023-08-10 11:54:00 +02:00
95cd2b9b1e Make filterInvisible a mandatory parameter of DispathEntitiesSparkJob 
Make filterInvisible a mandatory parameter of both dedup/consistency and graph/group oozie workflows
miriam.baglioni merged commit 35b8deb2c6 into beta 2023-08-10 12:56:19 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
claudio.atzori
miriam.baglioni
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#329
No description provided.
Delete Branch "dispatch_filter_invisible_entities"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?