DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag #329
No reviewers
Labels
No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: D-Net/dnet-hadoop#329
Loading…
Reference in New Issue
No description provided.
Delete Branch "dispatch_filter_invisible_entities"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This PR implements point 5 of 8898:
DispatchEntitiesSparkJob has been extended with a "--filterInvisible" flag to control whether invisible records have to be filtered out or not.
Also, DispatchEntitiesSparkJob has been refactored to dispatch all entities as parallel Spark jobs in the same application instead of parallel oozie jobs.
Note: graph/group and dedup/consistency workflows now require a mandatory filterInvisible parameters to be set.