D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 13 Projects 2 Releases 2 Wiki Activity

[dedup] use common saveParquet and save methods to ensure outputs are compressed #349

Merged
claudio.atzori merged 1 commits from fix_dedup_not_compressed into beta 2023-10-16 11:56:18 +02:00
Conversation 1 Commits 1 Files Changed 6 +8 -21
claudio.atzori commented 2023-10-16 10:58:32 +02:00
Owner
Copy Link

This PR uses the common dataset writing methods available in the eu.dnetlib.dhp.oa.dedup.AbstractSparkAction class to ensure that all the actions defined in the deduplication workflow produce a compressed output.

This PR uses the common dataset writing methods available in the `eu.dnetlib.dhp.oa.dedup.AbstractSparkAction` class to ensure that all the actions defined in the deduplication workflow produce a compressed output.
claudio.atzori added 1 commit 2023-10-16 10:58:32 +02:00
claudio.atzori requested review from giambattista.bloisi 2023-10-16 10:59:11 +02:00
claudio.atzori added this to the OpenAIRE project 2023-10-16 11:26:38 +02:00
giambattista.bloisi commented 2023-10-16 11:52:19 +02:00
Member
Copy Link

Looks good to me

Looks good to me
claudio.atzori merged commit 389e3fcc59 into beta 2023-10-16 11:56:18 +02:00
claudio.atzori deleted branch fix_dedup_not_compressed 2023-10-16 11:56:19 +02:00
claudio.atzori modified the project from OpenAIRE to OpenAIRE - DNet 2023-10-26 10:00:04 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
giambattista.bloisi
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
OpenAIRE - DNet
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#349
No description provided.
Delete Branch "fix_dedup_not_compressed"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?