D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 13 Projects 2 Releases 2 Wiki Activity

Add Collector Plugin for Zenodo Dumps #516

Merged
claudio.atzori merged 3 commits from zenodo_dump_collection into beta 2024-12-06 13:51:14 +01:00
Conversation 0 Commits 3 Files Changed 6 +194 -1
sandro.labruzzo commented 2024-12-04 14:02:02 +01:00
Owner
Copy Link

This pull request introduces a new collector plugin designed to handle Zenodo dumps. The plugin expects the following configuration in the APIDescriptor:

The plugin performs the following tasks:

  • Downloads the tar.gz file from the specified baseURL.
  • Extracts the contents of the tar.gz file.
  • Iterates over all files inside the archive and processes them as needed.

DNET Integration
I am unsure of the steps required to integrate this new plugin into the DNET infrastructure. Unfortunately, I don’t recall the legacy code details or the integration process.

This pull request introduces a new collector plugin designed to handle Zenodo dumps. The plugin expects the following configuration in the APIDescriptor: - baseURL: The URL pointing to the latest dump tarball (e.g., https://cernbox.cern.ch/remote.php/dav/public-files/hoJyvfqO6hDjDeh/zenodo-2024-09-18_155657.tar.gz). - hdfsURI (inside params): The main HDFS URI. The plugin performs the following tasks: - Downloads the tar.gz file from the specified baseURL. - Extracts the contents of the tar.gz file. - Iterates over all files inside the archive and processes them as needed. **DNET Integration** I am unsure of the steps required to integrate this new plugin into the DNET infrastructure. Unfortunately, I don’t recall the legacy code details or the integration process.
sandro.labruzzo added 2 commits 2024-12-04 14:02:03 +01:00
claudio.atzori was assigned by sandro.labruzzo 2024-12-04 14:04:20 +01:00
miriam.baglioni was assigned by sandro.labruzzo 2024-12-04 14:04:26 +01:00
giambattista.bloisi was assigned by sandro.labruzzo 2024-12-04 14:04:32 +01:00
sandro.labruzzo added 1 commit 2024-12-04 15:04:08 +01:00
miriam.baglioni reviewed 2024-12-06 11:20:05 +01:00
dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/collection/plugin/zenodo/CollectZenodoDumpCollectorPlugin.java
@ -0,0 +83,4 @@
org.apache.hadoop.io.IOUtils.closeStream(gzipInputStream);
}
} catch (Exception e) {
throw new CollectorException(e);
miriam.baglioni commented 2024-12-06 11:20:05 +01:00
Member
Copy Link

you should delete the temporary file

you should delete the temporary file
claudio.atzori merged commit 5c7f7fb3b8 into beta 2024-12-06 13:51:14 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
giambattista.bloisi
miriam.baglioni
claudio.atzori
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#516
No description provided.
Delete Branch "zenodo_dump_collection"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?