D-Net
/
dnet-hadoop
13
4
Fork 10
Code Issues 14 Pull Requests 7 Projects 2 Releases 1 Wiki Activity
dnet-hadoop/dhp-workflows/dhp-aggregation
History
Miriam Baglioni 55ea485783 [UsageCount] split the count for result at the level of the datasource. for each indicator one unit is specified for each datasource contrinuting to that indicator value. The datasource key is the value of the key element in the unit for the measure, while the count for that datasource is in the value 2023-06-30 18:39:30 +02:00
..
src [UsageCount] split the count for result at the level of the datasource. for each indicator one unit is specified for each datasource contrinuting to that indicator value. The datasource key is the value of the key element in the unit for the measure, while the count for that datasource is in the value 2023-06-30 18:39:30 +02:00
.scalafmt.conf [stats-wf]fixed the result_result table related to PR#191 2022-02-04 14:51:25 +01:00
README.md README.md for aggregation workflows 2021-03-03 16:18:34 +01:00
pom.xml [maven-release-plugin] prepare for next development iteration 2022-04-07 13:32:22 +02:00

README.md

Description of the Module

This module defines a set of oozie workflows for the collection and transformation of metadata records. Both workflows interact with the Metadata Store Manager (MdSM) to handle the logical transactions required to ensure the consistency of the read/write operations on the data as the MdSM in fact keeps track of the logical-physical mapping of each MDStore.

Metadata collection

The metadata collection workflow is responsible for harvesting metadata records from different protocols and responding to different formats and to store them as on HDFS so that they can be further processed.

Collector Plugins

Different protocols are managed by dedicated Collector plugins, i.e. java programs implementing a defined interface:

eu.dnetlib.dhp.collection.plugin.CollectorPlugin

The list of the supported plugins:

  • OAI Plugin: collects from OAI-PMH compatible endpoints
  • MDStore plugin: collects from a given D-Net MetadataStore, (identified by moogodb URI, dbName, MDStoreID)
  • MDStore dump plugin: collects from an MDStore dump stored on the HDFS location indicated by the path parameter

Transformation Plugins

TODO