dnet-hadoop

History

Miriam Baglioni dc0ad8d2e0 fixed issue related to change in the file name downloaded. Added sheet name as parameter and also a check if the name should change		2021-05-20 14:53:53 +02:00
..
src	fixed issue related to change in the file name downloaded. Added sheet name as parameter and also a check if the name should change	2021-05-20 14:53:53 +02:00
README.md	dhp-collection-worker integrated in dhp-workflows	2019-10-24 11:36:59 +02:00
pom.xml	depending on external dhp-schemas module	2021-04-23 17:52:36 +02:00

README.md

Description of the Module

This module defines a collector worker application that runs on Hadoop.

It is responsible for harvesting metadata using different plugins.

The collector worker uses a message queue to inform the progress of the harvesting action (using a message queue for sending ONGOING messages) furthermore, It gives, at the end of the job, some information about the status of the collection i.e Number of records collected(using a message queue for sending REPORT messages).

To work the collection worker need some parameter like:

hdfsPath: the path where storing the sequential file
apidescriptor: the JSON encoding of the API Descriptor
namenode: the Name Node URI
userHDFS: the user wich create the hdfs seq file
rabbitUser: the user to connect with RabbitMq for messaging
rabbitPassWord: the password to connect with RabbitMq for messaging
rabbitHost: the host of the RabbitMq server
rabbitOngoingQueue: the name of the ongoing queue
rabbitReportQueue: the name of the report queue
workflowId: the identifier of the dnet Workflow

##Plugins

OAI Plugin

Usage

TODO