forked from D-Net/dnet-hadoop
changed documentation since it didn't reflect the current status
This commit is contained in:
parent
07a0ccfc96
commit
cda210a2ca
|
@ -2,28 +2,15 @@ Description of the Module
|
||||||
--------------------------
|
--------------------------
|
||||||
This module defines a **collector worker application** that runs on Hadoop.
|
This module defines a **collector worker application** that runs on Hadoop.
|
||||||
|
|
||||||
It is responsible for harvesting metadata using different plugins.
|
It is responsible for harvesting metadata using different collector plugins and transformation into the common metadata model.
|
||||||
|
|
||||||
The collector worker uses a message queue to inform the progress
|
# Collector Plugins
|
||||||
of the harvesting action (using a message queue for sending **ONGOING** messages) furthermore,
|
|
||||||
It gives, at the end of the job, some information about the status
|
|
||||||
of the collection i.e Number of records collected(using a message queue for sending **REPORT** messages).
|
|
||||||
|
|
||||||
To work the collection worker need some parameter like:
|
|
||||||
|
|
||||||
* **hdfsPath**: the path where storing the sequential file
|
|
||||||
* **apidescriptor**: the JSON encoding of the API Descriptor
|
|
||||||
* **namenode**: the Name Node URI
|
|
||||||
* **userHDFS**: the user wich create the hdfs seq file
|
|
||||||
* **rabbitUser**: the user to connect with RabbitMq for messaging
|
|
||||||
* **rabbitPassWord**: the password to connect with RabbitMq for messaging
|
|
||||||
* **rabbitHost**: the host of the RabbitMq server
|
|
||||||
* **rabbitOngoingQueue**: the name of the ongoing queue
|
|
||||||
* **rabbitReportQueue**: the name of the report queue
|
|
||||||
* **workflowId**: the identifier of the dnet Workflow
|
|
||||||
|
|
||||||
##Plugins
|
|
||||||
* OAI Plugin
|
* OAI Plugin
|
||||||
|
|
||||||
## Usage
|
# Transformation Plugins
|
||||||
TODO
|
TODO
|
||||||
|
|
||||||
|
|
||||||
|
# Usage
|
||||||
|
TODO
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue