lsmyrnaios
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Pull Requests Activity
dnet-hadoop/dhp-workflows/dhp-continuous-validation/README.md

646 B
Raw Blame History

Continuous Validation

This module is responsible for deploying an Oozie Workflow (on the desired cluster), which executes a Spark action.
This action takes the HDFS-path of a directory of parquet files containing metadata records, and applies the validation process on all of them, in parallel. Then it outputs the results, in json-format, in the given directory.
The validation process is powered by the uoa-validator-engine2 software.

Install and run

Run the ./installProject.sh script and then the ./runOozieWorkflow.sh script.

[...]