lsmyrnaios
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Pull Requests Activity
dnet-hadoop/dhp-workflows/dhp-continuous-validation
History
Lampros Smyrnaios 8d0ed7d414 Continuous-Validation updates: 
- Update the "uoa-validator-engine2" dependency.
- Update the "installProject.sh" script to account for potential conflict with previous builds.
- Add documentation.
 		2024-01-25 18:23:14 +02:00
..
src/main - Add the "installProject.sh" script. 2024-01-18 18:06:50 +02:00
README.md Continuous-Validation updates: 2024-01-25 18:23:14 +02:00
installProject.sh Continuous-Validation updates: 2024-01-25 18:23:14 +02:00
pom.xml - Use "KryoSerializer" for Spark and register some "result" classes. 2024-01-17 18:08:05 +02:00
runOozieWorkflow.sh Continuous-Validation updates: 2024-01-25 18:23:14 +02:00

README.md

Continuous Validation

This module is responsible for deploying an Oozie Workflow (on the desired cluster), which executes a Spark action.
This action takes the HDFS-path of a directory of parquet files containing metadata records, and applies the validation process on all of them, in parallel. Then it outputs the results, in json-format, in the given directory.
The validation process is powered by the uoa-validator-engine2 software.

Install and run

Run the ./installProject.sh script and then the ./runOozieWorkflow.sh script.

[...]