[Aggregator graph|master] Discard invalid records #245

claudio.atzori · 2022-09-19T09:44:58+02:00

claudio.atzori commented

2022-09-19 09:44:58 +02:00

This PR modifyes the bahavior of the oozie workflow responsible to create the aggregator graph.

Background

Transformation rules might output XML records that are either invalid XMLs or might not satisfy certain characteristics that would make them to be excluded by the mapping layer that transforms them in the internal graph data model. So far this happened silently during the application of the mapping, hardly making the situation to emerge and thus not allowing to inspect the invalid records and take consequent measures.

Proposed change

The proposed modification alters the oozie workflow adding an extra spark action (VerifyRecords) that implements the verification. It actually applies the mapping and in case it returns an error, the input record is stored aside for further inspection in the path configured as:

<arg>--invalidPath</arg><arg>${workingDir}/invalid_records</arg>

This PR modifyes the bahavior of the oozie workflow responsible to create the aggregator graph. #### Background Transformation rules might output XML records that are either invalid XMLs or might not satisfy certain characteristics that would make them to be excluded by the mapping layer that transforms them in the internal graph data model. So far this happened silently during the application of the mapping, hardly making the situation to emerge and thus not allowing to inspect the invalid records and take consequent measures. #### Proposed change The proposed modification alters the oozie workflow adding an extra spark action (`VerifyRecords`) that implements the verification. It actually applies the mapping and in case it returns an error, the input record is stored aside for further inspection in the path configured as: ``` <arg>--invalidPath</arg><arg>${workingDir}/invalid_records</arg> ```

claudio.atzori added the

enhancement

label 2022-09-19 09:44:58 +02:00

alessia.bardi was assigned by claudio.atzori

2022-09-19 09:44:58 +02:00

miriam.baglioni was assigned by claudio.atzori

2022-09-19 09:44:59 +02:00

claudio.atzori self-assigned this 2022-09-19 09:44:59 +02:00

claudio.atzori added 6 commits 2022-09-19 09:45:10 +02:00

b99a011345 return empty Oaf list if record cannot be parsed

a0919ed495 [aggregator graph] save invalid records aside for further inspection

c48f6e9c57 [aggregator graph] save invalid records aside for further inspection