Support for the PromoteAction strategy #389

claudio.atzori · 2024-02-05T15:57:38+01:00

claudio.atzori commented

2024-02-05 15:57:38 +01:00

This PR introduces support for two strategies the workflow can use to promote the actionset contents in the graph.

So far the only strategy assumed to upsert each record against the corresponding entity table, merging with the records matched by OpenAIRE id and inserting those that do not match. This turned out to be introducing noisy or stable records often contained in the actionset, which pollute the graph and may cause failures along the graph processing pipeline.

The upsert strategy continues to be the default one, but this modification allows the workflow caller to speficy a different one, named ENRICH, which only updates the matching records, while discarding the non-matching ones.

Hence, the oozie workflow definition now suppoerts a new optional parameter named promoteActionStrategy that can assume two values

UPSERT (default)
ENRICH

This PR introduces support for two strategies the workflow can use to promote the actionset contents in the graph. So far the only strategy assumed to _upsert_ each record against the corresponding entity table, merging with the records matched by OpenAIRE id and inserting those that do not match. This turned out to be introducing noisy or stable records often contained in the actionset, which pollute the graph and may cause failures along the graph processing pipeline. The upsert strategy continues to be the default one, but this modification allows the workflow caller to speficy a different one, named `ENRICH`, which only updates the matching records, while discarding the non-matching ones. Hence, the oozie workflow definition now suppoerts a new optional parameter named `promoteActionStrategy` that can assume two values - `UPSERT` (default) - `ENRICH`

claudio.atzori self-assigned this 2024-02-05 15:57:38 +01:00

claudio.atzori added 2 commits 2024-02-05 15:57:38 +01:00

009dcf6aea [actiosets] introduced support for the PromoteAction strategy

fd17c1f17c [actiosets] fixed join type

claudio.atzori requested review from miriam.baglioni 2024-02-05 15:57:47 +01:00

claudio.atzori requested review from giambattista.bloisi 2024-02-05 15:57:53 +01:00

claudio.atzori commented

2024-02-05 15:58:22 +01:00

@giambattista.bloisi @miriam.baglioni can you review these changes? Thanks!