[ENRICHMENT][PROD] Use of community API in enrichment process AND addition to tagging result for communities through projects #361

claudio.atzori · 2023-11-14T12:38:02+01:00

claudio.atzori commented

2023-11-14 12:38:02 +01:00

In this PR the use of the IS is replaced by the community APIs in the bulktagging and resulttocommunityfromorganization modules. The code is also refactored.

There is also the addition of a new module resulttocommunityfromproject that associated the result to the community if the result is linked to a project relevant for the community. This will replace the IIS document_referencedProjects in the graph processing. We need to add a new step to the workflow (it could be added at any point in the enrichment pipeline) The parameters are:

sourcePath = the source path
outputPath = the output path
production = true/false If true it will query the communityapis for production, if false the ones for beta

Note: the production parameter has to be added also to the bulktag and resulttocommunityfromorganization steps with the same semantics.

In this PR the use of the IS is replaced by the community APIs in the bulktagging and resulttocommunityfromorganization modules. The code is also refactored. There is also the addition of a new module resulttocommunityfromproject that associated the result to the community if the result is linked to a project relevant for the community. This will replace the IIS document_referencedProjects in the graph processing. We need to add a new step to the workflow (it could be added at any point in the enrichment pipeline) The parameters are: sourcePath = the source path outputPath = the output path production = true/false If true it will query the communityapis for production, if false the ones for beta Note: the production parameter has to be added also to the bulktag and resulttocommunityfromorganization steps with the same semantics.

claudio.atzori self-assigned this 2023-11-14 12:38:02 +01:00

claudio.atzori added 17 commits 2023-11-14 12:38:03 +01:00

3d6be20989 changes to use the API instead of the IS the get the information for the communities to be used during bulktagging and context propagation

8448b9ebfb mergin with branch beta

a3d01ccb24 refactoring

89184d5b4f used the API instead of the IS for bulktagging and propagation for community through organization. Added a new propagation step for communities through projects. Still using the API and not the IS

8e9493fad9 mergin with branch beta

159388f9c2 testing and fix some issues

a5a89a702f new spark parrameter updated

f1b898c6b4 mergin with branch beta

a4214ced1e fixing issue on propagation organization. added --config to workflow definition. added oozie_app to communtiy project

e9a20fc8f6 mergin with branch beta