[ENRICHMENT][BETA] Use of community API in enrichment process AND addition to tagging result for communities through projects #359

miriam.baglioni · 2023-11-03T15:30:21+01:00

miriam.baglioni commented

2023-11-03 15:30:21 +01:00

In this PR the use of the IS is replaced by the community APIs in the bulktagging and resulttocommunityfromorganization modules. The code is also refactored.

There is also the addition of a new module resulttocommunityfromproject that associated the result to the community if the result is linked to a project relevant for the community. This will replace the IIS document_referencedProjects in the graph processing. We need to add a new step to the workflow (it could be added at any point in the enrichment pipeline) The parameters are:

sourcePath = the source path
outputPath = the output path
production = true/false If true it will query the communityapis for production, if false the ones for beta

Note: the production parameter has to be added also to the bulktag and resulttocommunityfromorganization steps with the same semantics.

In this PR the use of the IS is replaced by the community APIs in the bulktagging and resulttocommunityfromorganization modules. The code is also refactored. There is also the addition of a new module resulttocommunityfromproject that associated the result to the community if the result is linked to a project relevant for the community. This will replace the IIS document_referencedProjects in the graph processing. We need to add a new step to the workflow (it could be added at any point in the enrichment pipeline) The parameters are: sourcePath = the source path outputPath = the output path production = true/false If true it will query the communityapis for production, if false the ones for beta Note: the production parameter has to be added also to the bulktag and resulttocommunityfromorganization steps with the same semantics.

claudio.atzori was assigned by miriam.baglioni

2023-11-03 15:30:21 +01:00

miriam.baglioni added 16 commits 2023-11-03 15:30:22 +01:00

3d6be20989 changes to use the API instead of the IS the get the information for the communities to be used during bulktagging and context propagation

8448b9ebfb mergin with branch beta

a3d01ccb24 refactoring

89184d5b4f used the API instead of the IS for bulktagging and propagation for community through organization. Added a new propagation step for communities through projects. Still using the API and not the IS

8e9493fad9 mergin with branch beta

159388f9c2 testing and fix some issues

a5a89a702f new spark parrameter updated

f1b898c6b4 mergin with branch beta

a4214ced1e fixing issue on propagation organization. added --config to workflow definition. added oozie_app to communtiy project

e9a20fc8f6 mergin with branch beta

c80b768af0 test for project propagation

a737dd47b6 removed not needed test class

937ff6a7c7 -

edcb17ca98 refactoring and test

638ad9e74f changing test for new implementation

7b1e34f159 refactoring

miriam.baglioni added 1 commit 2023-11-03 15:35:15 +01:00

5bc97615d5 -

alessia.bardi was assigned by miriam.baglioni

2023-11-03 15:38:24 +01:00

claudio.atzori requested changes 2023-11-03 15:55:04 +01:00

dhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/api/QueryCommunityAPI.java Outdated

						
				@ -0,0 +14,4 @@

				 * @Date 06/10/23

				 */

				public class QueryCommunityAPI {

					private static final String PRODUCTION_BASE_URL = "https://services.openaire.eu/openaire/";

claudio.atzori commented

2023-11-03 15:53:44 +01:00

It is not a good idea to hardcode the service base urls. Please consider to factor it out as an input parameter: having the possibility to define the API base URL fits well with the definition of a test case where the API server application is mocked locally.

miriam.baglioni commented

2023-11-27 15:16:08 +01:00

I have changed this. Now we have a parameter baseURL where to specify the URL for production or beta.
It is expected to be "https://services.openaire.eu/openaire/communities/" and the equivalent one for beta "https://beta.services.openaire.eu/openaire/communities/"

I have changed this. Now we have a parameter baseURL where to specify the URL for production or beta. It is expected to be "https://services.openaire.eu/openaire/communities/" and the equivalent one for beta "https://beta.services.openaire.eu/openaire/communities/"

claudio.atzori commented

2023-11-28 11:58:51 +01:00

Thanks!

claudio.atzori marked this conversation as resolved

claudio.atzori changed title from ~~[ENRICHMENT]Use of community API in enrichment process AND addition to tagging result for communities through projects~~ to [ENRICHMENT][BETA] Use of community API in enrichment process AND addition to tagging result for communities through projects

2023-11-14 12:37:24 +01:00

miriam.baglioni added 4 commits 2023-11-27 15:13:20 +01:00

eaf0a702de -

48e0427a23 changed the parameter from production to baseURL. Fixed issue in tagging configuration

e3cce9a5a0 mergin with branch beta

8eb70e6657 refactoring

claudio.atzori added 1 commit 2023-11-30 14:20:22 +01:00

6f10791e77 Merge branch 'beta' into propagationapi

claudio.atzori merged commit 4cbabc9fbc into beta

2023-11-30 14:20:34 +01:00

claudio.atzori referenced this issue from a commit

2023-11-30 14:20:34 +01:00