Clean Country #241

miriam.baglioni · 2022-08-08T14:27:34+02:00

miriam.baglioni commented

2022-08-08 14:27:34 +02:00

This PR is the first naive implementation to clean the country element. It is done to remove from the records collected from NARCIS the country NL. It is needed because NARCIS has been included in the allowed datasources for the country propagation step. Being NARCIS an aggregator some of the repositories it collect from do not provide only elements from the NL. In that case the association to the country should be removed.

The criteria to be matched are:

in the pids of the records there is one doi with prefix 10.17632 (Mendely data) or 10.5061 (Dryad),
the record does not have in the hostedby.key any institutional repository with country NL,
the record is collectedfrom.value = NARCIS,
the country with value NL has been inserted via propagation.

If all the constraints above match, the country NL is removed from the set of the countries for the result.

The introduction of this new cleaning process modifies one parameter in the workflow: before we had the shouldCleanContext parameter, now we have the shouldClean parameter instand. If this param is set to true all the cleaning action for the result will be triggered. So far they are clean of context for sobigdata and clean of country for NARCIS

This PR is the first naive implementation to clean the country element. It is done to remove from the records collected from NARCIS the country `NL`. It is needed because NARCIS has been included in the allowed datasources for the country propagation step. Being NARCIS an aggregator some of the repositories it collect from do not provide only elements from the NL. In that case the association to the country should be removed. The criteria to be matched are: * in the pids of the records there is one doi with prefix `10.17632` (Mendely data) or `10.5061` (Dryad), * the record does not have in the `hostedby.key` any institutional repository with country `NL`, * the record is `collectedfrom.value = NARCIS`, * the country with value `NL` has been inserted via propagation. If all the constraints above match, the country `NL` is removed from the set of the countries for the result. The introduction of this new cleaning process modifies one parameter in the workflow: before we had the `shouldCleanContext` parameter, now we have the `shouldClean` parameter instand. If this param is set to true all the cleaning action for the result will be triggered. So far they are clean of context for sobigdata and clean of country for NARCIS

miriam.baglioni added the

enhancement

label 2022-08-08 14:27:34 +02:00

alessia.bardi was assigned by miriam.baglioni

2022-08-08 14:27:34 +02:00

claudio.atzori was assigned by miriam.baglioni

2022-08-08 14:27:36 +02:00

miriam.baglioni added 6 commits 2022-08-08 14:27:44 +02:00

65cc736e2f [Clean Country] first implementation to remove country NL from results collected from NARCIS when doi starts with mendely prefix

2c933f1158 mergin with branch beta

a12d28c644 [Clean Country] added logic not to remove country from result if it exist a hosting datasource with that country. Moreover the country will be removed only if added with propagation

5968ec018d [Clean Country] modified workflow and added param file

390013a4b2 mergin with branch beta

62d2138806 [Clean Context] changed a bit the logic. Added the check not to have result hosted by a datasource of type institutional repository from NL. Added also the check that the country should have been included in the result via propagation for it to be removed

claudio.atzori commented

2022-08-08 14:52:49 +02:00

Two comments without having seen the changeset on the PR description:

it misses the why we are introducing this change. I barely remember the fact that we are assuming the contents in Narcis to cover only the national scope, while we found out this is not the case. Is this the reason? If so, please describe it.
include a summary of the changes to be introduced in the parameters when invoking the graph cleaning workflow.

Two comments without having seen the changeset on the PR description: 1. it misses the _why_ we are introducing this change. I barely remember the fact that we are assuming the contents in Narcis to cover only the national scope, while we found out this is not the case. Is this the reason? If so, please describe it. 2. include a summary of the changes to be introduced in the parameters when invoking the graph cleaning workflow.

claudio.atzori reviewed 2022-08-10 08:50:10 +02:00

dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/clean/country/CleanCountrySparkJob.java

						
				@ -0,0 +94,4 @@

						List<String> hostedBy = spark

							.read()

							.textFile(datasourcePath)

							// .filter((FilterFunction<String>) ds -> !ds.equals(collectedfrom))

claudio.atzori commented

2022-08-10 08:50:07 +02:00

if not needed, clean it up please.

claudio.atzori marked this conversation as resolved

claudio.atzori reviewed 2022-08-10 08:52:16 +02:00

dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/clean/country/CleanCountrySparkJob.java Outdated

						
				@ -0,0 +113,4 @@

							if (r

								.getPid()

								.stream()

								.anyMatch(p -> p.getQualifier().getClassid().equals("doi") && pidInParam(p.getValue(), verifyParam))) {