enrichment_country #203

miriam.baglioni · 2022-03-11T14:15:35+01:00

miriam.baglioni commented

2022-03-11 14:15:35 +01:00

This PR is to fix the issue with the country propagation which arises when results that should have the country propagated have no URL and so no instance is generated. In cases like this one, no country was added to the result. To overcome this, we select the values of the collectedfrom at the result level. It also revises the code to get rid of hive and compute the preparation step through spark actions

miriam.baglioni added the

bug

label 2022-03-11 14:15:35 +01:00

alessia.bardi was assigned by miriam.baglioni

2022-03-11 14:15:35 +01:00

claudio.atzori was assigned by miriam.baglioni

2022-03-11 14:15:35 +01:00

miriam.baglioni added 3 commits 2022-03-11 14:15:35 +01:00

2b643059fa [Country Propagation] changed the logic to get the collectedfrom at the result level. To fix issue when no instance is created for a result that should have the country associated. Change the code to use spark instead of hive to prepare the data needed for the propagation step. Added new tests for the intermediate steps and new verification for the propagation itself

4437f9345d [Country Propagation] left out from previous commit

2fbb35ade5 mergin with branch beta

miriam.baglioni added 1 commit 2022-03-11 14:17:08 +01:00

12de9acb0d [Country Propagation] left out from previous commit

claudio.atzori reviewed 2022-03-11 14:24:25 +01:00

dhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/PrepareResultCountrySet.java

						
				@ -61,3 +62,3 @@

						SparkConf conf = new SparkConf();

						conf.set("hive.metastore.uris", parser.get("hive_metastore_uris"));

						// conf.set("hive.metastore.uris", parser.get("hive_metastore_uris"));

claudio.atzori commented

2022-03-11 14:24:25 +01:00

Please clenup: if we don't need to use Hive in this procedure, then we don't need to keep track of the hive-specific parameters, not even as comments.