D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 12 Projects 2 Releases 2 Wiki Activity

Patch the identifiers (source/target) in the relations #125

Merged
claudio.atzori merged 3 commits from fct_project_id_replacement into master 2021-07-29 12:11:28 +02:00
Conversation 2 Commits 3 Files Changed 4 +2 -4
1 changed files with 2 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 1923c1ce21 - Show all commits

6
dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/raw/PatchRelationsApplication.java
View File

@ -80,8 +80,7 @@ public class PatchRelationsApplication {
final Dataset<RelationIdMapping> idMapping = Utils.readPath(spark, idMappingPath, RelationIdMapping.class); final Dataset<RelationIdMapping> idMapping = Utils.readPath(spark, idMappingPath, RelationIdMapping.class);
rels rels
.joinWith(idMapping, rels.col("source").equalTo(idMapping.col("oldId")), "full") .joinWith(idMapping, rels.col("source").equalTo(idMapping.col("oldId")), "left")
.filter((FilterFunction<Tuple2<Relation, RelationIdMapping>>) t -> Objects.nonNull(t._1()))
.map((MapFunction<Tuple2<Relation, RelationIdMapping>, Relation>) t -> { .map((MapFunction<Tuple2<Relation, RelationIdMapping>, Relation>) t -> {
claudio.atzori marked this conversation as resolved
miriam.baglioni commented 2021-07-29 11:29:37 +02:00
Review
Copy Link

I think you could replace the filter step with a left join: you get all the original relations that is what you do with the full and the subsequent filter

I think you could replace the filter step with a left join: you get all the original relations that is what you do with the full and the subsequent filter
final Relation r = t._1(); final Relation r = t._1();
Optional.ofNullable(t._2()) Optional.ofNullable(t._2())
@ -89,8 +88,7 @@ public class PatchRelationsApplication {
.ifPresent(r::setSource); .ifPresent(r::setSource);
return r; return r;
}, Encoders.bean(Relation.class)) }, Encoders.bean(Relation.class))
.joinWith(idMapping, rels.col("target").equalTo(idMapping.col("oldId")), "full") .joinWith(idMapping, rels.col("target").equalTo(idMapping.col("oldId")), "left")
.filter((FilterFunction<Tuple2<Relation, RelationIdMapping>>) t -> Objects.nonNull(t._1()))
.map((MapFunction<Tuple2<Relation, RelationIdMapping>, Relation>) t -> { .map((MapFunction<Tuple2<Relation, RelationIdMapping>, Relation>) t -> {
final Relation r = t._1(); final Relation r = t._1();
Optional.ofNullable(t._2()) Optional.ofNullable(t._2())