D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 11 Projects 2 Releases 2 Wiki Activity

[webcrawl] the blacklist is now in json and no more in csv after the normalization process

This commit is contained in:
Miriam Baglioni 2024-07-25 15:20:18 +02:00
parent 7cff281d3e
commit c7f6669f1a
3 changed files with 2 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/actionmanager/webcrawl/CreateActionSetFromWebEntries.java
View File

@ -148,8 +148,7 @@ public class CreateActionSetFromWebEntries implements Serializable {
return spark
.read()
.option("header", true)
.csv(inputPath)
.json(inputPath)
.select("OpenAlexId");
}

2
dhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/actionmanager/webcrawl/blackListRemove/not_irish.csv
View File

@ -1,2 +0,0 @@
DOI / PMID,OpenAlexId,Comments,
https://doi.org/10.1098/rstl.1684.0023,https://openalex.org/W2124362779,,
1 DOI / PMID OpenAlexId Comments
2 https://doi.org/10.1098/rstl.1684.0023 https://openalex.org/W2124362779

1
dhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/actionmanager/webcrawl/blackListRemove/not_irish.json Normal file
View File

@ -0,0 +1 @@
{"doi":"https://doi.org/10.1098/rstl.1684.0023","OpenAlexId":"https://openalex.org/W2124362779"}