D-Net
/
dnet-hadoop
15
4
Fork 10
Code Issues 14 Pull Requests 10 Projects 2 Releases 1 Wiki Activity

[WebCrawl] adding affiliation relations from web information #428

Merged
claudio.atzori merged 3 commits from WebCrowlBeta into beta 2024-04-23 09:36:16 +02:00
Conversation 0 Commits 3 Files Changed 9 +664
miriam.baglioni commented 2024-04-22 11:06:44 +02:00
Member
Copy Link

This PR adds affiliation information from web resources. It takes everythong for IE and up to 2021 for the other countries

This PR adds affiliation information from web resources. It takes everythong for IE and up to 2021 for the other countries
claudio.atzori was assigned by miriam.baglioni 2024-04-22 11:06:44 +02:00
miriam.baglioni added 1 commit 2024-04-22 11:06:45 +02:00
claudio.atzori added 1 commit 2024-04-22 11:40:25 +02:00
claudio.atzori requested changes 2024-04-22 12:22:16 +02:00
claudio.atzori left a comment
Owner
Copy Link

Please address the inline comments

Please address the inline comments
dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/actionmanager/webcrawl/CreateActionSetFromWebEntries.java Outdated
@ -0,0 +78,4 @@
spark -> {
createActionSet(spark, inputPath, outputPath + "actionSet");
createPlainRelations(spark, inputPath, outputPath + "relations");
claudio.atzori commented 2024-04-22 12:22:02 +02:00
Owner
Copy Link

The output path is controlled by the actionset management framework and should not be altered by the workflow. Hence consider to place them elsewhere under a different root directory. Also the output of the createActionSet function should be placed directly inside outputPath.

The output path is controlled by the actionset management framework and should not be altered by the workflow. Hence consider to place them elsewhere under a different root directory. Also the output of the `createActionSet` function should be placed directly inside `outputPath`.
dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/actionmanager/webcrawl/CreateActionSetFromWebEntries.java Outdated
@ -0,0 +197,4 @@
return createAffiliatioRelationPair(
PMCID_PREFIX
+ IdentifierFactory
.md5(PidCleaner.normalizePidValue(PidType.pmc.toString(), "PMC" + pmcid.substring(43))),
claudio.atzori commented 2024-04-22 12:18:08 +02:00
Owner
Copy Link

It seems that PIDs found in the input dataset include their resolver. As we need them without it, it should be better to make this manipulation explicit.

It seems that PIDs found in the input dataset include their resolver. As we need them without it, it should be better to make this manipulation explicit.
miriam.baglioni added 1 commit 2024-04-22 13:52:55 +02:00
claudio.atzori merged commit c57cff2d6d into beta 2024-04-23 09:36:16 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
claudio.atzori
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
claudio.atzori
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#428
No description provided.
Delete Branch "WebCrowlBeta"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?