D-Net
/
dnet-hadoop
15
4
Fork 10
Code Issues 14 Pull Requests 10 Projects 2 Releases 1 Wiki Activity

Add Pubmed affiliations (inferred by BIP) as actionsets #353

Merged
claudio.atzori merged 5 commits from 9117_pubmed_affiliations into beta 2023-11-22 13:53:07 +01:00
Conversation 6 Commits 5 Files Changed 5 +58 -25
schatz commented 2023-10-20 11:51:34 +02:00
Member
Copy Link

Related to task 9117

Extends the WF that creates actionsets for Crossref by also adding affiliation relations from Pubmed.

New workflow param:

  • pubmedInputPath -> we can assume that this param can be set to /data/bip-affiiations/pubmed-data.json and with each update, we will update the file in this path.
Related to task [9117](https://support.openaire.eu/issues/9117) Extends the WF that creates actionsets for Crossref by also adding affiliation relations from Pubmed. New workflow param: * `pubmedInputPath` -> we can assume that this param can be set to `/data/bip-affiiations/pubmed-data.json` and with each update, we will update the file in this path.
claudio.atzori was assigned by schatz 2023-10-20 11:51:34 +02:00
schatz added 2 commits 2023-10-20 11:51:35 +02:00
claudio.atzori requested changes 2023-10-25 14:40:31 +02:00
claudio.atzori left a comment
Owner
Copy Link

Overall looks good, please address the minor changes I mentioned with the inline comments.

Overall looks good, please address the minor changes I mentioned with the inline comments.
dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/actionmanager/bipaffiliations/PrepareAffiliationRelations.java Outdated
@ -59,3 +60,3 @@
final String inputPath = parser.get("inputPath");
log.info("inputPath {}: ", inputPath);
log.info("inputPath: {}", inputPath);
claudio.atzori commented 2023-10-25 14:32:56 +02:00
Owner
Copy Link

As the workflow now assumes now two input paths, I suggest to align the parameter naming convention renaming this one to crossrefInputPath.

As the workflow now assumes now two input paths, I suggest to align the parameter naming convention renaming this one to `crossrefInputPath`.
schatz marked this conversation as resolved
dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/bipaffiliations/input_actionset_parameter.json Outdated
@ -8,7 +8,13 @@
{
"paramName": "ip",
"paramLongName": "inputPath",
claudio.atzori commented 2023-10-25 14:28:50 +02:00
Owner
Copy Link

As the workflow now assumes two input paths, I suggest to align the parameter naming convention renaming this one to crossrefInputPath.

As the workflow now assumes two input paths, I suggest to align the parameter naming convention renaming this one to `crossrefInputPath`.
schatz marked this conversation as resolved
dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/bipaffiliations/job.properties Outdated
@ -32,4 +32,5 @@ spark2SqlQueryExecutionListeners=com.cloudera.spark.lineage.NavigatorQueryListen
oozie.wf.application.path=${oozieTopWfApplicationPath}
inputPath=/data/bip-affiliations/data.json
claudio.atzori commented 2023-10-25 14:29:19 +02:00
Owner
Copy Link

parameter renaming to be reflected here as well

parameter renaming to be reflected here as well
schatz marked this conversation as resolved
dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/bipaffiliations/oozie_app/workflow.xml Outdated
@ -3,7 +3,11 @@
<property>
<name>inputPath</name>
claudio.atzori commented 2023-10-25 14:32:21 +02:00
Owner
Copy Link

again, parameter renaming to be reflected here as well

again, parameter renaming to be reflected here as well
schatz marked this conversation as resolved
dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/bipaffiliations/oozie_app/workflow.xml Outdated
@ -97,6 +101,7 @@
--conf spark.sql.warehouse.dir=${sparkSqlWarehouseDir}
</spark-opts>
<arg>--inputPath</arg><arg>${inputPath}</arg>
claudio.atzori commented 2023-10-25 14:38:25 +02:00
Owner
Copy Link

again, parameter renaming to be reflected here as well

again, parameter renaming to be reflected here as well
schatz marked this conversation as resolved
schatz added 1 commit 2023-10-25 21:05:08 +02:00
schatz commented 2023-10-25 21:07:36 +02:00
Author
Member
Copy Link

@claudio.atzori Thanks for the input! My intention was to keep the naming of the crossref input path in order not to change the different config files.
But, you are right, it is better to have consistent naming across all input params.
Please check again, thanks!

@claudio.atzori Thanks for the input! My intention was to keep the naming of the crossref input path in order not to change the different config files. But, you are right, it is better to have consistent naming across all input params. Please check again, thanks!
claudio.atzori added this to the OpenAIRE project 2023-10-26 09:41:07 +02:00
claudio.atzori modified the project from OpenAIRE to OpenAIRE - DNet 2023-10-26 09:58:35 +02:00
schatz added 1 commit 2023-10-26 22:47:10 +02:00
claudio.atzori commented 2023-10-30 15:49:37 +01:00
Owner
Copy Link

@claudio.atzori Thanks for the input! My intention was to keep the naming of the crossref input path in order not to change the different config files.
But, you are right, it is better to have consistent naming across all input params.
Please check again, thanks!

No prob! The rest looks OK, I'm going to keep this pending for integration until I will be ready to run again the pipeline to build beta graph.

> @claudio.atzori Thanks for the input! My intention was to keep the naming of the crossref input path in order not to change the different config files. > But, you are right, it is better to have consistent naming across all input params. > Please check again, thanks! No prob! The rest looks OK, I'm going to keep this pending for integration until I will be ready to run again the pipeline to build beta graph.
schatz commented 2023-10-30 15:50:33 +01:00
Author
Member
Copy Link

@claudio.atzori can we merge this for the next prod round? It seems to be important to get as many affiliations as possible for the Irish tender.

@claudio.atzori can we merge this for the next prod round? It seems to be important to get as many affiliations as possible for the Irish tender.
claudio.atzori requested review from alessia.bardi 2023-10-30 15:50:46 +01:00
claudio.atzori approved these changes 2023-10-30 15:51:29 +01:00
claudio.atzori removed review request for alessia.bardi 2023-10-30 15:51:50 +01:00
claudio.atzori commented 2023-10-30 15:55:08 +01:00
Owner
Copy Link

@claudio.atzori can we merge this for the next prod round? It seems to be important to get as many affiliations as possible for the Irish tender.

The changes are trivial, so I don't see major impediments in doing so. However, to keep things separated among beta and prod, can you propose the same PR against the master branch?

The epilogue of the ongoing beta graph update is still uncertain, hence I am not sure if I will be able to bring all the most recent changes included in the beta branch at this round to production.

> @claudio.atzori can we merge this for the next prod round? It seems to be important to get as many affiliations as possible for the Irish tender. The changes are trivial, so I don't see major impediments in doing so. However, to keep things separated among beta and prod, can you propose the same PR against the master branch? The epilogue of the ongoing beta graph update is still uncertain, hence I am not sure if I will be able to bring all the most recent changes included in the beta branch at this round to production.
schatz commented 2023-11-02 17:03:19 +01:00
Author
Member
Copy Link

PR for master branch is available here: #357

PR for master branch is available here: https://code-repo.d4science.org/D-Net/dnet-hadoop/pulls/357
schatz commented 2023-11-06 16:36:48 +01:00
Author
Member
Copy Link

@claudio.atzori I think, we can also merge this, to be inplace for the next beta round.

@claudio.atzori I think, we can also merge this, to be inplace for the next beta round.
claudio.atzori added 1 commit 2023-11-22 13:52:54 +01:00
claudio.atzori merged commit 836d7ec724 into beta 2023-11-22 13:53:07 +01:00
claudio.atzori deleted branch 9117_pubmed_affiliations 2023-11-22 13:53:07 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
claudio.atzori
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
OpenAIRE - DNet
Assignees
Clear assignees
No Assignees
claudio.atzori
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#353
No description provided.
Delete Branch "9117_pubmed_affiliations"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?