D-Net
/
dnet-hadoop
15
4
Fork 10
Code Issues 14 Pull Requests 10 Projects 2 Releases 1 Wiki Activity

8172_impact_indicators_workflow #284

Merged
miriam.baglioni merged 49 commits from 8172_impact_indicators_workflow into beta 2023-08-14 15:50:48 +02:00
Conversation 7 Commits 49 Files Changed 23 +2508 -75
schatz commented 2023-03-23 12:08:56 +01:00
Member
Copy Link

This PR adds an Oozie workflow for the calculation of impact indicators using https://github.com/athenarc/Bip-Ranker in graph provision.

This PR adds an Oozie workflow for the calculation of impact indicators using https://github.com/athenarc/Bip-Ranker in graph provision.
claudio.atzori was assigned by schatz 2023-03-23 12:08:56 +01:00
schatz added 8 commits 2023-03-23 12:08:56 +01:00
schatz added 1 commit 2023-04-07 15:30:20 +02:00
schatz added 3 commits 2023-04-26 19:40:15 +02:00
schatz added 1 commit 2023-04-27 11:37:20 +02:00
ilias.kanellos added 2 commits 2023-04-28 12:32:21 +02:00
ilias.kanellos added 1 commit 2023-04-28 12:45:39 +02:00
ilias.kanellos added 1 commit 2023-05-11 13:43:42 +02:00
ilias.kanellos added 1 commit 2023-05-11 14:37:41 +02:00
ilias.kanellos added 1 commit 2023-05-12 11:56:09 +02:00
schatz added 1 commit 2023-05-15 12:04:47 +02:00
ilias.kanellos added 1 commit 2023-05-15 14:25:21 +02:00
schatz added 2 commits 2023-05-15 14:53:16 +02:00
schatz added 1 commit 2023-05-15 15:20:30 +02:00
schatz added 1 commit 2023-05-15 16:52:23 +02:00
schatz added 1 commit 2023-05-15 18:55:38 +02:00
schatz added 1 commit 2023-05-15 21:28:52 +02:00
schatz added 1 commit 2023-05-16 13:40:10 +02:00
schatz added 1 commit 2023-05-16 15:28:52 +02:00
ilias.kanellos added 3 commits 2023-05-16 16:35:30 +02:00
ilias.kanellos added 1 commit 2023-05-23 15:44:18 +02:00
ilias.kanellos added 1 commit 2023-05-23 15:49:16 +02:00
ilias.kanellos added 1 commit 2023-05-23 16:17:25 +02:00
schatz added 1 commit 2023-06-23 11:51:55 +02:00
ilias.kanellos added 3 commits 2023-07-06 14:07:08 +02:00
schatz added 1 commit 2023-07-18 17:38:31 +02:00
ilias.kanellos added 2 commits 2023-07-21 12:43:34 +02:00
schatz added 2 commits 2023-07-21 14:27:21 +02:00
schatz added 1 commit 2023-07-21 15:07:13 +02:00
ilias.kanellos added 2 commits 2023-07-21 16:43:19 +02:00
schatz added 1 commit 2023-07-21 16:55:45 +02:00
schatz changed title from WIP: 8172_impact_indicators_workflow to 8172_impact_indicators_workflow 2023-07-25 11:50:43 +02:00
schatz commented 2023-07-25 11:55:05 +02:00
Author
Member
Copy Link

The workflow has been successfully tested and is ready to be merged.
It first calculates the impact indicators for results producing the relevant action sets, and then aggregates those indicators for projects, also producing appropriate action sets.

Since,the performance of this WF was discussed in the past, I attach the execution times using the last prod graph (v6.0.0).

The workflow has been successfully tested and is ready to be merged. It first calculates the impact indicators for results producing the relevant action sets, and then aggregates those indicators for projects, also producing appropriate action sets. Since,the performance of this WF was discussed in the past, I attach the execution times using the last prod graph (v6.0.0).
Screenshot 2023-07-25 at 12.45.19 PM.png
261 KiB
Screenshot 2023-07-25 at 12.56.25 PM.png
95 KiB
Screenshot 2023-07-25 at 12.45.19 PM.png Screenshot 2023-07-25 at 12.56.25 PM.png
schatz requested review from claudio.atzori 2023-07-25 11:55:16 +02:00
schatz requested review from miriam.baglioni 2023-07-25 11:55:23 +02:00
claudio.atzori requested review from giambattista.bloisi 2023-07-28 10:59:01 +02:00
miriam.baglioni commented 2023-08-07 16:03:59 +02:00
Member
Copy Link

I see you are creating two different action sets: one for projects and one fro results.
We already have an action set to store the update related to the computation of the impact indicators, you should use the same for both projects and results.

It seems they are not two distinct action sets, but that you are changing the destination directory for the same AS adding in one case /result and in the other case /project to the path you should use to store the files.

You should avoid doing it. The action sets are associated to profiles on the aggregator. This profiles are updated each time the action set is generated. The profile specifies the association between the last update for the action set and the directory that stores the update.
When the AS in included in the graph, the directory where to search for the files is the one specified in the profile.

~~I see you are creating two different action sets: one for projects and one fro results. We already have an action set to store the update related to the computation of the impact indicators, you should use the same for both projects and results.~~ It seems they are not two distinct action sets, but that you are changing the destination directory for the same AS adding in one case /result and in the other case /project to the path you should use to store the files. You should avoid doing it. The action sets are associated to profiles on the aggregator. This profiles are updated each time the action set is generated. The profile specifies the association between the last update for the action set and the directory that stores the update. When the AS in included in the graph, the directory where to search for the files is the one specified in the profile.
miriam.baglioni changed title from 8172_impact_indicators_workflow to WIP: 8172_impact_indicators_workflow 2023-08-07 16:23:52 +02:00
schatz added 1 commit 2023-08-11 14:56:59 +02:00
schatz changed title from WIP: 8172_impact_indicators_workflow to 8172_impact_indicators_workflow 2023-08-11 14:57:34 +02:00
schatz commented 2023-08-11 14:58:46 +02:00
Author
Member
Copy Link

Just updated the code to store the action sets for projects and results in the same file.

Just updated the code to store the action sets for projects and results in the same file.
giambattista.bloisi commented 2023-08-11 16:33:11 +02:00
Member
Copy Link

The changes in SparkAtomicActionScoreJob seem to break the other existing workflow using it (dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/bipfinder/oozie_app/workflow.xml)

The changes in SparkAtomicActionScoreJob seem to break the other existing workflow using it (dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/bipfinder/oozie_app/workflow.xml)
miriam.baglioni commented 2023-08-11 17:03:54 +02:00
Member
Copy Link

I think we can remove that workflow, since this implementation should replace the old asynchronous one

I think we can remove that workflow, since this implementation should replace the old asynchronous one
schatz commented 2023-08-11 17:46:00 +02:00
Author
Member
Copy Link

I think we can remove that workflow, since this implementation should replace the old asynchronous one

yes, I was thinking the same, but to be on the safe side, I will adapt this workflow in case it is needed.

> I think we can remove that workflow, since this implementation should replace the old asynchronous one > yes, I was thinking the same, but to be on the safe side, I will adapt this workflow in case it is needed.
miriam.baglioni commented 2023-08-14 15:50:37 +02:00
Member
Copy Link

As agreed with Serafeim, I will merge this PR and the changes to the other related WF (if not removed) will be done in afterwards.

As agreed with Serafeim, I will merge this PR and the changes to the other related WF (if not removed) will be done in afterwards.
miriam.baglioni merged commit 9c8b41475a into beta 2023-08-14 15:50:48 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
claudio.atzori
miriam.baglioni
giambattista.bloisi
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
claudio.atzori
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#284
No description provided.
Delete Branch "8172_impact_indicators_workflow"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?