D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 13 Projects 2 Releases 2 Wiki Activity

[Measures] added new measure (UsageCounts) #214

Merged
claudio.atzori merged 8 commits from eosc_dimitris into beta 2022-04-21 12:19:19 +02:00
Conversation 0 Commits 8 Files Changed 9 +632
miriam.baglioni commented 2022-04-20 16:21:06 +02:00
Member
Copy Link

This PR introduces a new oozie workflow for the creation of a new action set that includes the usage counts (views and downloads) as a new measure on the result level.

It will have as provenaceaction.classid the value measure:usage_counts and as id for the measure the keywords downloads and views referring respectively to the attribute with the same names in the usage stats table, while the key for the unit will be count.

The values are calculated as the sum of the counts, aggregated for each record id.

The remaining part of the measure element is set exactly as for the bip!Finder integration.

This PR introduces a new oozie workflow for the creation of a new action set that includes the usage counts (views and downloads) as a new measure on the result level. It will have as **provenaceaction.classid** the value **measure:usage_counts** and as id for the measure the keywords **downloads** and **views** referring respectively to the attribute with the same names in the usage stats table, while the **key** for the unit will be **count**. The values are calculated as the sum of the counts, aggregated for each record id. The remaining part of the measure element is set exactly as for the **bip!Finder** integration.
miriam.baglioni added the
enhancement
 label 2022-04-20 16:21:06 +02:00
claudio.atzori was assigned by miriam.baglioni 2022-04-20 16:21:06 +02:00
miriam.baglioni added 3 commits 2022-04-20 16:21:07 +02:00
miriam.baglioni added 1 commit 2022-04-21 11:27:39 +02:00
claudio.atzori changed title from [Measures] added new measure (usagecounts) to [Measures] added new measure (UsageCounts) 2022-04-21 11:49:02 +02:00
claudio.atzori reviewed 2022-04-21 11:50:39 +02:00
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/OafMapperUtils.java Outdated
@ -392,2 +392,4 @@
return null;
}
public static KeyValue newKeyValueInstance (String key, String value, DataInfo dataInfo){
claudio.atzori commented 2022-04-21 11:50:39 +02:00
Owner
Copy Link

Please compile before pushing. It seems the code formatting was not applied here.

Please compile before pushing. It seems the code formatting was not applied here.
claudio.atzori reviewed 2022-04-21 11:52:51 +02:00
dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/actionmanager/usagestats/SparkAtomicActionUsageJob.java
@ -0,0 +33,4 @@
/**
* created the Atomic Action for each tipe of results
claudio.atzori commented 2022-04-21 11:52:51 +02:00
Owner
Copy Link

typo: tipe == type

typo: tipe == type
miriam.baglioni marked this conversation as resolved
claudio.atzori reviewed 2022-04-21 11:56:25 +02:00
dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/actionmanager/usagestats/SparkAtomicActionUsageJob.java
@ -0,0 +65,4 @@
SparkConf conf = new SparkConf();
conf.set("hive.metastore.uris", parser.get("hive_metastore_uris"));
final String dbname = parser.get("statsdb");
claudio.atzori commented 2022-04-21 11:56:25 +02:00
Owner
Copy Link

Minor: I would name the parameter as usagestatsdb to avoid confusion with the other stats database. In this context there is no ambiguity, but from the workflow caller there might be.

Minor: I would name the parameter as `usagestatsdb` to avoid confusion with the other stats database. In this context there is no ambiguity, but from the workflow caller there might be.
miriam.baglioni marked this conversation as resolved
claudio.atzori reviewed 2022-04-21 11:57:08 +02:00
dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/usagestats/input_actionset_parameter.json Outdated
@ -0,0 +19,4 @@
},
{
"paramName": "sdb",
"paramLongName": "statsdb",
claudio.atzori commented 2022-04-21 11:57:07 +02:00
Owner
Copy Link

Minor: I would name the parameter as usagestatsdb to avoid confusion with the other stats database. In this context there is no ambiguity, but from the workflow caller there might be.

Minor: I would name the parameter as `usagestatsdb` to avoid confusion with the other stats database. In this context there is no ambiguity, but from the workflow caller there might be.
claudio.atzori reviewed 2022-04-21 11:57:25 +02:00
dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/usagestats/oozie_app/workflow.xml Outdated
@ -0,0 +5,4 @@
<description>the path where to store the actionset</description>
</property>
<property>
<name>statsdb</name>
claudio.atzori commented 2022-04-21 11:57:25 +02:00
Owner
Copy Link

Minor: I would name the parameter as usagestatsdb to avoid confusion with the other stats database. In this context there is no ambiguity, but from the workflow caller there might be.

Minor: I would name the parameter as `usagestatsdb` to avoid confusion with the other stats database. In this context there is no ambiguity, but from the workflow caller there might be.
claudio.atzori reviewed 2022-04-21 11:57:56 +02:00
dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/actionmanager/usagestats/oozie_app/workflow.xml Outdated
@ -0,0 +6,4 @@
</property>
<property>
<name>statsdb</name>
<description>the path where to store the actionset</description>
claudio.atzori commented 2022-04-21 11:57:56 +02:00
Owner
Copy Link

the description is wrong. Looks like a copy&paste

the description is wrong. Looks like a copy&paste
claudio.atzori requested changes 2022-04-21 11:59:51 +02:00
claudio.atzori left a comment
Owner
Copy Link

Overall looks ok, just minor changes. Please check the inline comments and remember to compile before issuing the commit&push so that the common formatting is applied.

Overall looks ok, just minor changes. Please check the inline comments and remember to compile before issuing the commit&push so that the common formatting is applied.
claudio.atzori added 1 commit 2022-04-21 12:00:08 +02:00
miriam.baglioni added 2 commits 2022-04-21 12:11:01 +02:00
miriam.baglioni added 1 commit 2022-04-21 12:14:13 +02:00
claudio.atzori merged commit a289c9eae2 into beta 2022-04-21 12:19:19 +02:00
claudio.atzori deleted branch eosc_dimitris 2022-04-21 12:19:22 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
claudio.atzori
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
claudio.atzori
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#214
No description provided.
Delete Branch "eosc_dimitris"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?