D-Net
/
dnet-hadoop
15
4
Fork 10
Code Issues 14 Pull Requests 8 Projects 2 Releases 1 Wiki Activity

Oozie workflow for cleancontext #216

Merged
claudio.atzori merged 7 commits from cleancontext into beta 2022-04-22 15:46:40 +02:00
Conversation 0 Commits 7 Files Changed 1 +142 -1
miriam.baglioni commented 2022-04-22 11:42:56 +02:00
Member
Copy Link

This PR extends the oozie workflow used for the cleaning with a last step to remove not wanted contexts from results. It adds also one parameter to the cleaning workflow shouldCleanContext that, if true, performes the cleaning

This PR extends the oozie workflow used for the cleaning with a last step to remove not wanted contexts from results. It adds also one parameter to the cleaning workflow **shouldCleanContext** that, if true, performes the cleaning
claudio.atzori was assigned by miriam.baglioni 2022-04-22 11:42:56 +02:00
miriam.baglioni added 4 commits 2022-04-22 11:42:58 +02:00
claudio.atzori reviewed 2022-04-22 11:46:50 +02:00
dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/clean/oozie_app/workflow.xml
@ -16,0 +18,4 @@
<description>true if the context have to be cleaned</description>
</property>
<property>
<name>contextId</name>
claudio.atzori commented 2022-04-22 11:46:50 +02:00
Owner
Copy Link

It should be better to include a description for this parameter, to explain its purpose and if it possible to include multiple contentIds, how they should be formatted.

It should be better to include a description for this parameter, to explain its purpose and if it possible to include multiple contentIds, how they should be formatted.
miriam.baglioni commented 2022-04-22 15:15:28 +02:00
Author
Member
Copy Link

This is just the first naive implementation of the context cleaning. I have no idea how it will be once done properly

This is just the first naive implementation of the context cleaning. I have no idea how it will be once done properly
claudio.atzori commented 2022-04-22 15:27:43 +02:00
Owner
Copy Link

It might be the 1st naive implementation, but looking at the oozie workflow, it is not obvious what a parameter plays when it is not accompanied by any description.

It might be the 1st naive implementation, but looking at the oozie workflow, it is not obvious what a parameter plays when it is not accompanied by any description.
miriam.baglioni commented 2022-04-22 15:42:35 +02:00
Author
Member
Copy Link

extended

extended
claudio.atzori reviewed 2022-04-22 11:47:22 +02:00
dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/clean/oozie_app/workflow.xml Outdated
@ -16,0 +22,4 @@
<value>sobigdata</value>
</property>
<property>
<name>verifyParam</name>
claudio.atzori commented 2022-04-22 11:47:22 +02:00
Owner
Copy Link

Missing description.

Missing description.
miriam.baglioni commented 2022-04-22 15:16:16 +02:00
Author
Member
Copy Link

Same holds as for the previous comment. Anyway if you think it is important to have the descriptions I will add them

Same holds as for the previous comment. Anyway if you think it is important to have the descriptions I will add them
claudio.atzori commented 2022-04-22 15:29:51 +02:00
Owner
Copy Link

I think it is. Again: this PR might just provide a first implementation, but the businness logic around these two parameters exists only in your head. To understand their role I'd need to open the actual job implementation and reverse engeener it. I would appreciate if you could add a description.

I think it is. Again: this PR might just provide a first implementation, but the businness logic around these two parameters exists only in your head. To understand their role I'd need to open the actual job implementation and reverse engeener it. I would appreciate if you could add a description.
claudio.atzori requested changes 2022-04-22 11:49:28 +02:00
claudio.atzori left a comment
Owner
Copy Link

Minor changes, please check the comments inline.

Minor changes, please check the comments inline.
dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/clean/CleanContextSparkJob.java
@ -1,16 +1,13 @@
claudio.atzori commented 2022-04-22 11:49:04 +02:00
Owner
Copy Link

At first glance, this class doesn't seem to include any significant change. If it was not changed, please revert to its original formatting. Otherwise the diff just creates noise.

At first glance, this class doesn't seem to include any significant change. If it was not changed, please revert to its original formatting. Otherwise the diff just creates noise.
claudio.atzori commented 2022-04-22 11:51:09 +02:00
Owner
Copy Link

This comment is outdated. It seems you did not issue git pull before introducing these further changes thus you did not get the reformatted file CleanContextSparkJob.java.

This comment is outdated. It seems you did not issue `git pull` before introducing these further changes thus you did not get the reformatted file CleanContextSparkJob.java.
claudio.atzori marked this conversation as resolved
claudio.atzori added 1 commit 2022-04-22 11:49:38 +02:00
miriam.baglioni added 2 commits 2022-04-22 15:41:50 +02:00
claudio.atzori merged commit 81242538e6 into beta 2022-04-22 15:46:40 +02:00
claudio.atzori deleted branch cleancontext 2022-04-22 15:46:45 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
claudio.atzori
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
claudio.atzori
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#216
No description provided.
Delete Branch "cleancontext"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?