D-Net
/
dnet-hadoop
15
4
Fork 10
Code Issues 14 Pull Requests 8 Projects 2 Releases 1 Wiki Activity

Labels Milestones

New Issue

parameter naming conversions across the provision workflows #29

Open
opened 4 years ago by claudio.atzori · 0 comments
claudio.atzori commented 4 years ago
Owner

Currently the different oozie workflows assembled as the OpenAIRE data provision pipeline doesn't follow precise conventions for naming the input/output parameters.

An overview of the parameters available through the D-Net provisionw workflow definition:

shows that some simple naming conventions could allveviate the burden when it comes to maintain the parameter coherency across the oozie workflows as well as to introduce new graph processing steps, when needed.

From the descriptions below, there are many parameters describing the same concepts across different workflow steps.

aggregatorGraph

  • graphOutputPath
  • isLookupUrl

promoteActions

  • inputGraphRootPath
  • outputGraphRootPath
  • isLookupUrl

duplicateScan

  • graphBasePath // input
  • dedupGraphPath // output
  • isLookUpUrl

dedupConsistency

  • graphBasePath // input
  • dedupGraphPath // output

graphCleaning

  • graphInputPath
  • graphOutputPath
  • isLookupUrl

orcidPropagation

  • sourcePath
  • outputPath

bulkTagging

  • sourcePath
  • outputPath
  • isLookUpUrl

affiliationPropagation

  • sourcePath
  • outputPath

communityOrganizationPropagation

  • sourcePath
  • outputPath

resultProjectPropagation

  • sourcePath
  • outputPath

communitySemrelPropagation

  • sourcePath
  • outputPath
  • isLookUpUrl

countryPropagation

  • sourcePath
  • outputPath

blacklistRelations

  • sourcePath
  • outputPath

In particular, the most recurring parameters describe

  1. the input/output graph. They currently refer to the HDFS path, but the parameter could abstract a bit and refer generically to non ambiguous input/output graph names;
  2. URL to the information system lookup service, which luckily only suffers from 2 different spellings, isLookUpUrl vs isLookupUrl.

So, I propose to adopt the same naming conventions for the three parameters:

  • inputGraph
  • outputGraph
  • isLookUpUrl
Currently the different oozie workflows assembled as the OpenAIRE data provision pipeline doesn't follow precise conventions for naming the input/output parameters. An overview of the parameters available through the D-Net provisionw workflow definition: * [BETA - Graph construction for IIS](https://beta.services.openaire.eu/is/mvc/ui/isManager.do#/profile/b05c97e6-69b5-497d-87fd-2137d3ff2c2e_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) * [BETA - Graph construction for PROVISION](https://beta.services.openaire.eu/is/mvc/ui/isManager.do#/profile/b05c97e6-69b5-497d-87fd-2137d3ff2c2e_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) * [PROD - Graph construction for IIS](https://services.openaire.eu/is/mvc/ui/isManager.do#/profile/4801c33c-66ca-4ab6-af64-aa812194ec68_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) * [PROD - Graph construction for PROVISION](https://services.openaire.eu/is/mvc/ui/isManager.do#/profile/74d90d54-bea4-4a79-82d9-adddcc89e660_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) shows that some simple naming conventions could allveviate the burden when it comes to maintain the parameter coherency across the oozie workflows as well as to introduce new graph processing steps, when needed. From the descriptions below, there are many parameters describing the same concepts across different workflow steps. **aggregatorGraph** * graphOutputPath * isLookupUrl **promoteActions** * inputGraphRootPath * outputGraphRootPath * isLookupUrl **duplicateScan** * graphBasePath // input * dedupGraphPath // output * isLookUpUrl **dedupConsistency** * graphBasePath // input * dedupGraphPath // output **graphCleaning** * graphInputPath * graphOutputPath * isLookupUrl **orcidPropagation** * sourcePath * outputPath **bulkTagging** * sourcePath * outputPath * isLookUpUrl **affiliationPropagation** * sourcePath * outputPath **communityOrganizationPropagation** * sourcePath * outputPath **resultProjectPropagation** * sourcePath * outputPath **communitySemrelPropagation** * sourcePath * outputPath * isLookUpUrl **countryPropagation** * sourcePath * outputPath **blacklistRelations** * sourcePath * outputPath In particular, the most recurring parameters describe 1. the input/output graph. They currently refer to the HDFS path, but the parameter could abstract a bit and refer generically to non ambiguous input/output graph names; 2. URL to the information system lookup service, which luckily only suffers from 2 different spellings, `isLookUpUrl` vs `isLookupUrl`. So, I propose to adopt the same naming conventions for the three parameters: * inputGraph * outputGraph * isLookUpUrl
claudio.atzori self-assigned this 4 years ago
miriam.baglioni was assigned by claudio.atzori 4 years ago
claudio.atzori added the
enhancement
 label 4 years ago
Sign in to join this conversation.
master
Branches Tags
Clear current reference
spark34-integration
beta
misc_fixes_merge_entities
beta-release-1.2.5
WebCrowlBeta
WebCrowl
master
provision_memoryOverhead
stats_step16_fix
doidoost_dismiss
CrossrefFundersMap
taggingProjects
9647_datacite_affiliations
UsageStatsRecordDS
mergeutils
oaf_country_beta
index_records
ocnew
FOSNew
bulkTaggingPathMapExtention
transformativeagreement
new_orcid_enhancement
9559_DBLP_data
base_stats_job_deprecated
SWH_issue_377
import_orps_fix
spark_join_param_tuning
crossref_mapping_vocabularies
promote_actions_join_type_master
promote_actions_join_type
provision_community_api
enrichmentSingleStepFixed
fosPreparationBeta
resource_types
enrichmentSingleStep
oldPropagationOrganizationCommunity
beta_to_master_dicember2023
orcid_import
9078_xml_records_irish_tender
clean_license_publisher
bulkTag
SWH_integration
importpoci
8172_impact_indicators_workflow
dedup-with-dataframe-spark34
8876
master_july23
distinct_pids_from_openorgs_beta
propagationProjectThroughParentChils
fulltext_url_validation
removeTaggingCondition
ticket_8369
tweaking_spark_params
fc4e-rsac
doiboost_authormerger
beta_dedup_configuration
apc_affiliation
bulkTagRefactor
organizationToRepresentative
graph_cleaning_refactoring
scholix_flat_indexing
scholix_data_type_openaire
advConstraintsInBeta
doiboostMappingExtention
mag_citation_relation
h2020classification
doiboostFunderExtention
citations_monodirectional
compatibility_order
8232-mdstore-synch-improve
subjectPropagation
pubmed_to_production
cleanCountryOnMaster
graph_cleaning
deduptesting
horizontalConstraints
enrichment
scholix_to_solr
transformation_wf
discard-non-wellformed
removeDump
eosc_context_tagging
pubmed_update
doiboost_refactor
clean_context_master
monitoring
dump_new_funded_products
dump_delta_projects
dump
7096-fileGZip-collector-plugin
oaf_relation_mapping
validation
native_records_migration
instance_group_by_url
hostedByMap_update
hostedByMap_oastartdate
sygma_indexing
Clear current reference
dhp-1.2.4
dhp-1.2.3
dhp-1.2.2
dhp-1.2.1
dhp-1.2.0
dhp-1.1.7
dhp-1.1.6
dhp-1.1.5
dhp-1.0.4
dhp-1.0.3
dhp-1.0.2
1.0.1
september-2023
archive/master_pre_stable_ids
archive/junit5
archive/islookup_timeout
archive/dedupTest
archive/bipFinder_master_test
Labels
Apply labels
Clear labels   
bug

Something is not working   
duplicate

This issue or pull request already exists   
enhancement

New feature / refactoring   
help wanted

Need some help   
invalid

Something is wrong   
question

More information is needed   
RDGraph

EOSC Research Discovery Graph   
RSAC

EOSC Research Software APIs and Connectors   
wontfix

This won't be fixed
No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Set milestone
Clear milestone
No items
No Milestone
Projects
Set Project
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
miriam.baglioni
claudio.atzori
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#29
Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch '%!s(<nil>)'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes