D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 13 Projects 2 Releases 2 Wiki Activity

parameter naming conversions across the provision workflows #29

New Issue
Open
opened 2020-07-27 17:37:04 +02:00 by claudio.atzori · 0 comments
claudio.atzori commented 2020-07-27 17:37:04 +02:00
Owner
Copy Link

Currently the different oozie workflows assembled as the OpenAIRE data provision pipeline doesn't follow precise conventions for naming the input/output parameters.

An overview of the parameters available through the D-Net provisionw workflow definition:

shows that some simple naming conventions could allveviate the burden when it comes to maintain the parameter coherency across the oozie workflows as well as to introduce new graph processing steps, when needed.

From the descriptions below, there are many parameters describing the same concepts across different workflow steps.

aggregatorGraph

  • graphOutputPath
  • isLookupUrl

promoteActions

  • inputGraphRootPath
  • outputGraphRootPath
  • isLookupUrl

duplicateScan

  • graphBasePath // input
  • dedupGraphPath // output
  • isLookUpUrl

dedupConsistency

  • graphBasePath // input
  • dedupGraphPath // output

graphCleaning

  • graphInputPath
  • graphOutputPath
  • isLookupUrl

orcidPropagation

  • sourcePath
  • outputPath

bulkTagging

  • sourcePath
  • outputPath
  • isLookUpUrl

affiliationPropagation

  • sourcePath
  • outputPath

communityOrganizationPropagation

  • sourcePath
  • outputPath

resultProjectPropagation

  • sourcePath
  • outputPath

communitySemrelPropagation

  • sourcePath
  • outputPath
  • isLookUpUrl

countryPropagation

  • sourcePath
  • outputPath

blacklistRelations

  • sourcePath
  • outputPath

In particular, the most recurring parameters describe

  1. the input/output graph. They currently refer to the HDFS path, but the parameter could abstract a bit and refer generically to non ambiguous input/output graph names;
  2. URL to the information system lookup service, which luckily only suffers from 2 different spellings, isLookUpUrl vs isLookupUrl.

So, I propose to adopt the same naming conventions for the three parameters:

  • inputGraph
  • outputGraph
  • isLookUpUrl
Currently the different oozie workflows assembled as the OpenAIRE data provision pipeline doesn't follow precise conventions for naming the input/output parameters. An overview of the parameters available through the D-Net provisionw workflow definition: * [BETA - Graph construction for IIS](https://beta.services.openaire.eu/is/mvc/ui/isManager.do#/profile/b05c97e6-69b5-497d-87fd-2137d3ff2c2e_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) * [BETA - Graph construction for PROVISION](https://beta.services.openaire.eu/is/mvc/ui/isManager.do#/profile/b05c97e6-69b5-497d-87fd-2137d3ff2c2e_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) * [PROD - Graph construction for IIS](https://services.openaire.eu/is/mvc/ui/isManager.do#/profile/4801c33c-66ca-4ab6-af64-aa812194ec68_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) * [PROD - Graph construction for PROVISION](https://services.openaire.eu/is/mvc/ui/isManager.do#/profile/74d90d54-bea4-4a79-82d9-adddcc89e660_V29ya2Zsb3dEU1Jlc291cmNlcy9Xb3JrZmxvd0RTUmVzb3VyY2VUeXBl) shows that some simple naming conventions could allveviate the burden when it comes to maintain the parameter coherency across the oozie workflows as well as to introduce new graph processing steps, when needed. From the descriptions below, there are many parameters describing the same concepts across different workflow steps. **aggregatorGraph** * graphOutputPath * isLookupUrl **promoteActions** * inputGraphRootPath * outputGraphRootPath * isLookupUrl **duplicateScan** * graphBasePath // input * dedupGraphPath // output * isLookUpUrl **dedupConsistency** * graphBasePath // input * dedupGraphPath // output **graphCleaning** * graphInputPath * graphOutputPath * isLookupUrl **orcidPropagation** * sourcePath * outputPath **bulkTagging** * sourcePath * outputPath * isLookUpUrl **affiliationPropagation** * sourcePath * outputPath **communityOrganizationPropagation** * sourcePath * outputPath **resultProjectPropagation** * sourcePath * outputPath **communitySemrelPropagation** * sourcePath * outputPath * isLookUpUrl **countryPropagation** * sourcePath * outputPath **blacklistRelations** * sourcePath * outputPath In particular, the most recurring parameters describe 1. the input/output graph. They currently refer to the HDFS path, but the parameter could abstract a bit and refer generically to non ambiguous input/output graph names; 2. URL to the information system lookup service, which luckily only suffers from 2 different spellings, `isLookUpUrl` vs `isLookupUrl`. So, I propose to adopt the same naming conventions for the three parameters: * inputGraph * outputGraph * isLookUpUrl
claudio.atzori self-assigned this 2020-07-27 17:37:04 +02:00
miriam.baglioni was assigned by claudio.atzori 2020-07-27 17:37:04 +02:00
claudio.atzori added the
enhancement
 label 2020-07-27 18:01:24 +02:00
Sign in to join this conversation.
master
Branches Tags
Clear current reference
main
beta
beta_jan2025
propagateorcid
COnnectSubCommunities
csv_collector_plugins
kubernetes
solr_v9
affiliation-matching
hive_graph_db
cleaning_10215
beta-ukripublication
scholix_small_index
Communities_patents
crossref_mapping_improvement
raid_actionset
zenodo_dump_collection
datacite_dump_import
incremental_graph_main
main_bulktag
actionset_promote_sequential
person_through_the_graph_relazioniattributate
pubmed_fix
dedup_new_comparators
abtracts_guidelines4
propagateorcid_publisher
merge_resulttypes
betaFixPerson
k8s_orcidpropagation
raw_affiliation_mapping
mergedids
beta_provision_alignment_9.0.0
bestinstancetype
person_through_the_graph
person_through_the_graph_newDevelopments
affroNewModelonBeta
beta_fixes_oct
dedup_local_test_implementation
fix_decision_tree
pid_cleaning
affRoModelModificationOnmain
beta2main_sept2024
osfPreprints_plugin
fix_missing_project_rels
enrich_more_subject_bug
gtr2Publications_plugin
merge_by_id_fix
dedup_blacklist_fix
claim-orgs
9126-impact-indicators-wf-optimisation
AffiliationFromPublisher
pubmed_aggregation_improvements
betaAffiliationFromPublisherPages
oozie_spark_params
GraphAnnotation
rest-collector-request-header-map2
dedup_countryInference_NPE
webCrawlLessBlackList
fos_l1l2
sdgnodoi
person
tagOrganization
entity_contexts
broker_orcid
metadata_collection_java_upgrade
research_fi_collector_plugin
openorgs_optimization
checkContextOnDatasourceAndProject
affRoFromRawStringmain
openorgs_fixes
master
9872-create-solr-collection-aliases
json_payload
pivotselectionbypid
import_openorg_type
irish-oaipmh-exporter
fix_mergedcliquesort
spark34-integration
beta_indexing_May2024
mag_only_doi
rest-collector-plugin-with-retry
dependency-revision
beta-release-1.2.5
misc_fixes_merge_entities
WebCrowlBeta
WebCrowl
doidoost_dismiss
CrossrefFundersMap
taggingProjects
9647_datacite_affiliations
UsageStatsRecordDS
mergeutils
oaf_country_beta
index_records
9559_DBLP_data
base_stats_job_deprecated
SWH_issue_377
spark_join_param_tuning
crossref_mapping_vocabularies
oldPropagationOrganizationCommunity
orcid_import
dedup-with-dataframe-spark34
ticket_8369
tweaking_spark_params
fc4e-rsac
doiboost_authormerger
beta_dedup_configuration
scholix_flat_indexing
scholix_data_type_openaire
subjectPropagation
enrichment
validation
Clear current reference
2024-10-BETA
2024-10#10068
september-2023
dhp-1.2.4
archive/master_pre_stable_ids
archive/bipFinder_master_test
archive/islookup_timeout
dhp-1.2.3
dhp-1.2.2
dhp-1.2.1
dhp-1.2.0
dhp-1.1.7
dhp-1.1.6
archive/junit5
archive/dedupTest
dhp-1.1.5
dhp-1.0.4
dhp-1.0.3
dhp-1.0.2
1.0.1
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
claudio.atzori
miriam.baglioni
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#29
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?