datacitePath
the path of Datacite spark dataset
isLookupUrl
The IS lookUp service endopoint
crossrefPath
the path of Crossref spark dataset
targetPath
the path of Crossref spark dataset
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
yarn-cluster
cluster
ImportDatacite
eu.dnetlib.dhp.actionmanager.datacite.GenerateDataciteDatasetSpark
dhp-aggregation-${projectVersion}.jar
--executor-memory=${sparkExecutorMemory}
--executor-cores=${sparkExecutorCores}
--driver-memory=${sparkDriverMemory}
--conf spark.sql.shuffle.partitions=3840
--conf spark.extraListeners=${spark2ExtraListeners}
--conf spark.sql.queryExecutionListeners=${spark2SqlQueryExecutionListeners}
--conf spark.yarn.historyServer.address=${spark2YarnHistoryServerAddress}
--conf spark.eventLog.dir=${nameNode}${spark2EventLogDir}
--sourcePath${datacitePath}
--targetPath${targetPath}/datacite_oaf
--isLookupUrl${isLookupUrl}
--exportLinkstrue
--masteryarn-cluster
yarn-cluster
cluster
FilterCrossrefEntities
eu.dnetlib.dhp.actionmanager.datacite.FilterCrossrefEntitiesSpark
dhp-aggregation-${projectVersion}.jar
--executor-memory=${sparkExecutorMemory}
--executor-cores=${sparkExecutorCores}
--driver-memory=${sparkDriverMemory}
--conf spark.sql.shuffle.partitions=3840
--conf spark.extraListeners=${spark2ExtraListeners}
--conf spark.sql.queryExecutionListeners=${spark2SqlQueryExecutionListeners}
--conf spark.yarn.historyServer.address=${spark2YarnHistoryServerAddress}
--conf spark.eventLog.dir=${nameNode}${spark2EventLogDir}
--sourcePath${crossrefPath}
--targetPath${targetPath}/crossref_oaf
--masteryarn-cluster