mainPath
the working path of Datacite stores
isLookupUrl
The IS lookUp service endopoint
blocksize
100
The request block size
exportLinks
false
instructs the transformation phase to produce the links or not
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
${wf:conf('resumeFrom') eq 'TransformDatacite'}
yarn-cluster
cluster
ImportDatacite
eu.dnetlib.dhp.actionmanager.datacite.ImportDatacite
dhp-aggregation-${projectVersion}.jar
--executor-memory=${sparkExecutorMemory}
--executor-cores=${sparkExecutorCores}
--driver-memory=${sparkDriverMemory}
--conf spark.extraListeners=${spark2ExtraListeners}
--conf spark.sql.queryExecutionListeners=${spark2SqlQueryExecutionListeners}
--conf spark.yarn.historyServer.address=${spark2YarnHistoryServerAddress}
--conf spark.eventLog.dir=${nameNode}${spark2EventLogDir}
--targetPath${mainPath}/datacite_update
--dataciteDumpPath${mainPath}/datacite_dump
--namenode${nameNode}
--masteryarn-cluster
--blocksize${blocksize}
yarn-cluster
cluster
TransformJob
eu.dnetlib.dhp.actionmanager.datacite.GenerateDataciteDatasetSpark
dhp-aggregation-${projectVersion}.jar
--executor-memory=${sparkExecutorMemory}
--executor-cores=${sparkExecutorCores}
--driver-memory=${sparkDriverMemory}
--conf spark.sql.shuffle.partitions=3840
--conf spark.extraListeners=${spark2ExtraListeners}
--conf spark.sql.queryExecutionListeners=${spark2SqlQueryExecutionListeners}
--conf spark.yarn.historyServer.address=${spark2YarnHistoryServerAddress}
--conf spark.eventLog.dir=${nameNode}${spark2EventLogDir}
--sourcePath${mainPath}/datacite_dump
--targetPath${mainPath}/datacite_oaf
--isLookupUrl${isLookupUrl}
--exportLinks${exportLinks}
--masteryarn-cluster