sourcePath
the working dir base path
targetPath
the working dir base path
workingPath
the working dir base path
sparkDriverMemory
memory for driver process
sparkExecutorMemory
memory for individual executor
sparkExecutorCores
number of cores used by single executor
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
yarn-cluster
cluster
Convert Mag to Dataset
eu.dnetlib.doiboost.mag.SparkImportMagIntoDataset
dhp-doiboost-${projectVersion}.jar
--executor-memory=${sparkExecutorMemory}
--executor-cores=${sparkExecutorCores}
--driver-memory=${sparkDriverMemory}
${sparkExtraOPT}
--sourcePath${sourcePath}
--targetPath${workingPath}
--masteryarn-cluster
yarn-cluster
cluster
Convert Mag to OAF Dataset
eu.dnetlib.doiboost.mag.SparkPreProcessMAG
dhp-doiboost-${projectVersion}.jar
--executor-memory=${sparkExecutorMemory}
--executor-cores=${sparkExecutorCores}
--driver-memory=${sparkDriverMemory}
--conf spark.sql.shuffle.partitions=3840
${sparkExtraOPT}
--sourcePath${workingPath}
--workingPath${workingPath}/process
--targetPath${targetPath}
--masteryarn-cluster