workingDirPath
the source path
sparkDriverMemory
memory for driver process
sparkExecutorMemory
memory for individual executor
index
index name
timestamp
timestamp from incremental harvesting
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
${jobTracker}
${nameNode}
eu.dnetlib.dhp.provision.update.RetrieveUpdateFromDatacite
-t${workingDirPath}/synch/input_json
-n${nameNode}
-ts${timestamp}
-ihip-90-147-167-25.ct1.garrservices.it
-indatacite
${jobTracker}
${nameNode}
yarn-cluster
cluster
resolve and generate Scholix
eu.dnetlib.dhp.provision.update.SparkResolveScholixTarget
dhp-graph-provision-scholexplorer-${projectVersion}.jar
--executor-memory ${sparkExecutorMemory} --driver-memory=${sparkDriverMemory} ${sparkExtraOPT} --conf spark.dynamicAllocation.maxExecutors="32"
-m yarn-cluster
-s${workingDirPath}/synch/input_json
-w${workingDirPath}/synch
-hip-90-147-167-25.ct1.garrservices.it
${jobTracker}
${nameNode}
yarn-cluster
cluster
index scholix
eu.dnetlib.dhp.provision.SparkIndexCollectionOnES
dhp-graph-provision-scholexplorer-${projectVersion}.jar
--executor-memory ${sparkExecutorMemory} --driver-memory=${sparkDriverMemory} ${sparkExtraOPT} --conf spark.dynamicAllocation.maxExecutors="8"
-mt yarn-cluster
--sourcePath${workingDirPath}/synch/resolved_json
--index${index}_scholix
--idPathidentifier
--typescholix