mdStorePath
the path of the native mdstore
apiDescription
A json encoding of the API Description class
dataSourceInfo
A json encoding of the Datasource Info
identifierPath
An xpath to retrieve the metadata idnentifier for the generation of DNet Identifier
metadataEncoding
The type of the metadata XML/JSON
timestamp
The timestamp of the collection date
workflowId
The identifier of the workflow
${jobTracker}
${nameNode}
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
eu.dnetlib.dhp.collection.worker.CollectorWorkerApplication
--hdfsPath${workingDir}/sequenceFile_${mdstoreVersion}
--apidescriptor${apiDescription}
--namenode${nameNode}
yarn
cluster
Generate Native MetadataStore
eu.dnetlib.dhp.collection.GenerateNativeStoreSparkJob
dhp-aggregation-${projectVersion}.jar
--executor-memory=${sparkExecutorMemory}
--executor-cores=${sparkExecutorCores}
--driver-memory=${sparkDriverMemory}
--conf spark.extraListeners=${spark2ExtraListeners}
--conf spark.sql.queryExecutionListeners=${spark2SqlQueryExecutionListeners}
--conf spark.yarn.historyServer.address=${spark2YarnHistoryServerAddress}
--conf spark.eventLog.dir=${nameNode}${spark2EventLogDir}
--encoding${metadataEncoding}
--dateOfCollection${timestamp}
--provenance${dataSourceInfo}
--xpath${identifierPath}
--input${workingDir}/sequenceFile
--output${mdStorePath}
-w${workflowId}