sourceNN
the source name node
isLookupUrl
the isLookup service endpoint
workingDirectory
working directory
distcp_memory_mb
6144
memory for distcp copying actionsets from remote cluster
distcp_task_timeout
60000000
timeout for distcp copying actions from remote cluster
distcp_num_maps
1
mmaximum number of map tasks used in the distcp process
transform_only
activate tranform-only mode. Only apply transformation step
sparkDriverMemory
memory for driver process
sparkExecutorMemory
memory for individual executor
sparkExecutorCores
number of cores used by single executor
oozieActionShareLibForSpark2
oozie action sharelib for spark 2.*
spark2ExtraListeners
com.cloudera.spark.lineage.NavigatorAppListener
spark 2.* extra listeners classname
spark2SqlQueryExecutionListeners
com.cloudera.spark.lineage.NavigatorQueryListener
spark 2.* sql query execution listeners classname
spark2YarnHistoryServerAddress
spark 2.* yarn history server address
spark2EventLogDir
spark 2.* event log dir location
${jobTracker}
${nameNode}
mapreduce.job.queuename
${queueName}
oozie.launcher.mapred.job.queue.name
${oozieLauncherQueueName}
oozie.action.sharelib.for.spark
${oozieActionShareLibForSpark2}
eu.dnetlib.dhp.actionmanager.migration.MigrateActionSet
-Dmapred.task.timeout=${distcp_task_timeout}
--isLookupUrl${isLookupUrl}
--sourceNameNode${sourceNN}
--targetNameNode${nameNode}
--workingDirectory${workingDirectory}
--distcp_num_maps${distcp_num_maps}
--distcp_memory_mb${distcp_memory_mb}
--distcp_task_timeout${distcp_task_timeout}
--transform_only${transform_only}
yarn
cluster
transform_actions
eu.dnetlib.dhp.actionmanager.migration.TransformActions
dhp-actionmanager-${projectVersion}.jar
--executor-cores=${sparkExecutorCores}
--executor-memory=${sparkExecutorMemory}
--driver-memory=${sparkDriverMemory}
--conf spark.extraListeners=${spark2ExtraListeners}
--conf spark.sql.queryExecutionListeners=${spark2SqlQueryExecutionListeners}
--conf spark.yarn.historyServer.address=${spark2YarnHistoryServerAddress}
--conf spark.eventLog.dir=${nameNode}${spark2EventLogDir}
--isLookupUrl${isLookupUrl}
--inputPaths${wf:actionData('migrate_actionsets')['target_paths']}
migrate_actions failed, error message[${wf:errorMessage(wf:lastErrorNode())}]