migrationPathStep1
the base path to store hdfs file
migrationPathStep2
the temporary path to store entities before dispatching
isLookupUrl
the address of the lookUp service
sparkDriverMemory
memory for driver process
sparkExecutorMemory
memory for individual executor
sparkExecutorCores
number of cores used by single executor
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
${jobTracker}
${nameNode}
yarn-cluster
cluster
GenerateEntities
eu.dnetlib.dhp.migration.step2.GenerateEntitiesApplication
dhp-aggregation-${projectVersion}.jar
--executor-memory ${sparkExecutorMemory} --executor-cores ${sparkExecutorCores} --driver-memory=${sparkDriverMemory} --conf spark.extraListeners="com.cloudera.spark.lineage.NavigatorAppListener" --conf spark.sql.queryExecutionListeners="com.cloudera.spark.lineage.NavigatorQueryListener" --conf spark.sql.warehouse.dir="/user/hive/warehouse"
-mt yarn-cluster
-s${migrationPathStep1}/db_records,${migrationPathStep1}/oaf_records,${migrationPathStep1}/odf_records
-t${migrationPathStep2}/all_entities
--islookup${isLookupUrl}