migrationClaimsPathStep1
the base path to store hdfs file
migrationClaimsPathStep2
the temporary path to store entities before dispatching
migrationClaimsPathStep3
the graph Raw base path
postgresURL
the postgres URL to access to the database
postgresUser
the user postgres
postgresPassword
the password postgres
mongoURL
mongoDB url, example: mongodb://[username:password@]host[:port]
mongoDb
mongo database
sparkDriverMemory
memory for driver process
sparkExecutorMemory
memory for individual executor
sparkExecutorCores
number of cores used by single executor
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
${jobTracker}
${nameNode}
eu.dnetlib.dhp.migration.step1.MigrateDbEntitiesApplication
-p${migrationClaimsPathStep1}/db_claims
-pgurl${postgresURL}
-pguser${postgresUser}
-pgpasswd${postgresPassword}
-aclaims
${jobTracker}
${nameNode}
eu.dnetlib.dhp.migration.step1.MigrateMongoMdstoresApplication
-p${migrationClaimsPathStep1}/odf_claims
-mongourl${mongoURL}
-mongodb${mongoDb}
-fODF
-lstore
-iclaim
${jobTracker}
${nameNode}
eu.dnetlib.dhp.migration.step1.MigrateMongoMdstoresApplication
-p${migrationClaimsPathStep1}/oaf_claims
-mongourl${mongoURL}
-mongodb${mongoDb}
-fOAF
-lstore
-iclaim
${jobTracker}
${nameNode}
yarn-cluster
cluster
GenerateClaimEntities
eu.dnetlib.dhp.migration.step2.GenerateEntitiesApplication
dhp-aggregation-${projectVersion}.jar
--executor-memory ${sparkExecutorMemory} --executor-cores ${sparkExecutorCores} --driver-memory=${sparkDriverMemory} --conf spark.extraListeners="com.cloudera.spark.lineage.NavigatorAppListener" --conf spark.sql.queryExecutionListeners="com.cloudera.spark.lineage.NavigatorQueryListener" --conf spark.sql.warehouse.dir="/user/hive/warehouse"
-mt yarn-cluster
-s${migrationClaimsPathStep1}/db_claims,${migrationClaimsPathStep1}/oaf_claims,${migrationClaimsPathStep1}/odf_claims
-t${migrationClaimsPathStep2}/claim_entities
-pgurl${postgresURL}
-pguser${postgresUser}
-pgpasswd${postgresPassword}
${jobTracker}
${nameNode}
yarn-cluster
cluster
GenerateClaimGraph
eu.dnetlib.dhp.migration.step3.DispatchEntitiesApplication
dhp-aggregation-${projectVersion}.jar
--executor-memory ${sparkExecutorMemory} --executor-cores ${sparkExecutorCores} --driver-memory=${sparkDriverMemory} --conf spark.extraListeners="com.cloudera.spark.lineage.NavigatorAppListener" --conf spark.sql.queryExecutionListeners="com.cloudera.spark.lineage.NavigatorQueryListener" --conf spark.sql.warehouse.dir="/user/hive/warehouse"
-mt yarn-cluster
-s${migrationClaimsPathStep2}/claim_entities
-g${migrationClaimsPathStep3}