pivot_history_db
Pivot history DB on hive
new_graph_db
New graph DB on hive
new_graph_date
Creation date of new graph db
hiveMetastoreUris
hive server metastore URIs
sparkSqlWarehouseDir
sparkClusterOpts
--conf spark.network.timeout=600 --conf spark.extraListeners= --conf spark.sql.queryExecutionListeners= --conf spark.yarn.historyServer.address=http://iis-cdh5-test-m3.ocean.icm.edu.pl:18088 --conf spark.eventLog.dir=hdfs://nameservice1/user/spark/applicationHistory
spark cluster-wide options
sparkResourceOpts
--executor-memory=3G --conf spark.executor.memoryOverhead=3G --executor-cores=6 --driver-memory=8G --driver-cores=4
spark resource options
sparkApplicationOpts
--conf spark.sql.shuffle.partitions=3840
spark resource options
${jobTracker}
${nameNode}
mapreduce.job.queuename
${queueName}
oozie.launcher.mapred.job.queue.name
${oozieLauncherQueueName}
oozie.action.sharelib.for.spark
${oozieActionShareLibForSpark2}
Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
yarn
cluster
Upgrade Pivot History
eu.dnetlib.dhp.oozie.RunSQLSparkJob
dhp-dedup-openaire-${projectVersion}.jar
--conf spark.sql.warehouse.dir=${sparkSqlWarehouseDir}
${sparkClusterOpts}
${sparkResourceOpts}
${sparkApplicationOpts}
--hiveMetastoreUris${hiveMetastoreUris}
--sqleu/dnetlib/dhp/oa/dedup/pivothistory/oozie_app/sql.sql
--pivot_history_db${pivot_history_db}
--new_graph_db${new_graph_db}
--new_graph_date${new_graph_date}