Skip to content

Instantly share code, notes, and snippets.

@Arvind282
Forked from NitinKumar94/oozie spark workflow
Created July 14, 2016 11:57
Show Gist options
  • Save Arvind282/82199ff8abf2ffdd63bd13ce511dd357 to your computer and use it in GitHub Desktop.
Save Arvind282/82199ff8abf2ffdd63bd13ce511dd357 to your computer and use it in GitHub Desktop.
<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns='uri:oozie:workflow:0.4' name='sparkjob'>
<start to='spark-process' />
<action name='spark-process'>
<spark xmlns='uri:oozie:spark-action:0.1'>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.service.SparkConfigurationService.spark.configurations</name>
<value>spark.eventLog.dir=hdfs://node1.analytics.sub:8020/user/spark/applicationHistory,spark.yarn.historyServer.address=http://node1.analytics.sub:18088,spark.eventLog.enabled=true</value>
</property>
<!--property>
<name>oozie.hive.defaults</name>
<value>/user/ambari-qa/sparkActionPython/hive-config.xml</value>
</property-->
<!--property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property-->
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/oozie/share/lib/lib_20150831190253/spark</value>
</property>
</configuration>
<master>yarn-client</master>
<!--master>local[4]</master-->
<mode>client</mode>
<name>wordcount</name>
<jar>/usr/hdp/current/spark-client/AnalyticsJar/wordcount.py</jar>
<spark-opts>--executor-memory 1G --driver-memory 1G --executor-cores 4 --num-executors 2 --jars /usr/hdp/current/spark-client/lib/spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar</spark-opts>
</spark>
<ok to='end'/>
<error to='spark-fail'/>
</action>
<kill name='spark-fail'>
<message>Spark job failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name='end' />
</workflow-app>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment