Skip to content

Instantly share code, notes, and snippets.

@seahrh
Created February 8, 2019 06:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save seahrh/fb4b390d98ab28391694c5bc1ac5521f to your computer and use it in GitHub Desktop.
Save seahrh/fb4b390d98ab28391694c5bc1ac5521f to your computer and use it in GitHub Desktop.
spark-submit airflow jinja template - optional params
#!/usr/bin/env bash
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--class {{ params.class }} {{ params.jar_path }} \
--sink_db {{ params.sink_db }} \
--sink_table {{ params.sink_table }} \
--sink_partition_column_ds {{ ds_nodash }} \
{% if params.sink_partition_column_post_date is defined %}--sink_partition_column_post_date {{ params.sink_partition_column_post_date }} \{% else %}\{% endif %}
{% if params.foo is defined %}--foo {{ params.foo }} \{% else %}\{% endif %}
{% if params.bar is defined %}--bar {{ params.bar }} \{% else %}\{% endif %}
--sink_db_path s3://{{ params.bucket }}/hive/warehouse/{{ params.sink_db }}.db
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment