Skip to content

Instantly share code, notes, and snippets.

View jeesim2's full-sized avatar

Jihun No jeesim2

  • Naver
  • Seoul, Korea
View GitHub Profile
@jeesim2
jeesim2 / spark-submit-streaming-yarn.sh
Created August 23, 2017 00:43 — forked from bernhardschaefer/spark-submit-streaming-yarn.sh
spark-submit template for running Spark Streaming on YARN
#!/bin/bash
# Minimum TODOs on a per job basis:
# 1. define name, application jar path, main class, queue and log4j-yarn.properties path
# 2. remove properties not applicable to your Spark version (Spark 1.x vs. Spark 2.x)
# 3. tweak num_executors, executor_memory (+ overhead), and backpressure settings
# the two most important settings:
num_executors=6
executor_memory=3g