Skip to content

Instantly share code, notes, and snippets.

View existeundelta's full-sized avatar

ExisteUnDelta existeundelta

  • Barcelona
View GitHub Profile
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@existeundelta
existeundelta / pspark_config.py
Created February 28, 2018 08:47 — forked from robenalt/pspark_config.py
Sample pyspark context setting with configs params
# Set up spark configuration
conf = SparkConf().setMaster("yarn-client").setAppName("sparK-mer")
#conf = SparkConf().setMaster("local[16]").setAppName("sparK-mer")
conf.set("yarn.nodemanager.resource.cpu_vcores",args.C)
# Saturate with executors
conf.set("spark.executor.instances",executorInstances)
conf.set("spark.executor.heartbeatInterval","5s")
# cores per executor
conf.set("spark.executor.cores",args.E)
# set driver cores
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
FROM ubuntu:14.04
ENV SCALA_VERSION=2.10.4
ENV CASSANDRA_VERSION=2.2.3
ENV SPARK_CASSANDRA_CONNECTOR_VERSION=1.4.0
ENV CONFLUENT_VERSION=1.0.1
ENV ELASTICSEARCH_VERSION=1.7.3
ENV ELASTICSEARCH_SPARK_CONNECTOR_VERSION=2.1.2
ENV LOGSTASH_VERSION=2.0.0
ENV KIBANA_VERSION=4.2.0