Tested with Apache Spark 2.1.0, Python 2.7.13 and Java 1.8.0_112
For older versions of Spark and ipython, please, see also previous version of text.
# Set up spark configuration | |
conf = SparkConf().setMaster("yarn-client").setAppName("sparK-mer") | |
#conf = SparkConf().setMaster("local[16]").setAppName("sparK-mer") | |
conf.set("yarn.nodemanager.resource.cpu_vcores",args.C) | |
# Saturate with executors | |
conf.set("spark.executor.instances",executorInstances) | |
conf.set("spark.executor.heartbeatInterval","5s") | |
# cores per executor | |
conf.set("spark.executor.cores",args.E) | |
# set driver cores |
Tested with Apache Spark 2.1.0, Python 2.7.13 and Java 1.8.0_112
For older versions of Spark and ipython, please, see also previous version of text.