Skip to content

Instantly share code, notes, and snippets.

@animeshtrivedi
Created March 23, 2018 13:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save animeshtrivedi/aed06c397095d3282a6e5655d1f1873a to your computer and use it in GitHub Desktop.
Save animeshtrivedi/aed06c397095d3282a6e5655d1f1873a to your computer and use it in GitHub Desktop.
Spark configuration for a local mode execution
# Command to launch TPCDS:
# ./bin/spark-submit -v --master local[2] --class com.ibm.crail.spark.tools.ParquetGenerator ~/jars/parquet-generator-1.0.jar -c tpcds -o crail://localhost:9060/F1/tpcds/ -p 4 -t 4 -tsf 1 -tdsd /home/atr/zrl/external/github/databricks/tpcds-kit/tools/ -tdd 1
# And you need to put core-site.xml from crail into the conf folder.
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.
# Example:
# spark.master spark://master:7077
# spark.eventLog.enabled true
# spark.eventLog.dir hdfs://namenode:8021/directory
# spark.serializer org.apache.spark.serializer.KryoSerializer
# spark.driver.memory 5g
# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.driver.memory 16g
spark.executor.memory 16g
spark.executor.extraJavaOptions -Dsun.nio.PageAlignDirectMemory=true -Xmn16G -Dio.netty.leakDetectionLevel=advanced
spark.driver.extraJavaOptions -Dsun.nio.PageAlignDirectMemory=true -Xmn16G -Dio.netty.leakDetectionLevel=advanced
spark.driver.extraClassPath /home/atr/zrl/external/github/patrickstuedi/pocket/assembly/target/crail-1.0-bin/jars/*
spark.executor.extraClassPath /home/atr/zrl/external/github/patrickstuedi/pocket/assembly/target/crail-1.0-bin/jars/*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment