Skip to content

Instantly share code, notes, and snippets.

@ijan10

ijan10/pandas_udf_spark_submit.sh Secret

Last active October 20, 2019 10:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ijan10/230ae52a98f1c2f0f56afe1fe534ee1e to your computer and use it in GitHub Desktop.
Save ijan10/230ae52a98f1c2f0f56afe1fe534ee1e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
pandas_udf_spark_submit.sh
spark-submit --conf spark.driver.maxResultSize=50g --executor-memory 70g --conf spark.executor.memoryOverhead=40g --conf spark.driver.memory=70g --executor-cores 14 --conf spark.yarn.am.memory=7g --conf spark.executor.extraJavaOptions=-XX:+UseG1GC --conf spark.yarn.am.cores=4 --master yarn --name create_ctr_model.py --verbose create_ctr_model.py --env=production --model_selected_feature_name=placement_type_name --running_date=2019_10_20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment