Skip to content

Instantly share code, notes, and snippets.

@afranzi

afranzi/pyspark.sh

Created May 21, 2019 11:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save afranzi/499c2a4cb389e9247f6d219549994e30 to your computer and use it in GitHub Desktop.
Save afranzi/499c2a4cb389e9247f6d219549994e30 to your computer and use it in GitHub Desktop.
Download ZIP
PySpark with org.apache.hadoop:hadoop-aws:2.8.5
Raw
pyspark.sh
export SPARK_OPTS='--packages org.apache.hadoop:hadoop-aws:2.8.5 --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider'
pyspark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment