Skip to content

Instantly share code, notes, and snippets.

@nfarah86

nfarah86/spark_session.py

Created October 21, 2021 23:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nfarah86/1aa679c02b74267a4821b145c2bed195 to your computer and use it in GitHub Desktop.
Save nfarah86/1aa679c02b74267a4821b145c2bed195 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark_session.py
def create_spark_session():
spark = SparkSession \
.builder \
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0") \
.config("spark.hadoop.fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem") \
.getOrCreate()
sc=spark.sparkContext
return spark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment