Skip to content

Instantly share code, notes, and snippets.

@thomas-maschler

thomas-maschler/my_pyspark.py

Last active November 16, 2016 16:13
Show Gist options
  • Save thomas-maschler/5771c9574cb2f0aabc039df6ffcef3c5 to your computer and use it in GitHub Desktop.
Save thomas-maschler/5771c9574cb2f0aabc039df6ffcef3c5 to your computer and use it in GitHub Desktop.
Download ZIP
Basic pyspark program
Raw
my_pyspark.py
from pyspark.sql import SparkSession
spark = SparkSession\
.builder\
.appName("MyPySpark")\
.getOrCreate()
sc = spark.sparkContext
my_file = sc.textFile("s3n://path/to/file")
print my_file.collect()
spark.stop()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment