Skip to content

Instantly share code, notes, and snippets.

@kaysush

kaysush/spark_gcs_demo.py

Created October 7, 2021 05:05
Show Gist options
  • Save kaysush/65fdd9a5d5bb03a198d8fb1e23125bf1 to your computer and use it in GitHub Desktop.
Save kaysush/65fdd9a5d5bb03a198d8fb1e23125bf1 to your computer and use it in GitHub Desktop.
Download ZIP
Sample PySpark Job
Raw
spark_gcs_demo.py
from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
print('Storing random numbers in a GCS bucket')
spark.range(100).write.mode("overwrite").parquet("gs://apt-task-314904/random")
print('complete')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment