Skip to content

Instantly share code, notes, and snippets.

@raisinbl

raisinbl/test_spark.py

Created May 3, 2023 05:04
Show Gist options
  • Save raisinbl/c21d22b036829b4ccde8fe3e0efca164 to your computer and use it in GitHub Desktop.
Save raisinbl/c21d22b036829b4ccde8fe3e0efca164 to your computer and use it in GitHub Desktop.
Download ZIP
testing if pyspark lib is installed correctly and spark-submit is work
Raw
test_spark.py
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
data = [("Alice", 1), ("Bob", 2), ("Charlie", 3)]
df = spark.createDataFrame(data, ["Name", "Age"])
df.show()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment