Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save dqtweb/f637951697bf2ddfcafba5d49c0092c4 to your computer and use it in GitHub Desktop.
Save dqtweb/f637951697bf2ddfcafba5d49c0092c4 to your computer and use it in GitHub Desktop.
Export/import a PySpark schema to/from a JSON file
import json
from pyspark.sql.types import *
# Define the schema
schema = StructType(
[StructField("name", StringType(), True), StructField("age", IntegerType(), True)]
)
# Write the schema
with open("schema.json", "w") as f:
json.dump(schema.jsonValue(), f)
# Read the schema
with open("schema.json") as f:
new_schema = StructType.fromJson(json.load(f))
print(new_schema.simpleString())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment