Skip to content

Instantly share code, notes, and snippets.

@RuneLind
Last active June 4, 2020 15:04
Show Gist options
  • Save RuneLind/9448367b77edf412077502103891c976 to your computer and use it in GitHub Desktop.
Save RuneLind/9448367b77edf412077502103891c976 to your computer and use it in GitHub Desktop.
from pyspark.sql.types import *
fødsel_type = StructType([
StructField('personidentifikator',StringType()),
StructField('foedselsaar', StringType())])
fødsel_data = [
('12071950857',2019),
('10052300393',1923),
]
fødsel = spark.createDataFrame(fødsel_data, fødsel_type)
fødsel.show()
fødsel.printDocTemplate(True)
fødsel.doc = {
"logical-record-root" : {
"name" : "spark_schema",
"instanceVariables" : [ {
"name" : "personidentifikator",
"description" : "personidentifikator"
}, {
"name" : "foedselsaar",
"description" : "foedselsaar"
} ]
}
}
fødsel.write\
.option("valuation", "INTERNAL")\
.option("state", "INPUT")\
.path("/user/rune.lind/test/f1")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment