Skip to content

Instantly share code, notes, and snippets.

@revolutionisme
Created April 12, 2020 16:54
Show Gist options
  • Save revolutionisme/9c72c7da4e08ad3aaeb81bec5e239306 to your computer and use it in GitHub Desktop.
Save revolutionisme/9c72c7da4e08ad3aaeb81bec5e239306 to your computer and use it in GitHub Desktop.
Split data from one column to multiple columns using pyspark and remove first row which contains the headers
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local").appName('testapp').getOrCreate()
df = spark.createDataFrame([("col1:col2:col3",),
("1:a:2001",),
("2:b:2002",),
("3:c:2003",)],
["value"])
df.show()
df.createOrReplaceTempView("dftable")
df_split = spark.sql("select split(value,':') as column1 from dftable")
header = df_split.first()['column1']
df_split.show()
df_split = df_split.rdd.flatMap(lambda x: x).toDF(schema=header)
df_split = df_split.filter("col1 not like '%col1%'")
df_split.show()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment