Skip to content

Instantly share code, notes, and snippets.

Created Oct 11, 2019
What would you like to do?
Repro for spark csv escape issue
df = spark.createDataFrame([{'a':"at_the_end\\", "b":1},
{'a':"in_\\_side", "b":1},
{'a':"comma,at_the_end\\", "b":1},
{'a':"comma,in_\\_side", "b":1}
path = "/tmp/spark-quote9"
df.coalesce(1).write.mode("overwrite").csv(path, header=True, escape="\\", quote='"')
! hdfs dfs -text {path}/* > /tmp/csv
pd.read_csv("/tmp/csv", escapechar="\\")
Copy link

07ARB commented Oct 11, 2019

hi Tomasz Bartczak,
This is my first jira, which i am going to check. please guide me to check the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment