Skip to content

Instantly share code, notes, and snippets.

@kretes

kretes/csv

Created Oct 11, 2019
Embed
What would you like to do?
Repro for spark csv escape issue
a,b
at_the_end\,1
in_\_side,1
"comma,at_the_end\\",1
"comma,in_\\_side",1
df = spark.createDataFrame([{'a':"at_the_end\\", "b":1},
{'a':"in_\\_side", "b":1},
{'a':"comma,at_the_end\\", "b":1},
{'a':"comma,in_\\_side", "b":1}
])
path = "/tmp/spark-quote9"
df.coalesce(1).write.mode("overwrite").csv(path, header=True, escape="\\", quote='"')
! hdfs dfs -text {path}/* > /tmp/csv
pd.read_csv("/tmp/csv", escapechar="\\")
@07ARB

This comment has been minimized.

Copy link

@07ARB 07ARB commented Oct 11, 2019

hi Tomasz Bartczak,
This is my first jira, which i am going to check. please guide me to check the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment