kadaliao/pyspark_dataframe_deep_copy.py

Forked from christinebuckler/pyspark_dataframe_deep_copy.py

Created February 20, 2023 07:21

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/kadaliao/18fe923529a3b329b197259e83f92a44.js"></script>
Save kadaliao/18fe923529a3b329b197259e83f92a44 to your computer and use it in GitHub Desktop.

Download ZIP

PySpark deep copy dataframe

Raw

pyspark_dataframe_deep_copy.py

	import copy

	X = spark.createDataFrame([[1,2], [3,4]], ['a', 'b'])
	_schema = copy.deepcopy(X.schema)
	_X = X.rdd.zipWithIndex().toDF(_schema)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment