kittipatkampa/convert_to_rdd_then_df.py

Created July 31, 2019 21:29

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/kittipatkampa/774a18b1f0717fcc56b231125959bf00.js"></script>
Save kittipatkampa/774a18b1f0717fcc56b231125959bf00 to your computer and use it in GitHub Desktop.

Download ZIP

Converting pyspark dataframe into RDD and back to DataFrame can resolve StackOverflow Error.

Raw

convert_to_rdd_then_df.py

	train_df = spark.createDataFrame(train_df.rdd, schema=train_df.schema)
	model_pred = pipeline_pred.fit(train_df)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment