Last active
July 8, 2022 12:53
-
-
Save mheilman/6ce261549b55bf4997ec102ad4e8d643 to your computer and use it in GitHub Desktop.
example for "Prediction at Scale with scikit-learn and PySpark Pandas UDFs" (https://medium.com/civis-analytics/prediction-at-scale-with-scikit-learn-and-pyspark-pandas-udfs-51d5ebfb2cd8)
thank you so much for your quick response, it worked for me as i downgraded pyarrow to 0.8.0
Hi - this code is helpful for applying an already trained model at scale... but is it possible to train a model at scale using pandas_UDF functions?
Another way of asking the same question: Is it possible to include the section called "Train a model with scikit-learn" within a pandas_UDF?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi, sorry about not including version numbers in there. I added them just now.
I suspect the pandas or pyarrow version was causing trouble because I had to use some older versions of those to get this notebook to run just now. I didn't look into it deeply, but it looks like the latest version of pyspark at the moment (2.4.4) doesn't work well with the latest versions of pandas and pyarrow. This apparently has been addressed for future releases (see, e.g., apache/spark#24867).