example for "Prediction at Scale with scikit-learn and PySpark Pandas UDFs" (https://medium.com/civis-analytics/prediction-at-scale-with-scikit-learn-and-pyspark-pandas-udfs-51d5ebfb2cd8)

pyspark_pandas_udf_sklearn.ipynb

Hi, im getting some kind of pyspark error on your code. I guess it's because of libs incompatibility.
Could you pls provide python, pandas and pyspark versions, which worked for u, thx

Hi, sorry about not including version numbers in there. I added them just now.

I suspect the pandas or pyarrow version was causing trouble because I had to use some older versions of those to get this notebook to run just now. I didn't look into it deeply, but it looks like the latest version of pyspark at the moment (2.4.4) doesn't work well with the latest versions of pandas and pyarrow. This apparently has been addressed for future releases (see, e.g., apache/spark#24867).

thank you so much for your quick response, it worked for me as i downgraded pyarrow to 0.8.0

Hi - this code is helpful for applying an already trained model at scale... but is it possible to train a model at scale using pandas_UDF functions?

Another way of asking the same question: Is it possible to include the section called "Train a model with scikit-learn" within a pandas_UDF?