Skip to content

Instantly share code, notes, and snippets.

View EmanuelFontelles's full-sized avatar

Emanuel Fontelles EmanuelFontelles

View GitHub Profile
@smoothml
smoothml / scikit-learn-predictions-on-spark.py
Last active October 18, 2023 19:07
How to apply a Scikit Learn machine learning model at scale using Apache Spark.
from pyspark.sql import functions as F
from pyspark.sql.types import DoubleType
import pandas as pd
from sklearn.externals import joblib
def make_predictions(sc, df, feature_cols, model_path):
"""
Make predictions.