Skip to content

Instantly share code, notes, and snippets.

@eddyxu
Last active January 27, 2022 03:38
Show Gist options
  • Save eddyxu/e72e2f90c899431b0b4865649c3d7183 to your computer and use it in GitHub Desktop.
Save eddyxu/e72e2f90c899431b0b4865649c3d7183 to your computer and use it in GitHub Desktop.
Create spark with rikai extension
spark = (
SparkSession
.builder
.config("spark.jars.packages", "ai.eto:rikai_2.12:0.1.0")
.config(
"spark.sql.extensions",
"ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions",
)
.config(
"spark.rikai.sql.ml.registry.mlflow.tracking_uri",
MLFLOW_TRACKING_URI,
)
.master("local[*]")
.getOrCreate()
);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment