SQL Server (Azure SQL Database) JDBC driver installation for Apache Spark on OSX or Linux

spark-sqlserver-jdbc.md

1. Download & unpack driver from https://www.microsoft.com/en-us/download/details.aspx?id=57175
2. Find jar file inside: sqljdbc_{version}/enu/jre{version}/sqljdbc{version}.jar

There are few ways to use it.

• Update Spark config to include this path (always included):

  • mv {SPARK_HOME}/conf/spark-defaults.conf.template {SPARK_HOME}/conf/spark-defaults.conf
  • Add line spark.driver.extraClassPath /path/to/sqljdbc.jar to spark-defaults.conf

• Or use it in SparkSession (per session):

  spark = SparkSession \
      .builder \
      .appName("app") \
      .config("spark.driver.extraClassPath", "/path/to/sqljdbc.jar") \
      .getOrCreate()
  

df = spark.read.jdbc(url=conn_string, table="schema.table")