Skip to content

Instantly share code, notes, and snippets.

@kevincolten

kevincolten/sparkMySQL.scala

Created June 7, 2015 20:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kevincolten/9c330492ad280f6f4f29 to your computer and use it in GitHub Desktop.
Save kevincolten/9c330492ad280f6f4f29 to your computer and use it in GitHub Desktop.
Download ZIP
Import MySQL into Spark
Raw
sparkMySQL.scala
SPARK_CLASSPATH=../mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar bin/spark-shell
val jdbcDF = sqlContext.load("jdbc", Map(
"url" -> "jdbc:mysql://localhost:8889/employees?user=root&password=root",
"dbtable" -> "(SELECT employees.*, departments.*, salaries.salary, titles.title FROM employees JOIN dept_emp ON dept_emp.emp_no = employees.emp_no JOIN departments ON departments.dept_no = dept_emp.dept_no JOIN dept_manager ON dept_manager.emp_no = employees.emp_no JOIN salaries ON salaries.emp_no = employees.emp_no JOIN titles ON employees.emp_no = titles.emp_no) AS employees",
"driver" -> "com.mysql.jdbc.Driver",
"partitionColumn" -> "emp_no",
"lowerBound" -> "10001",
"upperBound" -> "499999",
"numPartitions" -> "10"
))
jdbcDF.select("*").show()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment