Skip to content

Instantly share code, notes, and snippets.

@akhld
Created December 9, 2014 15:54
Show Gist options
  • Save akhld/ccafb27f098163bea622 to your computer and use it in GitHub Desktop.
Save akhld/ccafb27f098163bea622 to your computer and use it in GitHub Desktop.
Scala JdbcRDD MySQL Connector
val rdd = new org.apache.spark.rdd.JdbcRDD(
ssc.sparkContext,
() => {
Class.forName("org.mysql.Driver")
DriverManager.getConnection("jdbc:mysql://localhost:3306/sigmoid"), "akhld", "pass")
},
"SELECT * FROM logs WHERE ? <= id AND id <= ?",
0, 1000, 10,
row => {
var new_map: Map[String, Any] = Map()
for(key <- tmp_schema){
try{
new_map += key._1 -> row.getString(key._1)
}catch{ case e: Exception =>
println("Exception!! => " + e)
new_map += key._1 -> ""
}
}
new_map.toList
}
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment