Skip to content

Instantly share code, notes, and snippets.

@rustyrazorblade
Last active August 29, 2015 14:27
Show Gist options
  • Save rustyrazorblade/9a38a9499a7531eefd1e to your computer and use it in GitHub Desktop.
Save rustyrazorblade/9a38a9499a7531eefd1e to your computer and use it in GitHub Desktop.
failing migration
####### SPARK JOB #############
from pyspark import SparkContext
from pyspark.sql import SQLContext
sc = SparkContext()
sql = SQLContext(sc)
mysql_movies = sql.read.jdbc("jdbc:mysql://127.0.0.1:3307/movielens?user=root", "movielens.movies")
print mysql_movies.head()
tmp = mysql_movies.write.format("org.apache.spark.sql.cassandra").options(table="movie", keyspace="lens").save()
####### SUBMIT SCRIPT #############
spark-submit \
--jars /Users/jhaddad/Downloads/mysql-connector-java-5.1.36/mysql-connector-java-5.1.36-bin.jar,/Users/jhaddad/dev/cassandra-driver-spark/spark-cassandra-connector/target/scala-2.11/spark-cassandra-connector-assembly-1.5.0-M1-SNAPSHOT.jar \
--driver-class-path /Users/jhaddad/Downloads/mysql-connector-java-5.1.36/mysql-connector-java-5.1.36-bin.jar,/Users/jhaddad/dev/cassandra-driver-spark/spark-cassandra-connector/target/scala-2.11/spark-cassandra-connector-assembly-1.5.0-M1-SNAPSHOT.jar \
test.py
#################### OUTPUT ####################
Py4JJavaError: An error occurred while calling o32.save.
: java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.sql.cassandra.DefaultSource$.<init>(DefaultSource.scala:138)
at org.apache.spark.sql.cassandra.DefaultSource$.<clinit>(DefaultSource.scala)
at org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:83)
at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:309)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:144)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)
15/08/09 11:20:58 INFO SparkContext: Invoking stop() from shutdown hook
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment