Skip to content

Instantly share code, notes, and snippets.

@jfarcand
Created June 12, 2015 18:40
Show Gist options
  • Save jfarcand/245867c7cceaed0779e4 to your computer and use it in GitHub Desktop.
Save jfarcand/245867c7cceaed0779e4 to your computer and use it in GitHub Desktop.
/**
* Exception in thread "main" scala.MatchError: StringType (of class org.apache.spark.sql.types.StringType$)
at org.apache.spark.sql.json.InferSchema$.apply(InferSchema.scala:58)
at org.apache.spark.sql.json.JSONRelation$$anonfun$schema$1.apply(JSONRelation.scala:139)
at org.apache.spark.sql.json.JSONRelation$$anonfun$schema$1.apply(JSONRelation.scala:138)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.json.JSONRelation.schema$lzycompute(JSONRelation.scala:137)
at org.apache.spark.sql.json.JSONRelation.schema(JSONRelation.scala:137)
at org.apache.spark.sql.sources.LogicalRelation.<init>(LogicalRelation.scala:30)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:120)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104)
at org.apache.spark.sql.SQLContext.load(SQLContext.scala:1218)
at com.yulplay.yuly.db.SparkDiver.main(SparkDiver.java:52)
*/
//DataFrame frame = sqlContext.load(ROOT + g.getId() + ".json", "json");
PlayByPlay playByPlay = mapper.readValue(Files.readAllBytes(Paths.get(ROOT + g.getId() + ".json")), PlayByPlay.class);
List<String> jsonData = Arrays.asList(mapper.writeValueAsString(playByPlay));
JavaRDD<String> games = sc.parallelize(jsonData);
DataFrame frame = sqlContext.read().json(games);
frame.show();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment