Skip to content

Instantly share code, notes, and snippets.

@prodeezy
Last active April 2, 2019 05:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save prodeezy/cd8257899e90a52dfa37d5c58b3cc08f to your computer and use it in GitHub Desktop.
Save prodeezy/cd8257899e90a52dfa37d5c58b3cc08f to your computer and use it in GitHub Desktop.
Test for loading Iceberg table using relative path
bash-3.2$ ls -d test/iceberg-people
test/iceberg-people
spark-shell --jars runtime/build/libs/iceberg-runtime.jar
import org.apache.spark.sql.types._ ;
import org.apache.iceberg.hadoop.HadoopTables;
import org.apache.iceberg.Schema;
import org.apache.iceberg.spark.SparkSchemaUtil
val schema = new StructType().add("age", IntegerType).add("name", StringType).add("friends", MapType(StringType, IntegerType)).add("location", new StructType().add("lat", DoubleType).add("lon", DoubleType))
// load any pre-generated Iceberg table using a relative path
val iceDf = spark.read.format("iceberg").load("test/iceberg-people")
iceDf.createOrReplaceTempView("iceberg_people")
// fails here
spark.sql("select * from iceberg_people where age = 30").show()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment