Skip to content

Instantly share code, notes, and snippets.

@squito

squito/gist:ccd56fefefe4dfef808dc21196a89385

Created August 28, 2017 18:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save squito/ccd56fefefe4dfef808dc21196a89385 to your computer and use it in GitHub Desktop.
Save squito/ccd56fefefe4dfef808dc21196a89385 to your computer and use it in GitHub Desktop.
Download ZIP
random example of exploring spark internals w/ reflection while debugging cluster config
Raw
gistfile1.txt
// paste in
// https://gist.githubusercontent.com/squito/329d9cd82a21f645d592/raw/ca9217708293d6fe69ed6638f4feeb3038f8fd9c/reflector.scala
val xCat = spark.sessionState.catalog.externalCatalog
val catClient = get(xCat, "client")
catClient.reflectMethod("getConf", Seq("hive.metastore.uris", ""))
import org.apache.hadoop.fs.{FileSystem, Path}
val fs = FileSystem.get(sc.hadoopConfiguration)
val defaultDb = catClient.reflectMethod("getDatabase", Seq("default"))
val dbLoc = defaultDb.reflectField("locationUri").asInstanceOf[java.net.URI]
val qDbLoc = xCat.reflectMethod("org$apache$spark$sql$hive$HiveExternalCatalog$$makeQualified", Seq(dbLoc)).asInstanceOf[java.net.URI]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment