Skip to content

Instantly share code, notes, and snippets.

@eddies
Created February 19, 2016 08:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save eddies/f20241e161aa2bbbb788 to your computer and use it in GitHub Desktop.
Save eddies/f20241e161aa2bbbb788 to your computer and use it in GitHub Desktop.
import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset
// Zeppelin creates and injects sc (SparkContext) and sqlContext (HiveContext or SqlContext)
// So you don't need create them manually
// load map data
val myMapText = sc.parallelize(
IOUtils.toString(
new URL("https://gist.githubusercontent.com/Madhuka/74cb9a6577c87aa7d2fd/raw/2f758d33d28ddc01c162293ad45dc16be2806a6b/data.csv"),
Charset.forName("utf8")).split("\n"))
case class Map(Country:String, Name:String, lat : Float, lan : Float, Altitude : Float)
val myMap = myMapText.map(s=>s.split(",")).filter(s=>s(0)!="Country").map(
s=>Map(s(0),
s(1),
s(2).toFloat,
s(3).toFloat,
s(4).toFloat
)
)
// Below line works only in spark 1.3.0.
// For spark 1.1.x and spark 1.2.x,\n// use myMap.registerTempTable(\"myMap\") instead.\n
myMap.toDF().registerTempTable("myMap")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment