Skip to content

Instantly share code, notes, and snippets.

View Redwa's full-sized avatar

Redwa

4 followers · 25 following
View GitHub Profile
@Redwa
Redwa / scala spark action examples
Created December 25, 2015 22:57 — forked from tmcgrath/scala spark action examples
Spark Console Action functions in Scala
scala> val names1 = sc.parallelize(List("abe", "abby", "apple"))
names1: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at parallelize at <console>:12
scala> names1.reduce((t1,t2) => t1 + t2)
res0: String = abbyappleabe
scala> names1.flatMap(k => List(k.size) ).reduce((t1,t2) => t1 + t2)
res1: Int = 12
scala> val names2 = sc.parallelize(List("apple", "beatty", "beatrice")).map(a => (a, a.size))
@Redwa
Redwa / Code from Part 1 of Spark Transformations in Scala
Created December 25, 2015 22:58 — forked from tmcgrath/Code from Part 1 of Spark Transformations in Scala
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
@Redwa
Redwa / Scala based Spark Transformations Part 2 - Comparing and Contrasting RDDs
Created December 25, 2015 22:58 — forked from tmcgrath/Scala based Spark Transformations Part 2 - Comparing and Contrasting RDDs
Scala based Spark Transformations Part 2
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
@Redwa
Redwa / Spark Transformation Examples Part 3
Created December 25, 2015 22:58 — forked from tmcgrath/Spark Transformation Examples Part 3
Scala based Spark Transformations which require Key, Value pair RDDs
scala> val babyNames = sc.textFile("baby_names.csv")
babyNames: org.apache.spark.rdd.RDD[String] = baby_names.csv MappedRDD[27] at textFile at <console>:12
scala> val rows = babyNames.map(line => line.split(","))
rows: org.apache.spark.rdd.RDD[Array[String]] = MappedRDD[28] at map at <console>:14
scala> val namesToCounties = rows.map(name => (name(1),name(2)))
namesToCounties: org.apache.spark.rdd.RDD[(String, String)] = MappedRDD[29] at map at <console>:16
scala> namesToCounties.groupByKey.collect
@Redwa
Redwa / useful_pandas_snippets.py
Created November 30, 2017 15:41 — forked from bsweger/useful_pandas_snippets.md
Useful Pandas Snippets
# List unique values in a DataFrame column
# h/t @makmanalp for the updated syntax!
df['Column Name'].unique()
# Convert Series datatype to numeric (will error if column has non-numeric values)
# h/t @makmanalp
pd.to_numeric(df['Column Name'])
# Convert Series datatype to numeric, changing non-numeric values to NaN
# h/t @makmanalp for the updated syntax!
@Redwa
Redwa / DashMLWebinar.md
Created August 29, 2018 18:21 — forked from xhluca/DashMLWebinar.md