jaceklaskowski

Exercise

Develop a Spark standalone application (using IntelliJ IDEA) with Spark MLlib and LogisticRegression to classify emails.

Think about command line and what parameters you'd like to accept for various use cases.

TIP Use scopt

1. libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.1.1"

jaceklaskowski

IDEA:

• breaks on demand given the number of exercises
• break man who says we should have one

val wholeJsonRDD = sc.wholeTextFiles("input.json").map(_._2)
val mySchema = new StructType().add($"n".int)
wholeJsonRDD.toDF.withColumn("json", from_json($"value", mySchema)).show(truncate = false)

val jsonDF=spark.read.json("output.json")

jaceklaskowski

Exercise 1

Union only those rows (from large table) with keys in left small table, i.e. union two dataframes together but only those with the key in my small table.

Exercise 2

Aggregation on an array of nested json = How to sum the quantities across all lines for a given order (which would give 1 + 3 = 4 for the below sample dataset):

{

jaceklaskowski

Moved the notes to https://github.com/jaceklaskowski/blockchain-notes repository. See you there.

I'm going to remove the gist after Oct, 7th.

jaceklaskowski

Parquet

Introduction

• Stores schema information along with the data
• Columnar storage/file format
  • "reference file format on Hadoop HDFS"
  • "read-optimized view of data"
• excellent for local file storage on HDFS (instead of external databases).
• writing very large datasets to disk

jaceklaskowski

Notes

• Combinators == building blocks
• functions and higher order functions
• composition
• immutable == less moving parts to worry about
• a routine job == a boilerplate == a boring stuff
• a context (so the job varies)
• abstracting away == happening behind the scenes == cutting down repetitive code == eliminating boilerplate
• "The code becomes small, succinct, and more readable"

jaceklaskowski

Spark-a-thon — Development Activities

Spark Structured Streaming

1. Developing custom Source to handle formats like XML
   • Publish the streaming source / format to https://spark-packages.org/
   • As a follow-up answer How to read streaming data in XML format from Kafka?
2. Explore what's stored in checkpointLocation
3. How is streaming groupBy different with Append and Complete output modes?
4. Answering the question on StackOverflow — How to count items per time window?

jaceklaskowski

DC/OS

1. The latest documentation at https://dcos.io/docs/latest.

Mesosphere DC/OS 1.9

1. Announcing Mesosphere DC/OS 1.9
2. DC/OS 1.9.0 Release Candidate 2
3. Bringing Production-Proven Data Services to DC/OS 1.9 with our Partners

• DC/OS provides one-click installation of data services such as databases, message queues, and analytics engines, on-par with cloud providers such as Amazon Web Services.

jaceklaskowski

scala> Seq(A(4)).toDS
16/08/16 19:26:26 ERROR RetryingHMSHandler: AlreadyExistsException(message:Database default already exists)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
	at com.sun.proxy.$Proxy14.create_database(Unknown Source)

jaceklaskowski

1. Create a new Scala/sbt project in IntelliJ IDEA

• Project name: handleAllocatedContainers

2. Create a test (ScalaTest) for our exercise

• Hint: See https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L448-L470
• Input: sequence of elements
• 3 predicates to produce 4 sets <-- think about any number of predicates
• The test will fail!!!

3. Exercise is to make the test pass