Sparkathon in Warsaw - Development Activities

sparkathon-agenda.md

Spark-a-thon - Development Activities

Tuesday, September 12, 2017

Sparkathon - Developing Spark Structured Streaming Apps in Scala

Spark SQL

1. Multiple groupBy or groupByKey aggregations in a batch structured query

Spark Structured Streaming

1. Developing custom Sink (using StreamSinkProvider)
2. Answering the question from StackOverflow — How to count items per time window?
3. Understanding OutputMode
   • flatMapGroupsWithState operator
4. Using flatMapGroupsWithState operator to mimic the output modes: Complete, Append and Update
5. Multiple flatMapGroupsWithState in a streaming query
6. Multiple groupBy or groupByKey aggregations in a streaming structured query
7. Streaming aggregation with Append output mode requires watermark

Aug 22nd

1. Using flatMapGroupsWithState operator
2. Developing custom StreamSinkProvider (with particular focus on OutputMode)

Apr 26th

1. Creating custom Encoder
   • for java.time.LocalDateTime
2. Extend Dataset API to support GROUPING SETS (similarly to cube and rollup)
   • it's supported currently only in SQL mode

Spark SQL

1. Creating custom Encoder
   • SPARK-17668 Support representing structs with case classes and tuples in spark sql udf inputs
   • Create an encoder between your custom domain object of type T and JSON or CSV
   • See Encoders for available encoders.
   • Read Encoders - Internal Row Converters
   • (advanced/integration) Create an encoder for Apache Arrow (esp. after the arrow-0.1.0 RC0 release candidate has recently been announced) and ARROW-288 Implement Arrow adapter for Spark Datasets.
2. Custom format, i.e. spark.read.format(...) or spark.write.format(...)
3. Multiline JSON reader / writer
4. SQLQueryTestSuite - this is a very fresh thing in Spark 2.0 to write tests for Spark SQL

• Changelog

4. http://stackoverflow.com/questions/39073602/i-am-running-gbt-in-spark-ml-for-ctr-prediction-i-am-getting-exception-because
5. ExecutionListenerManager
6. (done) Developing a custom RuleExecutor and enabling it in Spark
   • Answering Extending Spark Catalyst optimizer with own rules on StackOverflow
   • Sparkathon - Developing Spark Extensions in Scala on Sep 28th

Structured Streaming

1. Developing a custom StreamSourceProvider
2. Migrating TextSocketStream to SparkSession (currently uses SQLContext)
3. Developing Sink and Source for Apache Kafka
4. JDBC support (with PostgreSQL as the database)

Spark MLlib

5. Creating custom Transformer

• Example: Tokenizer
• Jonatan + Kuba + lejdis (Justyna + Magda)
• Problem to zapis Pipeline z tym Transformera, odczyt i użycie.

8. Spark MLlib 2.0 Activator

Core

1. Monitoring executors (metrics, e.g. memory usage) using SparkListener.onExecutorMetricsUpdate.

Misc / Next Meetup / Backlog

1. Develop a new Scala-only TCP-based Apache Kafka client
   • A Guide To The Kafka Protocol
   • KAFKA-3360 Add a protocol page/section to the official Kafka documentation
   • See Scala Kafka Client for inspiration yet it's just "a thin Scala wrapper over the official Apache Kafka Java Driver"
2. Working on Issues reported in TensorFrames.
3. Review open issues in Spark's JIRA and pick one to work on.