Create a gist now

Instantly share code, notes, and snippets.

Introduction to Stream Data Processing

Introduction

This gist started with a collection of resources I was maintaining on stream data processing — also known as distributed logs, data pipelines, event sourcing, CQRS, and other names.

Over time the set of resources grew quite large and I received some interest in a more guided, opinionated path for learning about stream data processing. So I added the reading list.

Please send me feedback!

Reading List

The resources doc has a lot of good stuff, but no guidance. This reading list is meant to be a more guided, opinionated path for learning about stream data processing.

Some works are accompanied by alternative options and/or responses, but those are completely optional. If possible, try the main works first.

The Foundational Monograph

The Log: What every software engineer should know about real-time data's unifying abstraction by Jay Kreps (December 2013) kicked it all off for me. A seminal work.

Alternatives (optional)

Responses (optional)

The (Conference Talks made into Articles made into a) Book that Fills in all the Gaps

Making Sense of Stream Processing by Martin Kleppmann (March 2016) is a free ebook that compiles many of Kleppmann’s brilliant articles (based on his brilliant talks) on this topic.

This is a fantastic book that covers everything from theory to practice, history to the future. It’s all broken down into small incremental ideas and clearly explained.

Alternatives (optional)

If you’d prefer to start with videos of Kleppmann’s talks, I recommend starting with these:

  1. Turning the database inside out with Apache Samza describes how we might reimagine what a database is and reshape the entire Web application stack with event streams at every level. I heard this in person and it blew my mind.
  2. Staying agile in the face of data deluge illustrates that “using the right tool for the right job” can lead to incredibly complex and fragile application architectures, and how streaming data can simplify.
  3. Systems that enable data agility
  4. Samza and the Unix philosophy of distributed systems
  5. Data liberation and data integration with Kafka

For more, see Kleppmann’s playlist of all his talks on YouTube.

The Treatise (on Why and How Stream Data Processing Might be the Future of Application Development)

Introducing Kafka Streams: Stream Processing Made Simple by Jay Kreps (March 2016) Explains the why of the new Kafka Streams framework, and in doing so dives deep into what is all this stuff, really, and why does it matter, and what does it mean for application development — brilliant.

A Broader, Cogent, and Less Kafka-Centric Perspective

The world beyond batch: Streaming 101 by Tyler Akidau (August 2015) is a super-helpful alternative perspective that didn’t come out of LinkedIn but rather Google. Akidau has worked for years on data processing systems at Google, including MillWheel, and Cloud Dataflow, and Apache Beam. I haven’t yet read part 102 but suspect it will be similarly illuminating.

Resources on Stream Data Processing

The Monograph

The Book Version of the Monograph

Articles

Books

Talks

Blogs

@aprilrabkin

I'm looking forward to reading these. Justin lent me I <3 logs a year ago when it was just a vision at Park Assist, and it's really excited to see it finally being implemented. These other readings look good too.

@aviflax
Owner
aviflax commented May 5, 2016

@aprilrabkin thanks! Please let me know what you read next and what you think! Let’s discuss!

@WillEngler

Here is a working link to the free download of Making Sense of Stream Processing. The Confluent link redirects to the home page, apparently since May 7.

Thank you for this list @aviflax!

@aviflax
Owner
aviflax commented Nov 16, 2016

@WillEngler thank you! My pleasure!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment