Skip to content

Instantly share code, notes, and snippets.

@mikekaminskycc
Last active August 29, 2015 14:20
Show Gist options
  • Save mikekaminskycc/e64fd7613e776df1779f to your computer and use it in GitHub Desktop.
Save mikekaminskycc/e64fd7613e776df1779f to your computer and use it in GitHub Desktop.
Bottled Water Summary

#Bottled Water

##Summary

Bottled water takes advantage of logical decoding (available with PostgreSQL 9.4) to funnel changes made to the database to a Kakfka stream in an Avro format which can then be transformed and sent somewhere else in a stream. Until PostgreSQL 9.4, if you wanted to stream changes you had to use triggers which is unappealing because of the burden they place on the databse servers.

Bottled Water Diagram

Bottled water currently does not work on a replica/follower DB. I.e., you have to install and deploy on your master DB. See discussion here. This appears to be a conversation between Martin and a core PostgreSQL developer, so I would take them at their word. This would be a fairly signficant barrier for Case Commons.

##Author & Sponsor

###Martin Kleppman

This dude seems super sharp and active in the distributed-systems / database development community. For example, see this post on distributed systems I just grabbed off of the top of HN without knowing it was by him.

###Confluent

Confluent was founded by the team who created Kafka while at Linkedin. Confluent is trying to build a streaming platform based on the Kafka infrastructure.

The Confluent Platform is a stream data platform based on Apache Kafka that enables you to manage the barrage of data generated by your systems and makes it available throughout your organization. It provides various industries, from retail, logistics, and manufacturing, to financial services and online social networking, a scalable, unified, realtime data pipeline that enables applications ranging from large volume data integration, to big data analysis with Hadoop, to realtime stream processing.

Confluent lists Benchmark, Linkedin, and Data Collective as investors.

##Community

##Documentation & Examples

##Dependencies

See the dockerfile here

##Test coverage

##Alternatives Xavier Stevens has been working on a similar project for logical decoding PostgreSQL into Protocol Buffers called decoderbufs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment