5/5. This is pretty academic, but it's pretty great, too. It starts with datalog, which heavily inspires his project Dedalus; there's a way better dissection of this talk here.
- Main topic of discussion: "How would you [re]design your infrastructure" if you could never log in
- Containers are the MVP unit for deploying apps; VMs overkill
- Processes, not lightweight VMs
- Bunch of Docker demo crap
- Tons of cute names up in this business "kubelet," etc.
- Pods: logical applications
- 1+ containers
- Shared namespaces
- Like docker compose, etc.?
- Contains manifest, etc.
- One-off, by itself
- Kubernetes scheduler
- Does obvious things
- Pluggable
- Replication controller
- Used for long-running apps (like web servers)
- Self-healing
- Hand pod to replication controller
- Kubernetes has a p sweet UI
- Uses Raft, apparently
- Unclear at this point in time how to avoid issues like we saw where other containers need to be up before health check for main app starts passing...
- Dude played Tetris to explain bin-packing
- Pretty ok resource management with cgroups, etc.
- Fairly easy to implement canary pattern / roll new versions using service manifests
kubectl rolling-update --update-period=n00ms
1/5. All I got out of this talk was that they are using Datomic for something, Spark for something, and don't know anything about giving presentations. Notes
- Using Datomic
- Kinda cool, I guess
- Weak talk so far
- Handwaved over using timestamps on separate dbs for ordering?
- Interesting question about unifying/enriching data in microservice dbs with point-in-time dbs, though
- I guess no distributed transaction coordination in Datomic?
- These dudes put like zero effort into sensible, labeled graphics
- And they just reused the same colored ball to mean different things in different slides
- There's like nothing of value in this talk
4/5. Not the world's most technical talk, but I'm a fan of Bailis' work. Adrian Colyer does a blog that summarizes interesting CS papers every day, and he's covered a few of Bailis' work (the latter including Peter Alvaro). A lot of this has to do with consistency guarantees at the margins. This one's more about software in general. Notes
- What if we designed systems for worst-case scenarios
- Cluster provisioning: 7B simultaneous users!
- Hardware: put that shit on Mars
- Security: all developers malicious!
- Basic takeaway: designing for the worst case penalizes the average case
- This talk: When can designing for the worst-case improve average
- Distsys, beyond the network, lessons
- Networks make design hard: packets delayed, dropped, can't tell the diff
- CAP theorem, etc.
- Coordination-free systems: enable infinite scale-out, improve throughput, ensure low latency, guarantee always-on
- Follow-up: Harvest & Yield paper
- Is coordination required for read commited? Research: no! Buffer writes; OOM increase
- Other scenarios: failure helps devops; automated failover for upgrades, revocation, stragglers, etc.
- Strong business case for accessibility
- Universal design
- Pull quote: defining "normal" defines our designs
- Basically a VoltDB talk
- Active-active with some coordination service = logical log
- Be aware of sources of nondeterminism
- random numbers, wall clock time
- record order
- External systems
- Non-user sources: bad memory,
- Voltdb's sql planner understands determinism
- 100% of DML deterministic
- R/W transactions slow down and do ...smart things
- Warn you when you load a ND procedure
- Coordinator service ensures determinism, sends statements out to all replicas, receives hashes back to ensure consistent results
- Out-of-band checks with hashes to do consistency checks on replicas
- ND reads allowed
- One huge caveat: when a divergence is detected (obviously super-rare), cluster shuts down, essentially
- Voltdb apparently allows you to partition tables on a column
- https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/txnid-selfcheck2
4/5. This was a really great talk about, in the era of microservices (a common theme), how you can build state into systems without necessarily giving up the sweet scalability that stateless systems enjoy. Notes
- Services intrinsically have state
- Moved to the DB for consistently, state coordination leaking into services
- Stateless services: Data shipping paradigm
- Benefits
- Data locality for low-latency/data intensive programs
- Function shipping paradigm
- Sticky connections really make a difference in this model; enable read-your-write w/ AP semantics
- Building sticky connections
- Persistent connections: load balancing problems, no stickiness when connection breaks
- Need to implement backpressure for load balancing
- Routing logic respects stickiness
- Cluster membership / work distribution are the issues
- Static cluster membership sux
- Dynamic cluster membership
- Gossip protocols
- Consensus
- Availability vs. consistency
- Work distribution
- Random placement (write anywhere / read everywhere -- Bloom filters?)
- Consistent hashing
- Deterministic placement (hash ring)
- Distributed Hash Tables
- Persistent connections: load balancing problems, no stickiness when connection breaks
- Facebook Scuba DB
- Static membership
- Write to one random node, acts as broker to coordinate writes. After time period (configurable?) returns % of replicas written to, user decides if ok, etc.
- Read paper for more
- Uber ringpop
- App-layer sharding for dispatching platform services
- Swim/Gossip protocol + Consistent hashing
- Orleans
- Gossip protocol + Consistent Hashing + DHT
- Actor model
- 100ks of actors per machine
- First round: consistent hashing to get DHT for actor location, second round: route to machine with actual actor running
- Hot probs
- Unbounded data structures: implicit assumptions are the killers of dist systems
- Memory management
- Reloading state
- "Fast restarts at Facebook" paper
3/5. This is like The Kafka Talk. If you're familiar with Kafka, it's a lot of already-tread ground. If you're not, it can get you up to speed. Notes
- What is stream processing?
- Request/Response
- Batch (data locality huge advantage)
- Stream processing
- Doesn't necessarily mean transient, approximate, or lossy
- Confluent/Kafka trying to address fact that work often pushed down into batch or up into request/response
- Stream processing: The Hard Parts
- Partitioning/scalability
- Semantics & fault tolerance
- Unifying with tables
- Time
- Reprocessing
- Kafka is like a commit log in a database
- Stream processing = Logs + App code
- Log compaction: reducing space for redundant content
- Kafka streams (KIP-28)