redis.conf notes

gistfile1.txt

Via: http://news.ycombinator.com/item?id=4686338

https://gist.github.com/3932768
https://gist.github.com/3933451
https://gist.github.com/3934073
https://gist.github.com/3935007
https://gist.github.com/3935671

Redisconf Notes:

"We will begin this morning with a performance"
	"Is this a key store..." Bohemian Rhapsody Key/Value Store ballad.  Epic Win!
	"Any way the data flows doesn't really matter to me... flushdb..."
	"Lua killed a db by writing after calling a nondeterministic function..."
	"Got to leave SQL behind and face the truth..."
	"Redis, I don't want to join, sometimes I wish I'd never described a query..."
	"The ZSET has some data set aside for me, for me, for MEEEEEEE"

"Redis in perspective" - Antirez.
	- Can't be here because his baby is imminent.
	- Past, present and future of Redis. "Humble project", humble because it is supposed to be small and solve one problem.
	- Redis Manifesto, written 1.5 years ago, still valid.
	- Hetzner.de - dedicated 64GB RAM for 106 euro / month.
	- Will be in memory, always, but will replicate to disk.
	- "Don't touch the code too much, don't abuse the code."
	- "Fight against complexity" - "Virtual Memory was removed"
	- Redis codebase is less than 30k LOC, including Hiredis
	- Simpler systems are easier to understand [in their failure modes]
	- Two APIs for two different worlds
		- Some commands are tricky in distributed systems
		- Many K/V dbs that are distributed are simple get/put keys
		- With set intersection, etc, it is hard to do this in distriubted way.
		- So, one is extended API for single-server, one subset that is implemented in Distributed way.
	- "Optimize for joy. Redis is simple to get started, to understand, to use."
		- "I right much of the documentation myself because I think the documentation is at least as important as the code; for the users and for me...[because it helps to identify deficiencies in the design]"
		- "Open-Source is optimized for joy"

"Two things Redis is not (and likely never will be)" - Pieter Noordhuis
	- "Redis does a few things well, and a few things not so good... and this is something people seem to forget."
	- "It is SUPER easy to get started.. and that is both a good and a bad thing."
	- "I work at VMWare where I split my time between Redis and CloudFoundry...started contributing in 2010..."
	- "I also maintain Hireds, but I'm not a python, ruby or node guy.. send pull requests if stuff doesn't work"
	- Doesn't know where we are in the Software Hype cycle
	- "1 million ops/second on a commodity box... but there are tradeoffs... and that is important to talk about"
	- "Redis is NOT an available system." (CAP theorom, it is a CP system.)
	- Half of audience uses redis master with multiple slaves, a quarter uses single-node.
	- Aware of cumatative distributed datastructures, but says it does not apply to Redis
	- Describes Sentinel and consensus wrt promoting master
	- Redis can be "highly available", but data loss still available due to replication lag
	- Responding to my question, "Right now, nothing to mitigate master dying while writing out buffer, leading some slaves to have some data that other slaves don't, so you can get inconsistent data across slaves.  In the future, there may be a command count so we can pick the slave with the highest count.."
	- "Redis does NOT grow beyond RAM"  Because it is single threaded, if you have to wait for kernel to page in memory, everything blocks.

Using Redis to build analytics - Jeff Seibert from Crashalytics
  - http://www.slideshare.net/crashlytics/crashlytics-on-redis-analytics
	- Crashalytics reports back crash data from the line of code that your app crashed on.
	- On a quarter-billion devices right now
	- Strings for Activity Tracking. Hashes for event tracking.
	- We don't care about 2 years ago, just last few months.
	- Bitmap users to bits and mark bit when user is active in a string for a time period; one key per time period
	- Do weekly active by bitop OR'ing your days for the week, then do bitcount to sum
	- % Active last week vs this week: bitwise OR the two weeks
	- Churn: OR last N months and then NOT that from the current set.
	- Counts not worth using bitmasks, use HINCRBY, with a hash per app, key per period. Harder to expire, but easy to read (HMGET all keys for time period)
	- For grouped things, put time componant in keyname, put groups as keys in the hashes
	- "And then there's MongoDB... and that always adds a wrinkle", wants an auto-incr primary key, just like mysql
	- Uses zadd + zcard to store mongodbid->auto-incr primary key, using lua scripting to do so atomically.
	- "How do you handle persistence if it is only in redis?" - "Internal reporting, slaves accross AZs.  For production data, we have two-tiered...[speed layer, batch layer]..we're also persisting all raw data...into cassandandra [if we lost all of it, we could restore from cassandra]"
	- "How do you prune the data if it is embeded?"- "[Good question]... so far, we haven't addressed that problem..."
  
Dr Josiah Carlson
	- Redis user for 2 1/2 years. Extremely active on redis mailing list, #3 poster to the list (applause from PN)
	- Author of Redis In Action (available electronically)
	- What is search? scan text, use BM/BMG/KMP or regex or recursive-descent.. or suffix tries.. or you can search through the zip-format (BWT)
	- Inverted index: words to documents, instead of crawling documents for words.
	- SET per word with items for each docid, using redis' intersection for AND, union for OR
	- "Simplest example that will be useful..[but it wont be that useful... buy the book!]"
	- Demo of his editor.. its on sourceforge.
	- Tokenization, stop words in some python code.. "with this stuff, we're going to implement search" already has failing tests written.. live coding, so not many notes... buy his book.
	- "I have 5 minutes? I thought this was a 45-minute talk".. "Nope!"..."Shoot!"
	- "If you like intersecting sets and unioning them...you're welcome" (Josiah created them in fork at old company...)
	- "In the book, I cover fake-bayesian learning..[so it learns ads per word]"
	- "You can build an ad-targeting engine in 3-4 weeks and be making money...if you do, I accpet donations!"
	- User: "How do you do exact phrase search [if you use inverted index]?" A: "View content as non-stopword to non-stopword keys.. so you would store "lord" and 'rings' but also 'lord rings'" U: "Like trigrams?" A:"[Like fake trigrams, but more space-efficient]"

Aren Sandersen
	- @Pinterest for 6 months, which is longer than 75% of the engineers. Pinterest is all scaling firedrill, use redis a lot.
	- Overview of pinterest. Compressed-air powered Nerf gun pic + lols.. tour of screens
	- Users can follow other users, or a subsection (board) of their posts, so graph is 'interesting'
	- "Been called the fastest growing website..[it's all about scaling]" shows hockey-stick growth to jan, flat til may.
	- 1 year ago: 4 Cassie, 15 membase, 10 memcache, 10 redis nodes, 2 engineers
	- May: 59 redis, 51 memcache, 66 mysql, 6 engineers, sharded solr
	- Nowish: over 200 redis, 50 engineers..
	- "Would not have been able to keep up without the cloud...[sometimes, I spin up 50 servers for a project]"
	- Justifies using all MySQL, Memcache and Redis.
	- "Sharding is the #1 way we scale..[pre-shard so we can scale easily]...[earlier the better]"
	- Started with many dbs per physical server. master-master, but only write to some.  To scale, shift around where physically some of the dbs are located and update config.
	- Pin ID is structured with shardId,type,localId (localid is id within table)
	- They shard based on user join time, and maintain giant lookup structure for dbId to physical layout.
	- Using redis for: pyres (resque for python), followers, feeds, caching in memcache and redis, anti-spam
	- Truth in MySQL, redis as cache vs memcache because it allows paging.  use memcache for id:small blob
	- Starting to use redis persistently, so not sure how long mysql will be source of truth.
	- Stagger bgsave.. but if a slave comes down and comes back up and all try to reconnect, the master's procs will all have to bgsave to let them sync. Use aof + hourly (staggared) bgsaves.
	- use lua to avoid race conditions in updating list of followers. 
	- Twitter's twemproxy is adding redis support - run one per f/e host to pool connections.
	- U: "Why not just redis instead of redis+memcache?" A:"We haven't explored that..[historical reasons]"
	- U: "I noticed you went from having Mongo to not having Mongo..[why'd you leave it?]" A:"I wasn't there at the time..[they just weren't having good experiences..hard to say if that was the product or our usage..wanted something we could get support on if needed..]"
	- U: "Any issues with lua scripting & expensive ops being slow and causing problems?" A: "Yes..there's a learning curve like everything.. but I think the team is very happy now with their decision.."
	- U: "What happens when master fails?" A:"[It's all manual, there is a page..] code push to promote slave to master...working to streamline process..use zookeeper [to avoid code push]"
	- Hourly backups of aof to S3
	- U:"How do you force staggared BGSAVES?" A: "Cronjob"

Antirez - Sentinel Solution
	- Sentinel is distributed monitoring so it is more reliable from Ops' point of view.
	- Notification system
	- Automatic failover, client config server
	- Monitoring: if not distributed, availability from one point of view.  We want to know if it is up from multiple POVs with an agreement protocol.
	- Image of gossip protocol: N:N pings
	- Demo using terminal. redis-sentinel IS redis-server (same sha), just detecting name. its a mode of execution
	- Sentinel conf is easy because slaves and other sentinels are auto-detected
	- You can run dozens of sentinels without issue if you have room for the processes. Set quorum as a fixnum in conf..
	- In sentinal mode, redis responds to ping, 'info sentinal', and 'sentinal' command. NO set or other keyspace commands
	- sdown vs odown: sdown is "down for me", owdown is once quorom is reached. "slaves don't require quorum because quorum is just used to promote masters"!
	- You can subscribe to status using pub/sub channels, or you can have scripts that are executed when event happens. Demo of 'subscribe *' + bringing down slave yielding an 'sdown' event, and subbing to jus 'sdown' for selective message subscription.
	- Automatic failover: requires client cooperation. odown detected, failover leader sentinal elected, slave selected, turned into master, other slaves repointed @ it. demo of it in-action. takes several seconds. uses timeouts to avoid race conditions.
	- Walkthrough of sentinals following promotion-master and the slave promotion, use of sentinal get-master-addr-by-name
	- Sentinal is 3k LOC. Uses redis codebase as framework; developed in few weeks instead of months. Planning to update sentinel with every point release of 2.6
	- Ruby, Python, Node.js clients already support it.
	- Redis 2.8 will support config through sentinal, so no need to say "slaveof" on startup.

Lightning talk: BigCache:
	* redis distributed fault-tolerant memory cache as a service (OSS)
	* memcache binary protocol compatible
	* TCP loadbalanced / consistent hashing algo, ZK for coordination services
	* github.com/mercadolibre/bigcache


EvilSha: misadventures in the land of lua. Adam Baldwin: @adam_baldwin
	* What can we do that is evil with Redis?
	* Listed all functions you have access to in lua in redis; pretty locked-down
	* redis.sha1hex() - fritz got this in as a patch.
	* github.com/evilpacket/redis-sha-crack : distributed rainbowtable cracking
	* Scanned ec2 and found 1805 redis servers "waiting to be talked to"
	* Most are not running unstable branch :(
	* "The internet is full of redis instances.. 360G of key data hanging out there.."
	* Showed redis instances running on this network, cracking pws on local box.
	* "This is my verbal bug report because I'm bad at that".. dofile points to lua file on disk or reads from STDIN if no argument, debug.debug() puts you into a Lua repl

Qless - Dan from SEOMOZ, one of the sponsors!
	* Problem with dropped jobs, which was not good because some jobs can take a week.
	* B2B, very low tolerance for bugs.  Needed better support for debugging, time to execute, stats.
	* Moved from old in-house system to new OSS system.. need features like dependency tracking between jobs
	* github.com/seomoz/qless-core maintain ruby/pytho, community perl, java coming soon
	* description of job data structure, demo, "this is not as good looking at low-rez."
	* Dashboard easy for helpdesk to use.
	* All logic in Lua scripts, git submodule of clients.  Clients just wrap them but don't have logic.
	* Doesn't fork for each job. Speaking of jobs, SEOMOZ is hiring. bit.ly/work-at-seomoz

Reddish - GUI for Redis: Danielle & Jim from Freeflow Labs
	* Danielle: prototyper / ux developer
	* Reddish is a GUI for redis for rapid prototyping as npm module
	* Demo on heroku, but offline right now because heroku is down as well (showing from localhost)
	* looks like phpmyadmin for Redis
	* supports INFO, updating through gui, easy documentation showing from redis json docs.  expiration counting down. 
	* Uses `monitor`, so don't use in production.  Doesn't support pubsub or lua scripting (yet!)
	* Virtual scrolling so it only renders what you see, supporting thousands of values easily
	* github.com/FreeFlow/Reddish MIT-licensed
	* need help with documentation and tests.. have docco docs so please just add some comments
	* all coffeescript sourcecode. open to switching over if people need js..
	* MONITOR parsing is handled within the app.
	* Q:"What was the project that you were building that [made you build this?]" A:"[something we aren't working on anymore] but it gave rise to reddish, which is a much better project."

Redis Pain - Matt @mranney from Voxer (did node-redis)
	* Asked to talk about redis stress points.. "no stress, things work for a while and there is no stress and then... you enter a world of pain."
	* Pain from how we use it at Voxer. Explanation of Voxer and its use cases.
	* People assume Voxer is "how hard can it be?".. "That's how it used to be until we got a bunch of pictures..."
	* Growth curve looks like Pinterests "might be the same because there was no label on the y-axis.. might be EXACTLY the same."
	* Computers are hard... nothing works as it is supposed to... and eventually you fly into nerd rage and then you become a curmmudgeon... getting serious, this isn't real pain -- we aren't selling stree sheets -- this are great problems to have...
	* They use redis as cache for Riak, also for rapidly changing data, "data we can afford to lose", throttling, NO SAVING.
	* "We really like redis."
	* "When we first started building voxer, I never understood why people use Redis at all... we have a db and ... you can make them go faster ... buy ssds ... why use redis at all?" Initial pain of getting an idea about why and how to think about redis.
	* Docs aren't clear about what you are trading off to get all of this magic.
	* Temptation to dive in...setting yourself up for failure down the line.
	* Native clients... had to implement rate-limiting... not for hackers, but for ourselves.
	* Everything was going to couchdb, which had no way of handling rapidly-changing data.. started using redis for throttling
	* "We were already using CouchDB so [when we added redis] we had this hipster devops thing going"
	* Pain 2/6: Problems with BGSAVE. Copy-on-write is fine if you don't have quickly-changing data.. but quickly changing data is why you use redis -- so if the bgsave takes a while, then you can consume 2x memory and get yourself into "a deep, deep hole if you are changing your data faster than it can be written".. this bgsave also happens whenever a slave connects.. if your slave connects while a bgsave is happening -- wah-wah, now you have TWO bgsaves going on..
	* Pain 3/6: Reading back the data from disk.  "As if writing wasn't bad enough, reading is worse".. If you have lots of data, your server is **offline** for minutes.  "The server is up and accepting tcp connections, but the only valid command is INFO."
	* You know, AOF, I want to mention. We don't actually use it, but theoretically, it might limit the exposure to data loss, but it makes MORE disk i/o.
	* Right solution: expect less!  Disk store and vm were failures, but that's fine! I'd say aof and snapshots were also failures, but that's fine because it is a GREAT in-memory db.  If you just expect it to not [do persistence] then, it is great.
	* "The things redis is great at, being fast, are directly at-odds with writing to disk"
	* Pain 4/6: Single-CPU
	* "We'd really like the API of Redis but the clustering of Riak.."
	* "Somebody should do that.. take redis and maybe a postgres backend..?"
	* Pain 5/6: Scaling. SPOF.
	* Even at riak meetups, everyone is talking about redis cluster.
	* You HAVE to shard eventually.. "I would argue the best way is in your application...it sucks but at the moment, that's just kind of how it is.."
	* Pain 6/6 Operations: Just because it doesn't crash very often, and it is true it doesn't crash often, doesn't mean things don't go wrong.  When they do, it is VERY hard to tell what is going on -- memory, cpu, etc.  Complaints about impact of MONITOR (double CPU, gets backlogged, etc.)  "What actions can you take to get your memory back, or have you screwed it up?" It would be nice if we had DTrace support...
	* It would be great it redis had better visibility so you knew where your memory and cpu were going.  "Copy-on-write is clever, but it is very surprising and awkward for ops people."
	* The issue of support -- and it is kind of lame to suggest this is a problem -- ... but people who run businesses, they want support contracts.  Some people on mailing lists offer support, but it seems amateurish...
	* Eventually this will be solved... but until then, we have work to-do.
	* "It's all WAY more efficient to pipeline...fewer packets on the network..."

Redis PubSub and ActivityStreams - Monica Wilkinsin from PSHB @ciberch
	* Lessons learned building realtime activity stream with node, redis and mongodb
	* "About me": Recently joined crushpath, developer for a while, fond of Node.js. Prior to that, worked at VMWare, facebook, socialcast, myspace
	* Defining activity streams a la facebook, jira, github, "general event tracking"
	* "Generally, activities are shared passively...[unlike twitter's active posting]...they are usually being tracked"
	* How can we *share* activity streams (standards?) http://activitystrea.ms/ worked with people from a variety of companies, settled on json.  
	* Open Web Foundation licensing, so anyone can use it.
	* Example of a post.
	* Defines activity stream engine, managing pub/sub, fanning out, etc
	* Coded up node/redis/mongo implementation of the spec that helped write, using express and node-express-boilerplate and socket.io, switched plaintext for activitystream, added persistence, used redis pub/sub to overcome socket.io's single-server limitation
	* "Unlike Danielle, my UX skills are terrible, so please don't judge me on that."" The final app: http://github.com/ciberch/node-activities-boilerplate
	* "I'll talk one slide about Mongo, don't kill me!"
	* Explains joinless mongodb usage, but having it be more structured than Riak because of in-document queries. "Some people would call it schemaless, I call it flexible schema.. just like activitystreams schema, anybody can extend it."
	* Replaced in-memory store of messages with redis. Description of how PubSub in redis works.
	* "In my first version, I forgot to unsubscribe, so in a couple of days I crashed the app.. so don't forget to unsubscribe."
	* node module: activity-streams-mongoose. Wraps getters/setters to auto-publish.
	* Description of client-side with Backbone hooked up to socket.io and jade templates.. moved rendering from server-side to client-side.. "was really cool to be able to use jade templates in both places.."
	* Demo: asms.cloudfoundry.com, posting photos and watching it updates.  In redis-cli, subscribing to the photo
	* Filtering client side vs server-side, trade-offs.
	* Talking about use-cases of activity streams.. started with Facebook just to fill the page provide opportunities to Like and comment.. enterprise tools / "knowledge sharing"..
	* Now, started with rails, lots of 3rd-party integration;linked in, salesforce, superfeeder, lots of complexity.
	* Talking about using redis as message bus for integration, kind of like an enterprise service bus.

Rate-limiting w/ Alan Shreve, core engineering and architecture at twilio
	* New distributed rate-limiting queues with redis 2.6
	* Talk about why it was built, and the implementation.
	* The problem is that the phone network has rules.  One of the rules is that you can only enter the phone network at a bounded rate -- you can only originate requests at a certain rate -- they provide buffering and queueing to hide this from users.
	* Different rules; some regulatory, some bizdev / 3rd party / contracts, "origination" is a limited resource
	* Capacity management is tricky, this lets them smooth it out so they can have lead-time to scale to accomodate spikey demand. Animation that illustrates temporal smoothing.
	* "Leaky bucket queue" algorithm; no matter how fast you fill it, it always leaks out at the same rate.. form of "traffic shaping".
	* This is easy if you have one queue, but it becomes tricky when you scale and have millions of open and active queues.
	* Additional twilio-imposed constraints: must be horizonally scalable. low-latency (<10ms).  HA & Fault-tolerant. NO DATA LOSS ALLOWED. Runs majority of software on EC2. So, in the case of arbitrary node failure, need to keep all of these properties.
	* At least 1k originations/sec per node, 1k queues per node at least.  Multi-tenant and transactional.  Pull-based, not pushed.  Customizable, transparent, introspectable.
	* Queueing is very common problem, not too many rate-limiting queues that are out there.  Most are counter-based, not leaky-bucket.
	* "Redis didn't even exist when we first took a crack at this problem".. issues with existing solutions not having fanning support, durability issues.
	* Tried several approaches: 1st pass, sql, "when all you have is a hammer..".  SQL's over-capacity failure modes for queues didn't fit, deadlock issues, etc.  2nd pass: redis and timers.  Service that enqueues and a dequeuer. (Detailed description of system.)  3rd pass: redis w/ lua scripting. (detailed description of how it works).
	* Redis wins because it is high-performance and is easily customizable through Lua scripting.  "What is advantage over just writing a networked server?" "Easy and free replication and persistence if you want.."
	* MONITOR shows all of the commands that your Lua script executes. 


Lightning talk from Brock  @sintaxi
	* Complain about antipattern.
	* APIs that have functions, so we end up creating SDKs to access our resources
	* Thinks we should change how we structure our models: rolodex is a project that supports this paradigm where interface is the same if it is local or remote.
	* Also created a project called thug which is a "non-orm", which is Express middleware
	* "The theme is that the tool's shouldn't be distributing the system, it should be our applications..."

Notes from my talk are here:
	https://gist.github.com/3935383

"Doing crazy stuff in redis" - Fitzy @fritzy from &Yet
	* Everything on the internet is feeds in feeds, your follower list, your status updates..
	* XMPP pubsub interface, called ZUMP, was XEP60.. it "kinda sucked" and "was really slow"
	* I wanted something that fans out messages and was fast and if only there was something that already did that -- well they did, and it was redis.
	* Implemented XEP60 in redis, and it was fast and awesome. "And then I had an epiphany"
	* "I didn't need XMPP, this could be useful for ANY API..." events, and publishing messages could be useful for all the apis. "I know some of you JavaScripters think there's just HTTP, but there is a whole world of Protocols and APIs that could benefit."
	* "What if we had CRUD events that came out of custom objects that weren't just feeds?"
	* "[In node] there are events, and they can interrupt your watch/multi/exec... and life is too short to just crush bugs..."
	* "Lua was there so we could implement our own commands.. and we had taken it there.. but we can take it farther[SIC]" .. Henrik asked: "What about ACL or object validation?" .. Yea! .. And then Henrik started driving!  But we were doing something crazy, so I was into it...
	* The scripts started getting very long... found a crazy hack to call scripts from other scripts.. then started playing with making macros... 
	* Boss: "Is this a good idea?"  Fritzy: "No! But it is AWESOME!"
	* Started doing tests, an 800 line lua script can run thousands of times a second on a modest PC.
	* It's no longer a database, it is an application service.  "It is crazy shit, don't do it!"

Michel Martens - Author of redis.rb and longtime redis contributor
	* soveren, cyz, djanowski
	* made redis-cli -i, brpoplpush, redis.io, redis-doc, redis-rb
	* "Ruby is a community that has a tendency to create loaded tools...at some point, I realized that everyone thinks that what they do is simple.. nobody goes around saying 'I write unneccessarily complex code'..."
	* Discussing "Worse is Better", we can focus on completeness or simplicity of implementation even if we leave out some features.
	* "I identify with the second group...for people of the other group, simplicity is solving more problems with the same tool"
	* (Discussion of 'Shield' vs 'Devise' ruby libraries...) Shield is 100 LOC + 100 LOC for dependencies, Devise is 60,0000 LOC if you include deps (railties, which pulls in 1/2 of rails. Tilt, which has lots of deps as well..)
	* LOC is a controversial measure because it favors people who prefer simple implementations.
	* "Complex tools are hard to maintain"
	* He wants to "use a tool that knows how to handle dependencies, implicit and explicit"
	* "The thing with Bundler is that it has to be complex itself...7,000 LOC...very fragile tool... at any time, 300 reported bugs"
	* Mote, cuba, shield, ost, ohm.
	* Three instances of how Salvatore confronts complexity and deals with it.
	* 1. Started small: Existing tools for analytics were too big or too slow
	* 2. redis-cli made it interactive when txns added because the client/server had to maintain state.  In beginning, it didnt have editing or history.  Salvatore hated that readline had more LOC than redis at the time, so he wrote LineNoise
	* 3. virtual memory: "in the early days, i was a heavy proponant of vm".."i wanted to use redis for everything.. [when I and others started using it, we realized it was not good and too complex]. So what happened to my dream of using redis for everything?"
	* You can either change which tool you're using, or redefine the problem.
	* Adding things in redis is easy, removing them is tricky. Created Ohm to wrap common patterns.
	* Simple tools let you know not only what is happening, but make it easier to understand things when they go wrong.

Antirez (via video)
	* 2.6 is live Now! 1383 commits.
	* Lua scripting, bitops, better expired keys
	* Fewer latency spikes, better slow client handling
	* Now we are ready for 2.8, which will be incremental release on 2.6