aaronblohowiak/gist:3933451

## gistfile1.txt
Dr Josiah Carlson
	- Redis user for 2 1/2 years. Extremely active on redis mailing list, #3 poster to the list (applause from PN)
	- Author of Redis In Action (available electronically)
	- What is search? scan text, use BM/BMG/KMP or regex or recursive-descent.. or suffix tries.. or you can search through the zip-format (BWT)
	- Inverted index: words to documents, instead of crawling documents for words.
	- SET per word with items for each docid, using redis' intersection for AND, union for OR
	- "Simplest example that will be useful..[but it wont be that useful... buy the book!]"
	- Demo of his editor.. its on sourceforge.
	- Tokenization, stop words in some python code.. "with this stuff, we're going to implement search" already has failing tests written.. live coding, so not many notes... buy his book.
	- "I have 5 minutes? I thought this was a 45-minute talk".. "Nope!"..."Shoot!"
	- "If you like intersecting sets and unioning them...you're welcome" (Josiah created them in fork at old company...)
	- "In the book, I cover fake-bayesian learning..[so it learns ads per word]"
	- "You can build an ad-targeting engine in 3-4 weeks and be making money...if you do, I accpet donations!"
	- User: "How do you do exact phrase search [if you use inverted index]?" A: "View content as non-stopword to non-stopword keys.. so you would store "lord" and 'rings' but also 'lord rings'" U: "Like trigrams?" A:"[Like fake trigrams, but more space-efficient]"

Aren Sandersen
	- @Pinterest for 6 months, which is longer than 75% of the engineers. Pinterest is all scaling firedrill, use redis a lot.
	- Overview of pinterest. Compressed-air powered Nerf gun pic + lols.. tour of screens
	- Users can follow other users, or a subsection (board) of their posts, so graph is 'interesting'
	- "Been called the fastest growing website..[it's all about scaling]" shows hockey-stick growth to jan, flat til may.
	- 1 year ago: 4 Cassie, 15 membase, 10 memcache, 10 redis nodes, 2 engineers
	- May: 59 redis, 51 memcache, 66 mysql, 6 engineers, sharded solr
	- Nowish: over 200 redis, 50 engineers..
	- "Would not have been able to keep up without the cloud...[sometimes, I spin up 50 servers for a project]"
	- Justifies using all MySQL, Memcache and Redis.
	- "Sharding is the #1 way we scale..[pre-shard so we can scale easily]...[earlier the better]"
	- Started with many dbs per physical server. master-master, but only write to some.  To scale, shift around where physically some of the dbs are located and update config.
	- Pin ID is structured with shardId,type,localId (localid is id within table)
	- They shard based on user join time, and maintain giant lookup structure for dbId to physical layout.
	- Using redis for: pyres (resque for python), followers, feeds, caching in memcache and redis, anti-spam
	- Truth in MySQL, redis as cache vs memcache because it allows paging.  use memcache for id:small blob
	- Starting to use redis persistently, so not sure how long mysql will be source of truth.
	- Stagger bgsave.. but if a slave comes down and comes back up and all try to reconnect, the master's procs will all have to bgsave to let them sync. Use aof + hourly (staggared) bgsaves.
	- use lua to avoid race conditions in updating list of followers.
	- Twitter's twemproxy is adding redis support - run one per f/e host to pool connections.
	- U: "Why not just redis instead of redis+memcache?" A:"We haven't explored that..[historical reasons]"
	- U: "I noticed you went from having Mongo to not having Mongo..[why'd you leave it?]" A:"I wasn't there at the time..[they just weren't having good experiences..hard to say if that was the product or our usage..wanted something we could get support on if needed..]"
	- U: "Any issues with lua scripting & expensive ops being slow and causing problems?" A: "Yes..there's a learning curve like everything.. but I think the team is very happy now with their decision.."
	- U: "What happens when master fails?" A:"[It's all manual, there is a page..] code push to promote slave to master...working to streamline process..use zookeeper [to avoid code push]"
	- Hourly backups of aof to S3
	- U:"How do you force staggared BGSAVES?" A: "Cronjob"

Antirez - Sentinel Solution
	- Sentinel is distributed monitoring so it is more reliable from Ops' point of view.
	- Notification system
	- Automatic failover, client config server
	- Monitoring: if not distributed, availability from one point of view.  We want to know if it is up from multiple POVs with an agreement protocol.
	- Image of gossip protocol: N:N pings
	- Demo using terminal. redis-sentinel IS redis-server (same sha), just detecting name. its a mode of execution
	- Sentinel conf is easy because slaves and other sentinels are auto-detected
	- You can run dozens of sentinels without issue if you have room for the processes. Set quorum as a fixnum in conf..
	- In sentinal mode, redis responds to ping, 'info sentinal', and 'sentinal' command. NO set or other keyspace commands
	- sdown vs odown: sdown is "down for me", owdown is once quorom is reached. "slaves don't require quorum because quorum is just used to promote masters"!
	- You can subscribe to status using pub/sub channels, or you can have scripts that are executed when event happens. Demo of 'subscribe *' + bringing down slave yielding an 'sdown' event, and subbing to jus 'sdown' for selective message subscription.
	- Automatic failover: requires client cooperation. odown detected, failover leader sentinal elected, slave selected, turned into master, other slaves repointed @ it. demo of it in-action. takes several seconds. uses timeouts to avoid race conditions.
	- Walkthrough of sentinals following promotion-master and the slave promotion, use of sentinal get-master-addr-by-name
	- Sentinal is 3k LOC. Uses redis codebase as framework; developed in few weeks instead of months. Planning to update sentinel with every point release of 2.6
	- Ruby, Python, Node.js clients already support it.
	- Redis 2.8 will support config through sentinal, so no need to say "slaveof" on startup.
	Dr Josiah Carlson
	- Redis user for 2 1/2 years. Extremely active on redis mailing list, #3 poster to the list (applause from PN)
	- Author of Redis In Action (available electronically)
	- What is search? scan text, use BM/BMG/KMP or regex or recursive-descent.. or suffix tries.. or you can search through the zip-format (BWT)
	- Inverted index: words to documents, instead of crawling documents for words.
	- SET per word with items for each docid, using redis' intersection for AND, union for OR
	- "Simplest example that will be useful..[but it wont be that useful... buy the book!]"
	- Demo of his editor.. its on sourceforge.
	- Tokenization, stop words in some python code.. "with this stuff, we're going to implement search" already has failing tests written.. live coding, so not many notes... buy his book.
	- "I have 5 minutes? I thought this was a 45-minute talk".. "Nope!"..."Shoot!"
	- "If you like intersecting sets and unioning them...you're welcome" (Josiah created them in fork at old company...)
	- "In the book, I cover fake-bayesian learning..[so it learns ads per word]"
	- "You can build an ad-targeting engine in 3-4 weeks and be making money...if you do, I accpet donations!"
	- User: "How do you do exact phrase search [if you use inverted index]?" A: "View content as non-stopword to non-stopword keys.. so you would store "lord" and 'rings' but also 'lord rings'" U: "Like trigrams?" A:"[Like fake trigrams, but more space-efficient]"

	Aren Sandersen
	- @Pinterest for 6 months, which is longer than 75% of the engineers. Pinterest is all scaling firedrill, use redis a lot.
	- Overview of pinterest. Compressed-air powered Nerf gun pic + lols.. tour of screens
	- Users can follow other users, or a subsection (board) of their posts, so graph is 'interesting'
	- "Been called the fastest growing website..[it's all about scaling]" shows hockey-stick growth to jan, flat til may.
	- 1 year ago: 4 Cassie, 15 membase, 10 memcache, 10 redis nodes, 2 engineers
	- May: 59 redis, 51 memcache, 66 mysql, 6 engineers, sharded solr
	- Nowish: over 200 redis, 50 engineers..
	- "Would not have been able to keep up without the cloud...[sometimes, I spin up 50 servers for a project]"
	- Justifies using all MySQL, Memcache and Redis.
	- "Sharding is the #1 way we scale..[pre-shard so we can scale easily]...[earlier the better]"
	- Started with many dbs per physical server. master-master, but only write to some. To scale, shift around where physically some of the dbs are located and update config.
	- Pin ID is structured with shardId,type,localId (localid is id within table)
	- They shard based on user join time, and maintain giant lookup structure for dbId to physical layout.
	- Using redis for: pyres (resque for python), followers, feeds, caching in memcache and redis, anti-spam
	- Truth in MySQL, redis as cache vs memcache because it allows paging. use memcache for id:small blob
	- Starting to use redis persistently, so not sure how long mysql will be source of truth.
	- Stagger bgsave.. but if a slave comes down and comes back up and all try to reconnect, the master's procs will all have to bgsave to let them sync. Use aof + hourly (staggared) bgsaves.
	- use lua to avoid race conditions in updating list of followers.
	- Twitter's twemproxy is adding redis support - run one per f/e host to pool connections.
	- U: "Why not just redis instead of redis+memcache?" A:"We haven't explored that..[historical reasons]"
	- U: "I noticed you went from having Mongo to not having Mongo..[why'd you leave it?]" A:"I wasn't there at the time..[they just weren't having good experiences..hard to say if that was the product or our usage..wanted something we could get support on if needed..]"
	- U: "Any issues with lua scripting & expensive ops being slow and causing problems?" A: "Yes..there's a learning curve like everything.. but I think the team is very happy now with their decision.."
	- U: "What happens when master fails?" A:"[It's all manual, there is a page..] code push to promote slave to master...working to streamline process..use zookeeper [to avoid code push]"
	- Hourly backups of aof to S3
	- U:"How do you force staggared BGSAVES?" A: "Cronjob"

	Antirez - Sentinel Solution
	- Sentinel is distributed monitoring so it is more reliable from Ops' point of view.
	- Notification system
	- Automatic failover, client config server
	- Monitoring: if not distributed, availability from one point of view. We want to know if it is up from multiple POVs with an agreement protocol.
	- Image of gossip protocol: N:N pings
	- Demo using terminal. redis-sentinel IS redis-server (same sha), just detecting name. its a mode of execution
	- Sentinel conf is easy because slaves and other sentinels are auto-detected
	- You can run dozens of sentinels without issue if you have room for the processes. Set quorum as a fixnum in conf..
	- In sentinal mode, redis responds to ping, 'info sentinal', and 'sentinal' command. NO set or other keyspace commands
	- sdown vs odown: sdown is "down for me", owdown is once quorom is reached. "slaves don't require quorum because quorum is just used to promote masters"!
	- You can subscribe to status using pub/sub channels, or you can have scripts that are executed when event happens. Demo of 'subscribe *' + bringing down slave yielding an 'sdown' event, and subbing to jus 'sdown' for selective message subscription.
	- Automatic failover: requires client cooperation. odown detected, failover leader sentinal elected, slave selected, turned into master, other slaves repointed @ it. demo of it in-action. takes several seconds. uses timeouts to avoid race conditions.
	- Walkthrough of sentinals following promotion-master and the slave promotion, use of sentinal get-master-addr-by-name
	- Sentinal is 3k LOC. Uses redis codebase as framework; developed in few weeks instead of months. Planning to update sentinel with every point release of 2.6
	- Ruby, Python, Node.js clients already support it.
	- Redis 2.8 will support config through sentinal, so no need to say "slaveof" on startup.