- Roger's Innovation Adoption Curve - Same for open source and normal products
- Nested Russian Doll Cache
- Stack
- Ruby 1.9.3-p385
- Rails 3.2
- Percona MySQL 5.5
- Unicorn 4.5
- Memcached 1.4.14
- Redis 2.6
- To Scale:
- Know the system and the constraints in it
- Avoid network calls during req
- speed up unavoidable network calls
- the storefront and checkout
- the chive
- Constraints
- One Request, One Process
- RPM = Workers * 60 / Response Time
- increase workers, reduce response time
- Measure everything, every piece of data
- New Relic (help you narrow down where the problems are so you know where to scale)
- Splunk (generate graphs for historical purposes)
- StatsD
- Cacti
- Conan
- Caching
- cacheable
- identity cache
- Resque
- Put payment process in the background
- Redis
Used for:
- Inventory reservation system
- Sessions
- Theme upload
- Throttling
- Carts
- after_commit
- Services
- Adapt and Evolve as Needed
- Demographics
- US: Mostly female
- 20-30
- Bachelors degrees
- middle class people
- RTurk, Ruby-aws, Turkee
- Display of HIT
- Soliciting feedback
- Skip deployment entirely
- forward gem
- Make appeal to Turker to be a personal appeal
- Ensure that you invite Turkers to give negative feedback
- A/B Testing
- Vanity gem
- Intent is hard to test with MTurk
- Paid workers, not customers interested in product
- Interaction can be measured
- SPAM
- Gold standard question: Four plus five equals?
- Even better; ask a validating question about their submission e.g. How many words in your second sentence?
- This causes the Turker to slow down and re-read their work. Also serves as validation.
- Correct mental modal
- Measure business value
- Compare with historical data
- If it is important for your business, measure it!
- Performance is business value.
- only worry about clusters around middle
- if we focus on the middle, lost a large chunk of data
- web responses are long tail
- You want consistent performance.
- pick a tolerance and optimize to the right
- Software architecture - most important thing is abstraction
- network-based application software
- monorail <---> cache <---> database
- Object Oriented Design
- State
- Behavior
- Objects interacting with other object but cannot manipulate state (fully encapsulated) but can modify behavior
- Single responsibility
- do the smallest thing possible and nothing more
- do one thing and do it right
- make things as simple as possible
- etc
- DRY
- depend on behavior not on data
- Law of Demeter
- Cache is King
- network-based software applications mean more overhead
- serialization is expensive
- when you don't have to, cache-control and etag headers good
- Classes are nouns and messages are often verbs
- GitHub hypermedia API - approaching a behavioral API
- HATEOS
- The good: link relations are already appearing in media types
- The bad: you can implement custom media types
- The ugly: the necessary HTTP verb is a missing
- NLP: Analyzing, understanding and generating the language that humans use to interface with computers.
- in short, searching, extracting and grouping information
- no perfect solution exists
- language itself is a moving target
- many aspects are computationally complex
- demand/need is increasing steeply
- 3 common approaches:
- Rule-Based Analysis
- Statistical Analysis
- Machine Learning
- Basic Building Blocks
- POS tagging
- Word Stemming
- Chunking
- Tokenizing
- Sentence Detection
- Word Relationships
- Co-Reference Resolution
- Named-Entity Recognition
- Tools & Libraries
- NLP Toolkit (Python)
- [Chronic](https://github.com/mojombo/chronic) (date parser)
- [Linguistics](https://github.com/ged/linguistics)
- [Punkt Segmenter](https://github.com/lfcipriani/punkt-segmenter)
- [Ruby Stemmer](https://github.com/aurelian/ruby-stemmer)
- [Treat](https://github.com/louismullie/treat)
- What happens when you need more?
- JRuby
You've got a Sinatra on your Rails
- historically, rails avoid js altogether
- js - avoiding complex MVC structures
- must think about where the heart of your application lives
- you can make things more complex, you can't make them less complex
- simplicity is being able to deal with components separately
- simplicity is using simple structures
- simplicity is consistency
- server-side has three distinct output modes
- client side has generally one mode of communication but business logic can be duplicated
- it's not clear that one side or the other requires less code
- both sides claim speed
- server side interaction is proportional to number of clicks
- client side interaction is proportional to new data requirements
- serverside is optimized for least disruption to Rails
- Questions to ask:
- client style is optimized for simplicity of server logic
- is the representation of your data stable?
- is interaction with the server primarily notification?
- do client components interact?
- what do you already have?
- will a hybrid approach work?
- Tests are:
- slow
- fragile
- Just delete some tests
- Unit tests: Goals
- thorough
- stable
- fast
- few
- Focus on messages
- object under test receive messages from outside and send outgoing messages
- cannot see anything outside
- Message Origin
- incoming
- sent to self
- outgoing
- Types of messges
- query: return something/change nothing
- command: return nothing/change something
- we conflate commands and quieries at our peril but we do it all the time
- message origin x type
- incoming-query:
- test incoming query messages by making assertions about what they send back
- test the interface (not the implementation)
- incoming-command:
- test incoming command message by testing direct public side effects
- sent to self-query: Ignore
- sent to self-command: Ignore
- outgoing-query:
- do not test
- do not make assertions about their result
- do not expect to send them
- Ignore
- outgoing-command:
- expect to send outgoing messages
- Summary
- be a minimalist
- test everything once
- test the interface
- trust collaborators
- insist on simplicity
- template method pattern
- Every API has two users, developer and computer
- George Orwell - Writing
- API extends our language
- Orwell's 6 rules of writing:
1. Never use a metaphor, simile or other figure of speech which you are used to seeing in print.
2. Never use a long word where a short one will do.
3. If it is possible to cut a word out, always cut it out.
4. Never use the passive where you can use the active.
5. Never use a foreign phrase, a scientific word or a jargon word if you can think of an everyday English equivalent.
6. Break any of these rules sooner than say anything barbarous.
- Dieter Rams - Industrial Design
- Ten Principles - Good Design is...:
1. Is innovative
2. Makes a product useful - useful errors
3. Is aesthetic
4. Makes a product understandable - show exact error
5. Is unobstrusive - disappear when not needed
6. Is honest - give full insight into system
7. Is long-lasting - versioning API, UUID keys
8. Is through down to the last detail
9. Is environmentally friendly
10. Is as little design as possible
- Noriaki Kano - Customer satisfaction
- Kano Model:
1. Attractive Quality - Delighters, good support
2. One-Dimensional Quality - Performance needs
3. Must-be Quality
4. Indifferent Quality
5. Reverse Quality
- Five guiding principles
1. Minimalism
2. Get out of the way
- Consistency
3. Design for extremes
4. Be predictable - standard verb names
5. Excite users
- Focus on delight and performance
- Stack: mongoDB, heroku, rails, twilio, sendgrid
- results
- notifications in quiet spaces
- personally configurable notifications
- mapping data for accurate response
- textual data for recall
- full notes for decision support
- firefighters are losing pages and forgetting how to read maps
- Two different kinds of downtime:
1. Us - internal, heroku down etc
2. Them - other people, email gateway, external database, caching provider, payment gateway, no control over etc.
- What if our payment gateway goes down?
- initially, if down, curstomer turn away == sad customer
- first solution, manual shutdown - "failover mode"
- accept low risk orders
- store them and charge customer later
- risk assertion (< 100 bucks?)
- Pros: Customers can make purchase, no lost orders
- Cons: requires a human all the time, humans do not stay up 24/7
- second solution, automation
- timeout & accept
- wrap a charge in a timeout
- if it times out, evaluate risk
- if low risk, save it and return success
- cron task to retry timed-out orders
- but race condition... no locks
- Pros: no humans required, developers can get some sleep
- Cons: not really, it worked well for quite a while, very rarely might mistakenly find the wrong order
- What about when WE are down?
- Reasons:
- Application error
- Heroku failing
- AWS went away
- Always use CDN to serve your static pages
- What if Heroku/AWS goes down for us?
- "Chocolate" - Request Replayer
- Dynamic failover service - powered by Akamai
- How we layer the stack
- Internet -> Akamai Dynamic Router -> Rails 3 Application (Heroku)
-> Chocolate
-> Akamai CDN
- Chocolate
- Separate Sinatra Application
- Perform Risk Assertion
- Store raw request in DB
- "Replay request" back to production
- Completely separate
- Sinatra app
- deployed to a separate cloud (not AWS)
- Same risk as before
- If an order accepted that can't be charged, still on the hook
- support team has to follow up with customers to keep lost $$ low
- Single POST endpoint to save an Order into the database
- If order looks real
- calculate risk
- if low, saves everything: params, headers etc to DB
- returns a response that looks identical to
a production response
- What is replay? (Replaying orders)
- when we're back up
- Order method on chocolate has a replay method
- Manual process run by support team to track results (and follow up if necessary)
- De-duping
- Could be a case where an order is in chocolate and in production
- Don't want to double-charge the customer
- Need to de-dupe
- Akamai injects a unique requiest ID for every order we create
- Store this on each order in prodution and on replays in chocolate
- Chocolate sends this as part of a replay
- Triggering (When to de-dupe)
- Akamai has a rule that if a POST to our order #create
endpoint takes > 15s, retry the exact same request on
chocolate
- sometimes production will actually succeed, but not
a problem: chocolate de-dupes
- Pros of using something like Akamai
- allows you to auto-replay to separate endpoints
- if done correctly, your site will never appear to be down
- Cons of using something like Akamai
- adds a fairly significant layer of complexity
- adds non-trivial costs
- Even though site is up... Stil see orders fail over to chocolate..
- Random routing...
- Dynos get backed up
- every day, a handful of orders still end up failing
over to chocolate
- Solutions
- make all endpoint fast to free up dynos quickly
- keep tuning unicorn and failover timeouts
- no guaranteed way to solve this
- Things to remember
1. Your site will go down
2. Use a replayer for critical web requests
3. Accept some risk to keep customers happy
4. Keep your endpoints lean and fast
5. Use a CDN
- What is configuration?
- config/database.yml
- config/feature_flags.yml
- config/locales/es.yml
- app/models/post.rb
- app/models/quest.rb?????
- config/env/production.rb
- everything you want it to be - values that you may want to share or change
- Why do we care about configuration so much?
- game company - set the game world
- When pushing configurations
- technical QA
- playtesting QA
- deploying new configs just as rigorously as new code
- bad config can be just as damaging as bad code
- Configuration context
- composite configuration - cascading
- data centers
- AWS regions
- roles
- machine names
- tags
- rails_config
- Baking configurations
- Requirements
- decouple generation process entirely form app/repo
- use code deployment principles
- build artifact
- version and store
- deploy
- Dump directly from DB?
- slow
- dogpile
- harder to replicate than static http asset
- unnecessary load on your most precious resource
- may be ok for your use case
- Build process
- transform config values into a single YAML doc - building an
artifact
- take note of: builder, build num, timestamp, changelog
- include checksum to prevent alteration
- Promotion
- map a revision to an environment or other dimension
- promotion can be end-user accessible
- store YAML files on a remote disk somewhere
- Deploying configurations
- run test/CI suites against config
- run QA against config
- run playtest against config
- deploy to production
- Don't want to restart
- spawn a thread to poll for updates in background
- Celluloid timer is perfect for this
- doesn't work with GIL (MRI)
- pretty gross multi-process (Unicorn)
- Don't forget to be threadsafe
- Why poll and not push?
- what happens if things go wrong?
- failed push means app carries on with old config
- responsibility in the wrong place
- if poll fails, app can shut itself down
- Applications
- rapid A/B testing
- I18n and translation management
- feature flags
- Bonus non-Ruby content
- Archaius: Netflix's Java composite configuration library
- ZooKeeper: Cluster synchronization service
- jones: Python ZK-based config app
- Further exploration
- show don't tell
- body language - hands outwards
- you have to be more interesting than the internet
- power poses are good - fake it till you make it
- good information in passionate delivery - be passionate about your talk
- make sure people can read - don't say i know you can't see this
- don't ask filler questions
- hands behind back, listening pose, move forward when the question is asked
- don't talk first on the last day
- don't ever give talks in a giant room - small room
- make them sit as close as possible
- can use puts "============" like a tracer round
- Kernal#caller - gives you back trace
- User.last.method(:github_url).source_location -> ["path", line_number]
- Module#ancestors - tells you all the modules included in the mod
- Module#method_defined?(:upcase) - tells if the method is defined or not
- Method#instance_method
- Object#method
- Module#instance_methods