saas_advice.md

SaaS Advice

1. Logs
   1. Logging Decisions
   2. Logging Stories
   3. Logging Negatives - hard to know if you're not finding it, or if the log entry doesn't exist
   4. Logging Storage
   5. Structured Logging
      1. Include a human readable "msg" which helps reading the flow of events - tells the story
      2. Keep the same naming for entities across your logging do not have pipeline_id, pipeline_uuid, linked_pipeline_id
2. Error Logging - Rollbar, Sentry
   1. Review and Squash - no broken windows!
   2. Capture as much context as possible when you are handling the error
   3. Failure to capture errors and logs leads to failing the customer twice, they've already had the issue that caused the error, don't let them down again by not capturing enough to fix it!
3. Cost of Support.
   1. Think about the cost of tickets
   2. Review them for small things that could reduce that cost
4. Circuit Breakers
   1. Things will fail, you will need to isolate the failure to avoid cascading errors
5. Rollback isn't trivial
6. Deploy then enable Feature Flag
7. Roll-forward
8. Automate Checks
   1. Automate them at the start otherwise you'll forget and new folks won't follow
9. Dashboards - separate rather than massive
10. Retrospectives - want a retrospective!
11. Transparency with Customers
12. Take time to upgrade if necessary - communicate to customers in advance
13. Playbooks
14. Productionisation of Services
    1. Must have Metrics, counts, failures etc.
    2. Alerts a human if it's down
    3. Alerts a human if error rates above specific thresholds
    4. Alerts a human if latency is high
15. Security
    1. Reviews
    2. Prioritise Security over Functionality
16. Configuration
    1. Twelve-Factor it, but centralise the configuration
    2. Don't arbitrarily pull from the environment inside the code otherwise it's harder to know what config needs to be set
17. Documentation
    1. Developers are terrible at it - we have too much retained knowledge
    2. The act of documenting something often helps to step back and see improvements