Some things that every application or service you build should have:
- Logging (preferably centralized)
- Metrics (to know how the app is performing (slow endpoints, etc.))
- Alerts (to be notified when things aren't working properly)
- Telemetry (to know which parts of the app / service are used most, or not at all and can be removed)
- User Feedback Mechanism (so users can easily report issues or request new features)
- Feature Flags (to enable rolling features / changes out in a controlled manner)
- Automated CI and CD for builds and deployments
- Unit Tests (ran on every build)
- Integration Tests (ran at least before every deploy, but on every build if possible, depending on how flaky they are)
- Load Tests (how much traffic can the app / service handle)
- Stress Tests (how much data can the app / service handle, or how long can it run for before breaking)
- Automatic Rollback Mechanism (so if a deployment fails or new version isn't working correctly it can easily be reverted)
A few extra things that we can do for web services:
- /Ping endpoint for services (to get back basic information about the service, such as version, environmental variable values, etc.)
- /Validate endpoint for services (that is called after deployments to have them excersise their own endpoints to ensure they get back expected responses)
- /Warm endpoint for services (that is called after deployments to warm the service before directing traffic to it)
- /Alert endpoint for services (that will send an email / Slack / etc. notification to easily test that they are working as expected)
- Availability monitoring and alerts for services (to know when service goes down and use as an SLI to meet our SLO)
Ideally documentation for the app/service should be stored in the source control repository itself, rather than in some separate website or wiki. If it is stored somewhere outside of the repository, you should at least have links to it in your repository documentation (e.g. in the ReadMe.md).
Things you should include in your documentation:
- Description of what the app/service does. Why does it exist?
- Proper documentation for your end users on how to use the software
- Proper documentation for developers who will be maintaining the code base in the future (high level architecture description / diagram, brief description of code patterns used, file structure layout, etc.)
- Runbook information (What monitoring and alerting is in place and links to it, description of common problems and how to resolve them, etc.)
- How to build and deploy the software
- How to contribute to the project (if there are any special requirements)