- This is a blameless Post Mortem.
- We will not focus on the past events as they pertain to "could've" "should've"...
- All follow up action items will be assigned to a team/individual before the end of the meeting. If the item is not going to be top priority leaving the meeting, don't make it a follow up item.
The CLA handling, including Curry Bot, was missing from the UI and not labeling pull requests.
(All times UTC)
- 20:25: Bryan McLellan(btm) noticies an issue and tries to raise it in the gitter chat for supermarket.
- 20:33: btm reaches out to Christopher Webber(cwebber) to notify that CLA stuff is missing
- 20:36: Discussion begins in Operations Channel about the missing CLA stuff.
- 20:40: Decision is made to escalate as incident
- 20:41: cwebber assumes role as IC
- 20:42: Incident is announced
- 20:42: Nathen Smith(nlsmith) begins looking at features that are enabled and disabled
- 20:44: Support is notified
- 20:48: nlsmith confirms that .env is correct
- 20:49: nlsmith confirms that the rails console shows the features are enabled
- 20:52: Mark Harrison (mh) Begins verifying that there wasnt a redis issue
- 20:56: There was an supermarket-app-test role in play. Ian Garrison (ian) works to verify it isn't part of the issue.
- 20:57: mh verifies that there have been no cache events
- 21:01: nlsmith leaves for another meeting.
- 21:07: mh verifies that the prod logs look fine
- 21:13: Josh Black (jblack) confirms issue is with github not being enabled in the features
- 21:14: jblack restores Contributors tab in prod
- 21:15: Incident resolved
- 21:21: Status site updated that incident is resolved
The github feature was not in the enabled state causing the contributors tab to go away.
- rails console was used to manually enable the feature
- production vault item was updated to add the feature to the enabled features
Contributors Tab, CCLA/ICLA signups, and CLA verification of PRs was down from 2014-11-06 23:30 until 2014-11-10 21:15.
- Deploy more frequently - jblack
- Document all the available feature flags - cwebber
- Document how to find the last deploy - cwebber
- Add check to verify contributors page returns 200 - cwebber/mh
- Add info to /status with enabled features - jblack