2014-11-19 - `Chef` repo force pushed to master

force_push.md

19 Nov 2014 - Chef repo force pushed to master

Start every PM stating the following

1. This is a blameless Post Mortem.
2. We will not focus on the past events as they pertain to "could've", "should've", etc.
3. All follow up action items will be assigned to a team/individual before the end of the meeting. If the item is not going to be top priority leaving the meeting, don't make it a follow up item.

Incident Leader: Tyler Ball

Description

In the Chef Repo a branch was accidentally force pushed to master, causing existing pull requests to include unnecessary diffs. We discovered the issue, rolled the git tree back to the correct state, and applied the commits which were originally meant to be included.

Timeline

Times in PST on November 19th, 2014

• 8:57 - Force push occurred and deleted 6 commits from the master branch of Chef
• 12:00-13:00 - During office hours Bryan McLellan and Tyler Ball started seeing pull requests including the same 6 commits which were not originally part of the code review.
• 13:00-15:00 - Jay Mundrawala helped investigate and discovered the cause of the issue - a force push. Mark Harrison restored the correct HEAD of the tree to a new branch and Jay force pushed this over the bad master. They then cherry picked the commits where were originally force-pushed.
• ~15:00 - The issue was resolved.

Contributing Factor(s)

A force push can deleted or change SHAs for existing commits in a branch. This is fine in a feature branch but can cause issues like the above in a master or stable branch. Old versions of git had a bug where issuing git push --force in a feature branch would ALSO push the master branch.

Stabilization Steps

The branch commits were restored following the steps in the Timeline above.

Impact

All branches created between November 14th and the time the issue were resolved suddenly had 'additional' commits in them when the force push occurred on the morning of the 19th. These looked like extra commits but were really just the commits which had been deleted out of master. This caused pull requests to have an extra 6 unrelated commits in them, causing confusion during code reviews.

This issue has occured before, which prompted the creation of a post mortem.

Corrective Actions

• Pull requests had comments added which explained the issue and an email was sent to the developers group asking people to resync/rebase their branches.
• (Tyler) Contact github support - Definitely disable for master, we would also like to disable for stable branches (on principle) but this requires more maintenance because we change stable branch names.
  • If we can do this, also prevent force pushes in other Chef managed repos.

If Github support cannot disable force pushes, we will try:

• (Serdar) Notifications - Github supports webhooks and service hooks so we can setup one that notifies one us when a force push occurs to master or a stable branch. Research and roll this out on the Chef repo.
  • Apply this same webhook to all repos after we test and apply it to the Chef repo.
  • Pagerduty notification to the client goalie so they can run point on reverting it.
    • Does pagerduty support different levels of priority, so this alert doesn't wake anyone up in the middle of the night?
  • Email to the team responsible for the repo and anyone with commit access.
  • Priority - fix this issue ASAP during the next available work hours.
  • Documentation for rolling back branch to a safe commit (https://github.com/mivok/tools/blob/master/recover_github_commit.sh).

I'm curious, what ever came from this?