Skip to content

Instantly share code, notes, and snippets.

@btm
Last active August 29, 2015 14:17
Show Gist options
  • Save btm/641d3b0ec331ac34fbe9 to your computer and use it in GitHub Desktop.
Save btm/641d3b0ec331ac34fbe9 to your computer and use it in GitHub Desktop.
Chef Client 12.1.0 release post-mortem

Post-mortem Report

Post-mortem Facilitator: Bryan McLellan

  1. Failure Happens. Mistakes Happen. This is a blameless Post Mortem.
  2. We assume everyone participating in the project wants to do a good job.
  3. We will not focus on the past events as they pertain to "could've", "should've", etc.
  4. All follow up action items must be actionable and have a participating individuals commitment to implement, design, or otherwise achieve them.

Description

Significant number of regressions in 12.1.0 release, requiring unexpected patches, tests, and releases.

Timeline

  • 2015-02-21 Chef Client 12.1.0.rc.0 released
  • 2015-03-03 Chef Client 12.1.0 released
  • 2015-03-03 chef/chef#3008 filed: yum source bug
  • 2015-03-03 chef/chef#3009 filed: strings sent to run_action don't work
  • 2015-03-03 chef/chef#3011 filed: easy_install broke due missing command mixin
  • 2015-03-03 chef/chef#3018 filed: knife search -a output wrong
  • 2015-03-04 chef/chef#3022 filed: homebrew provider broken
  • 2015-03-05 chef/chef#3030 filed: yum version bug
  • 2015-03-09 Chef Client 12.1.1 released, fixed #3008, #3011, #3018
  • 2015-03-11 chef/chef#3059 filed: different yum source bug?
  • 2015-03-17 chef/chef#3096 filed: openbsd package bug
  • 2015-03-18 chef/chef#3108 filed: aix source bug
  • 2015-03-18 chef/chef#3105 filed: easy_install source bug

Contributing Factor(s)

  • Lack of automated integration testing of providers across a broad range of platforms with visibility to all Maintainers
  • Insufficient testing of release candidates by users in complex environments. Even if ChefInc runs it, we're primarily an Ubuntu shop.
  • Unsupported platforms aren't going to tested even with unicorn-ci.
  • Approximately three months between releases created a large set of changes, increasing risk of regression, and decreasing ability to complete directed testing of affected areas.

Stabilization Steps

  • Some testing happened well before release, for example we found #2914 on 2015-02-14, where we discovered that apt virtual packages where broken.
  • Bugfix releases

Impact

  • Many users were unable to install packages using the most recent Chef client release. Package is part of the "holy trinity" of Chef, the resources that are required for platform support
  • Some community cookbooks were affected by the changes as well, e.g. rbenv, homebrew, jenkins

Corrective Actions

  • Find test-kitchen patch that allows checkout of a git sha, merge and release - Fletcher Nichol (Internal Ticket)
  • Execute the existing Ubuntu test-kitchen tests internally at ChefInc using build-kite - Seth Chisamore, Fletcher Nichol, Bryan McLellan
  • Organize CI Discussion at Open Space at ChefConf - Fletcher Nichol, Seth Chisamore, Jay Mundrawala
@charlesjohnson
Copy link

We didn't get to #3040 in the meeting today (That's fine, had plenty going on to fill the hour), but I do want to make sure it gets followup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment