Skip to content

Instantly share code, notes, and snippets.

@jamsyoung

jamsyoung/mss-expansion-release-process.md Secret

Last active September 9, 2016 20:01
Show Gist options
  • Save jamsyoung/6af435ff4c42e41f1b1e to your computer and use it in GitHub Desktop.
Save jamsyoung/6af435ff4c42e41f1b1e to your computer and use it in GitHub Desktop.
Download ZIP
MSS Expansion Release Process for Development and Operational
Raw
mss-expansion-release-process.md

MSS Expansion Release Process

Version 1.2.1

Previous versions:

Table of Contents

Introduction

Building a release goes quick. If it takes you longer than 10 minutes you are doing it wrong. That said, reading this document and understanding everything will take a little longer.

Think of this process as a conveyor belt that is always moving forward. Sometimes it will stop briefly, but it never moves backwards.

Here is a high level diagram of this conveyor belt:

  specific   -> PRODUCTION artifact -> REF/STAGE -> PRODUCTION
approved PRs                              |
                                Test / Swarm on Issues

Specific approved PRs go into a PRODUCTION artifact that is deployed to REF and STAGE for testing. Once testing is completed on STAGE the PRODUCTION artifact is deployed to PRODUCTION. If any problems are found along the way the conveyor belt stops and resources swarm on fixing the problem to allow the process to start moving again.

The Tools

There are a variety of sources that are available to get the information needed to do a deployment. Here is a high level list of them:

Terminology

  • DEV, REF, TRAIN, ESAND, TRAIN, PROD: The names of the environments that code will be deployed to. They should always be referred to in all caps to indicate that you are referring to the environment name.

  • etrain: Etrain is not a real environment. It is the sub-domain on the url of the ESAND environment.

  • PRODUCTION: PRODUCTION refers to a grouping of production level environments. This is currently ESAND, TRAIN, and PROD.

  • artifact: This is a .tgz (tarball) that is created on a Codeship build agent and uploaded to S3.

  • PRODUCTION artifact: This is an artifact that is built from the master branch and does not have any tags in the version number. See Anatomy of an Artifact Filename for more details.

  • PRODUCT: Refers to the team of people that comprise all of the product and project managers that are working on the Expansion project.

  • RELEASE MANAGER: Refers to the person that is creating the release.

  • QA: Refers to the team of people that are testing the site.

Pull Requests

Approval

For a pull request to be approved, it must meet some requirements.

  • There must be no open tasks on the pull request.

  • There must be at least 2 approvals on the pull request.

  • One of the approvals must be from a Tech Lead or an Architect.

Once all of these requirements have been met, then a pull request is considered "APPROVED".

Merging

Pull requests are approved, but are NOT merged into develop on approval. They are only merged into develop when certain requirements are met.

  • When QA requests a new build, specific approved pull requests are merged in.

  • Work with PRODUCT to determine which specific tickets are eligible for the build. In some cases there will be approved PRs that are intentionally not going into the production build.

  • When merging a pull request, check if there is a corresponding pull request in Michonne or PAL. If there is, both pull requests must be approved before either one can be merged in.

  • If there are more approved pull requests than QA's capacity, they are merged based on order of priority. Check the ticket in Jira for the priority level.

  • When merging, merge from the bottom of the list up.

  • If there are conflicts on an approved pull request, the owner of the pull request should be notified in HipChat to resolve the conflict. If the conflict is not resolved in a reasonable time frame (10 minutes), it should be skipped.

Deployment

When an artifact is deployed to an environment, an email will be sent detailing the version of package, where it was deployed, and what tickets and pull requests are in the build. The email should look exactly like this:

DEV, REF, and STAGE email notification

To: ExpansionReleaseNotification@turner.com

Subject: Deployment to DEV, REF, STAGE


Deployed to REF

cnn-michonne-app-1.11.0-2387018
- Merged in feature/AVNGRS-1303 (pull request #1823)
- Merged in feature/AVNGRS-1300 (pull request #1822)
- Merged in feature/AVNGRS-1260 (pull request #1820)
- Merged in feature/AVNGRS-1272 (pull request #1821)
- Merged in feature/AVNGRS-1268 (pull request #1803)


pal-server-1.9.0-2387032
- Merged in feature/AVNGRS-1306 (pull request #953)

PRODUCTION email notification

To: ExpansionReleaseNotification@turner.com

Subject: Deployment to PRODUCTION at 9:30am ET 2/27/2015


Deployment to PRODUCTION

cnn-michonne-app-1.11.0-2387018
- Merged in feature/AVNGRS-1303 (pull request #1823)
- Merged in feature/AVNGRS-1300 (pull request #1822)
- Merged in feature/AVNGRS-1260 (pull request #1820)
- Merged in feature/AVNGRS-1272 (pull request #1821)
- Merged in feature/AVNGRS-1268 (pull request #1803)


pal-server-1.9.0-2387032
- Merged in feature/AVNGRS-1306 (pull request #953)

NOTE: The emails are slightly different between DEV, REF, STAGE and PRODUCTION. The email to DEV, REF, STAGE is a notification that a deployment has occurred. The email to PRODUCTION is a notification that a deployment will happen in the future. The PRODUCTION email notification may encompass multiple builds that went to DEV, REF, STAGE.

Jira Work

As tickets move through this process, their state in Jira must reflect the current state of the ticket.

  • When a developer is working on a ticket, Jira should show it as IN PROGRESS. The developer is responsible for setting the Jira status when work is started on the ticket.

  • When a developer creates a pull request, Jira should show it as POST DEVELOPMENT. This will show on the Rapid Board as IN PR. The developer is responsible for setting the Jira status when a pull request is created.

  • When the RELEASE MANAGER deploys a build to DEV, REF, and STAGE Jira should show all of the tickets in the build as TESTING/QA. This will show on the Rapid Board as IN QA. The RELEASE MANAGER is responsible for setting the Jira status when a build is deployed.

  • QA will test the build. As each ticket passes testing, Jira should show the ticket as PASSED QA. If a ticket fails QA, Jira should show the ticket as FAILED QA. QA is responsible for setting the Jira status when a ticket passes testing.

  • When the RELEASE MANAGER deploys a build to PRODUCTION, Jira should show all of the tickets in the build as CLOSED/OTHER with a comment of "RELEASED - PRODUCT-VERSION", like "RELEASED - pal-server-1.9.0-2387032". This will show on the Rapid Board as DONE. The RELEASE MANAGER is responsible for setting the Jira status when a build is deployed.

The Process, Step by Step

Step 1

QA will tell PRODUCT they are ready for another build to test.

Step 2

PRODUCT will tell the RELEASE MANAGER to create another build and what it should contain. This is not specific ticket numbers. This is general guidelines like: "Don't check in CENA or EPIC tickets yet, but everything else that is ready".

Step 3

The RELEASE MANGER will look at the approved pull requests in Bitbucket for Michonne and Pal and merge in the qualifying pull requests based on what PRODUCT has said should be deployed, what is on the rapid board, and what is within the capacity of QA to test in a single testing cycle.

If a ticket is not on the rapid board, it will not be merged in. PRODUCT should be made aware of any such tickets so they can decide what to do. If a decision is made to merge it in, then it needs to be added to the rapid board.

At this point, the RELEASE MANAGER should start composing the email that is discussed in the Deployment section of this document.

As each pull request is merged in, the following should be done:

  • Click the MERGE button on the pull request.

  • When the Bitbucket dialog box comes up, copy the first line of the commit message. This should look like: Merged in feature/AVNGRS-1323 (pull request #1839)

  • Paste this line into an email message in the appropriate place.

  • If this is NOT the last pull request being merged in for a specific product, add a blank line and --skip-ci to the bottom of the commit message. It should look like this:

    Merged in feature/AVNGRS-1374 (pull request #1854)

AVNGRS-1374: Remove 'live-bug' from muted player

--skip-ci

    You always want to build the last merged in pull request to verify that the develop branch is still building correctly after all of the pull requests have been merged in.

Step 4

Create the PRODUCTION artifact. See How To Create A Production Artifact section for more details.

Step 5

Wait on the builds to complete on Codeship. Once they have completed building go into the last build for Michonne and Pal that was built from the master branch and get the app name and full version out of the artifact file name. Refer to Anatomy of an Artifact Filename for more details.

Put this in the appropriate places in the email (subject and body) that you are creating.

Step 6

Now that you know what version needs to be deployed, ssh to admin.56m.dmtio.net and deploy the build to the DEV, REF, and STAGE environments. This can be done with the trigger script that is available from the deployit-cli-utils package that you will need to pull down into your home directory if you do not already have it on admin.56m.

To deploy to DEV, REF, STAGE

$ ssh admin.56m
$ cd deployit-cli-utils
$ ./trigger -w -n -e dev -e ref -e stage -p cnn-michonne-domestic -p cnn-michonne-international -p cnn-michonne-domestic-preview -p cnn-michonne-international-preview
$ ./trigger -w -n -e dev -e ref -e stage -p cnn-pal -p cnn-pal-preview

To deploy to PRODUCTION, see CNN.com Production Deployment Information

Step 7

Send the email, unless this is for PRODUCTION, which should be done at least 10 minutes prior to a deployment.

Step 8

Paste the contents of the email into the MSS Templates Dev HipChat room, using /code. Again, for PRODUCTION, do this at least 10 minutes before the deployment.

Step 9

QA will test the build that has been deployed to DEV, REF and STAGE. If any problems are found they must be fixed before the process can move forward. There is no "pulling a feature out", there is only moving forward. If there are problems that must be fixed, go back to Step 1 when a PR is approved to fix the problems that have clogged up the conveyor belt.

Step 10

Once QA says the build is good to move forward to PRODUCTION, deploy the artifact and the runtime configs to PRODUCTION. Refer to CNN.com Production Deployment Information for more details. Notify the DOC Incident Bridge Support room about 10 minutes or so prior to the PRODUCTION deployment, and send the email notification early as well, not after the fact.

How To Create a PRODUCTION Artifact

After merging in all the pull requests that need to be in the build:

Summary

$ git fetch -p
$ git checkout master
$ git pull origin master
$ git checkout develop
$ git pull origin develop
$ grep version package.json
  "version": "1.18.0",
$ git checkout -b release/1.19.0
$ vim package.json
$ git commit -am '1.19.0'
$ git checkout master
$ git merge --no-ff release/1.19.0
$ git push origin master
$ git tag 1.19.0
$ git push origin 1.19.0
$ git checkout develop
$ git merge --no-ff master
$ git push origin develop
$ git branch -d release/1.19.0

Detailed Breakdown

Step 1

Get latest in your working directory. You can optionally install git up to do this faster and in less commands.

$ git fetch -p
$ git checkout master
$ git pull origin master
$ git checkout develop
$ git pull origin develop

Step 2

Create a release branch for the next version. Bump the version number accordingly. It will not always be minor version bump.

$ grep version package.json
  "version": "1.18.0",
$ git checkout -b release/1.19.0

Step 3

Bump the version number on the reelase branch and check it in.

$ vim package.json
$ git commit -am '1.19.0'

Step 4

Merge the release into master and push it up. This creates the PRODUCTION artifact once Codeship completes the build.

$ git checkout master
$ git merge --no-ff release/1.19.0
$ git push origin master

Step 5

Tag the release and push up the tag.

$ git tag 1.19.0
$ git push origin 1.19.0

Step 6

Merge master into develop and push up.

$ git checkout develop
$ git merge --no-ff master
$ git push origin develop

Step 7

Delete the local release branch

$ git branch -d release/1.19.0

Anatomy of an Artifact Filename

This is here for historical knowledge. Michonne and Pal do not currently use qa tags on QA artifacts. They always use PRODUCTION artifats.

 cnn-michonne-app-1.18.0-2501113.qa.20141015155427.tar.gz
+----------------+------+-------+--+--------------+------+
     app name    |version   |   tag|  timestamp   extension
                 |    build number |
                 +-----------------+
                     full version

Changelog

1.2.1:

  • Updated a typo in Step 6
  • Removed a step about deploying runtime configs, which is handled in cnn-trinity now.

1.2.0:

1.1.0:

  • Removed In The Beginning section.
  • Updated documentation to remove references to creating QA artifacts.
  • Updated to match Markdown guidelines, spacing and line wrapping
  • Fixed typos
  • Updated email format

1.0.0-draft:

  • Working toward final 1.0.0 version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment