Upgrading circleci pipelines to authenticate with docker

1. Overview.md

On 1st November dockerhub (and underlying platform used by circleci) will be imposing rate-limits on all unauthenticated requests.

In order to avoid this making it impossible for us to build any of our projecs, we need to add authentication to every circleci config.

Below is an outline of how to apply these changes to a typical config file

2. Step by step.md

There are a lot of concepts in here, and it can be confusing. As there's a lot to get done in just 2 weeks, don't get too hung up on fully understanding what's going in - if it works, that's the main thing. If you're interested in getting a fuller understanding later, please leave a comment at teh bottom of this gist and maybe we'll run some training.

1. Assign dockerhub credentials to a reusable yaml block

Copy the following into your .circleci/config.yaml file. It can be anywhere, but near the top is advisable as it makes it easier to check if the project has already been done. This creates a docker-auth 'anchor' (which is like declaring a variable) which can be referenced using *docker-auth

docker-auth: &docker-auth
  auth:
    username: $DOCKERHUB_USERNAME
    password: $DOCKERHUB_ACCESS_TOKEN

2. Apply credentials to any docker images referenced in jobs/executors

Any job or executor which contains a docker property will need to reference the credentials block above. To reuse it use the following syntax to inject it.

executors:
  golang:
    docker:
      - image: circleci/whatever
        <<: *docker-auth
      
jobs:
  job1:
    docker:
      - image: circleci/whatever
        <<: *docker-auth

If your config.yaml is complex and contains lots of cross references, the docker statements may not all be within the jobs section of the file. You will need to add teh credentials reference to every place a docker image is referenced.

3. Use a context with credentials in it in your workflows

The environment variables referenced above - $DOCKERHUB_USERNAME & $DOCKERHUB_ACCESS_TOKEN can be set in your project's envirnment variables settings, but it's far easier to pull in the dockerhub-shared context that Reliability Engineering maintain.

For jobs that don't already reference a context add context: dockerhub-shared under teh job definition. For those that already do, convert the context property into an array and add dockerhub-shared.

workflows:
  version: 2
  workflow:
    jobs:
      - job1:
          context: dockerhub-shared
      - job2:
          context: 
            - dockerhub-shared
            - some-other-context-if-necessary

4. Orbs

Some jobs do not have a docker property defined, but rather define an executor. This normally means that the config imports an orb which defines an executor using some underlying docker image.

Passing in dockerhub credentials to these is, at this stage, still being worked on. For now it is at least worth following step 3 for all jobs, so that when we have a solution to passing credentials around, we at least already have them being injected at the top level.

If a non orb-based alternative exists for what your build is doing, consider using it instead.

3. config.yaml

version: 2.1

docker-auth: &docker-auth
  auth:
    username: $DOCKERHUB_USERNAME
    password: $DOCKERHUB_ACCESS_TOKEN

jobs:
  job1:
    docker:
      - image: circleci/whatever
        <<: *docker-auth
    steps:
      - checkout
      - ...
  job2:
    docker:
      - image: anyorg/whatever
        <<: *docker-auth
    steps:
      - checkout
      - ...
      
workflows:
  version: 2
  workflow:
    jobs:
      - job1:
          context: dockerhub-shared
      - job2:
          context: 
            - dockerhub-shared
            - some-other-context-if-necessary