Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Running a docker private registry on EC2

tl;dr This blog post describes the best way to host private Docker Registry instance on Amazon EC2 and Amazon S3. You can skip the lyrics and go to the 'Resources' section to get the ECS Task Definition JSON.

When working with microservices nowadays you can hardly avoid using Docker. After building your first Docker container your next step woudl be to share it with the world (or your colleauge). To achieve it you have three alternatives:

  1. Use a Docker Hub or any other SaaS registry.
  2. Deploy your own instance of open source Docker Registry project (now called 'Distribution').
  3. Buy an enterprise version of the Docker registry which is based on open source docker registry project with some addons from Docker Inc.

For a small project the Docker Hub 'way' is the best alternative, you can host one 'private' image for free, where commercial pricing plans will allow you to upload more. For a more serious projects we would like to retain a control of our most precious asset - our code, so we wanted to run our own private instance of Docker Registry on Amazon EC2.

After a little googling we found a great article from Codeship guys (disclaimer: we are long time and happy users of Codeship CI). This blog post describes the process very well and up to the smallest details, however some of the aspects of 'production' deployment like Authentication and Encryption (aka HTTPS) are not described, and that's what we want to describe in this blog post.

Our goal is to give you a ready-to-use recipe for deploying your private instance of Docker Registry that is running on top of Amazon Elastic Container Service (which run on top of EC2) and uses Amazon S3 as storage.

Enable Amazon S3 and Authentication in registry config

Note: you don't acutally need to do it, just use a ready-made elasticio/docker-registry-ecs, no worries you can set all custom configuration properties (e.g. S3 access credentials) via environment variables later.

We will use an Amazon Container Service to run the trusted open source Docker container from Docker Inc. Docker Registry is distributed as a Docker container that is named registry:2. However we need to enable some of the features of the Docker Registry, e.g. S3 as a storage and authentication, so we need to customize the configuration - instead of default we need to place our changed config.yml in the image with following content:

version: 0.1
log:
    fields:
        service: registry
auth:
    htpasswd:
        realm: basic-realm
        path: /auth/htpasswd
http:
    addr: :5000
    headers:
        X-Content-Type-Options: [nosniff]
storage:
    cache:
        layerinfo: inmemory
    s3:
        accesskey: 
        secretkey: 
        region: set-via-env-vars
        bucket: set-via-env-vars
        encrypt: true
        secure: true
        v4auth: true
        chunksize: 5242880
        rootdirectory: /

The simplest way to place configuration in the Docker image is to build a new image with new configuration based on the original image. You can achieve it with following two lines in the Dockerfile:

FROM registry:2
COPY config.yml /etc/docker/registry/config.yml

The best part is actually that you don't have to do it as we prepared a ready-made image you can use with exactly content described above. You can find this image under elasticio/docker-registry-ecs name. Source of it you can find on Github.

Note that we don't have to enable HTTPS on the docker image itself - we'll use an HTTPS termination feature on Amazon ELB and save us in-container certificate deployment step.

Create a task definition in ECS

We'll run registry docker image on Amazon Elastic Container Service (that run on top of EC2). ECS run applications (called Tasks) inside the set of EC2 instances called Cluster. So first sep you need to do is to create a new cluster on ECS. It's easy to do via Amazon Console, just launch a sample cluster called 'default'.

After your cluster is up and running you need to upload a Task definition. Task definition is a JSON that defines which docker image to run, what resources it needs and environment variables and their values. Here you can see the tasks definition with one task:

{
  "containerDefinitions": [
    {
      "volumesFrom": [],
      "portMappings": [
        {
          "hostPort": 5000,
          "containerPort": 5000,
          "protocol": "tcp"
        }
      ],
      "command": [
        "/etc/docker/registry/config.yml"
      ],
      "environment": [
        {
          "name": "REGISTRY_AUTH_HTPASSWD_REALM",
          "value": "elastic.io private registry"
        },
        {
          "name": "REGISTRY_STORAGE_S3_ACCESSKEY",
          "value": "PLACE-YOUR-S3-ACCESSKEY-HERE"
        },
        {
          "name": "REGISTRY_STORAGE_S3_SECRETKEY",
          "value": "PLACE-YOUR-S3-SECRET-HERE"
        },
        {
          "name": "REGISTRY_STORAGE_S3_REGION",
          "value": "PLACE-YOUR-S3-BUCKET-REGION-HERE"
        },
        {
          "name": "REGISTRY_STORAGE_S3_BUCKET",
          "value": "PLACE-YOUR-S3-BUCKET-NAME-HERE"
        },
        {
          "name": "REGISTRY_AUTH_HTPASSWD_PATH",
          "value": "/auth/htpasswd"
        }
      ],
      "essential": true,
      "entryPoint": [],
      "links": [],
      "mountPoints": [
        {
          "containerPath": "/auth",
          "sourceVolume": "auth",
          "readOnly": true
        }
      ],
      "memory": 1000,
      "name": "registry",
      "cpu": 1024,
      "image": "elasticio/docker-registry-ecs:latest"
    }
  ],
  "volumes": [
    {
      "host": {
        "sourcePath": "/home/ec2-user/auth"
      },
      "name": "auth"
    }
  ],
  "family": "registry"
}

In your AWS ECS Console go to Task Definitions, then click on Create new Taks Definition and paste the JSON above to the JSON tab.

As you can see here you need to set following environment variables:

  • S3 Bucket name and region - this S3 bucket will be used by Registry to store. Set via REGISTRY_STORAGE_S3_BUCKET and REGISTRY_STORAGE_S3_REGION. Note registry support V4 authentication so it also works with S3 in Frankfurt (Datenschutz rocks!).
  • S3 Authentication credentials - we recommend to create a new user with restricted S3 permissions to the registry's bucket. Set via REGISTRY_STORAGE_S3_ACCESSKEY and REGISTRY_STORAGE_S3_SECRETKEY.
  • Authentication realm name - set via REGISTRY_AUTH_HTPASSWD_REALM

Create an HTPASSWD file

As a next step we need to configure users/passwords for HTTP Authentication that will be used for the registry. Unfortunately we haven't yet found the way to deploy it via environment variables therefore it's done via filesystem mount.

You would need to SSH into your cluster's node and create a new HTPASSWD. By default Security group

ssh -i ~/.ssh/your-key.pem ec2-user@your-cluster-node

Then you need to create a folder auth and create a HTPASSWD file inside it:

mkdir -p auth
cd auth
docker run --entrypoint htpasswd registry:2 -Bbn testuser testpassword >> htpasswd

The last line generate a username/password line and append it to htpasswd file.

Launch your service

Next step you would need to create a 'Service' from your ECS Task, just click on the Clusters, select your cluster and click on the Create new Service. Then select a task description and specify the number of instances of your task.

https://www.dropbox.com/s/obqznuji2uvqjs1/Screenshot%202015-09-22%2011.08.29.png?dl=0

Then launch the service and wait a minute or two until it will fully start:

https://www.dropbox.com/s/dqwuo8ec2tg3jwj/Screenshot%202015-09-22%2011.10.38.png?dl=0

Configure HTTPS on Load Balancer

So, now we have a docker registry running on Amazon ECS with HTTP Basic authentication to protect your knowhow, the only missing piece is encryption - enabling HTTPS. One possible way to achieve it is to enable HTTPS inside the Docker Registry image like described [here] (https://docs.docker.com/registry/deploying/#get-a-certificate) we however will use HTTPS Termination feature of the AWS Elastic Load Balancer.

Go to the EC2 console (from ECS console) and find a load balancer that was created together with your ECS cluster. Now you need to do two things:

  1. Change the listener's health checking settings to TCP:5000 - this is important so that LB knew when your registry instance is up and running. Originally configured to HTTP:5000/ it won't work as ELB expect HTTP 200 while it will be getting HTTP 404 from registry instead. https://www.dropbox.com/s/id4fnamknvdluup/Screenshot%202015-09-22%2018.08.14.png?dl=0
  2. Change listeners configuration, you need a single listener that listens on HTTPS:443 and forward requests to HTTP:5000 of the instance. Here you can deploy your HTTPS certificate. https://www.dropbox.com/s/tms45un2dqokjfs/Screenshot%202015-09-22%2018.08.41.png?dl=0

So, now you should be all set, your private docker registry will run on EC2 instance, store data to S3. Your registry will be protected by HTTP Basic Auth which is in combination with HTTPS (terminated on ELB within your AWS VPC) is a good way to protect your docker images.

Test your setup

We gave our registry an human readable name registry.elastic.io which is a CNAME alias to ELB DNS name. Now take the CURL and try to query your newly deployed docker registry.

curl -i https://registry.elastic.io/v2/

And you should see something like this:

HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Date: Tue, 22 Sep 2015 15:00:39 GMT
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Basic realm="elastic.io private registry"
Content-Length: 114
Connection: keep-alive

{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":null}]}

Now let's try basic authorization:

curl -i https://testuser:testpassword@registry.elastic.io/v2/

And you should see

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 22 Sep 2015 15:21:53 GMT
Docker-Distribution-Api-Version: registry/2.0
Content-Length: 2
Connection: keep-alive

{}~     

Now you can try to login with your with docker to your new registry:

docker login registry.elastic.io

Summary & Resources

So let me summarise what we did:

  1. We created an docker registry image with customized config.yml that could be found here
  2. We created a new Amazon ECS cluster
  3. We deployed a task description JSON with the configured docker registry image. Task definition could be found here
  4. In the JSON file we placed S3 credentials and S3 bucket name + region where Docker Registry will store artifacts
  5. We configured authentication settings file on the cluster node(s)
  6. We started cluster and updated the Load Balancer confiugration to terminate HTTPS

That's it - now you have a running private docker registry on your infrastructure hosted on Amazon Web Services.

TODOs

The setup described above is not ideal, due to lack of time following actions are skipped (feedback and change suggestions are welcome):

  • One could assign an AWS Role with associated S3 Write permission for S3 bucket. In this case we won't need to specify S3 credentials in the Task definition
  • Authentication could be made configurable via environment variables instead of complicated volume mounting
  • Cluster specific configuration and auto-scale group for transparently scaling registry
  • Caching into Redis or AWS-proprietary alternative
version: 0.1
log:
fields:
service: registry
auth:
htpasswd:
realm: basic-realm
path: /auth/htpasswd
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
storage:
cache:
layerinfo: inmemory
s3:
accesskey:
secretkey:
region: set-via-env-vars
bucket: set-via-env-vars
encrypt: true
secure: true
v4auth: true
chunksize: 5242880
rootdirectory: /
{
"containerDefinitions": [
{
"volumesFrom": [],
"portMappings": [
{
"hostPort": 5000,
"containerPort": 5000,
"protocol": "tcp"
}
],
"command": [
"/etc/docker/registry/config.yml"
],
"environment": [
{
"name": "REGISTRY_AUTH_HTPASSWD_REALM",
"value": "elastic.io private registry"
},
{
"name": "REGISTRY_STORAGE_S3_ACCESSKEY",
"value": "PLACE-YOUR-S3-ACCESSKEY-HERE"
},
{
"name": "REGISTRY_STORAGE_S3_SECRETKEY",
"value": "PLACE-YOUR-S3-SECRET-HERE"
},
{
"name": "REGISTRY_STORAGE_S3_REGION",
"value": "PLACE-YOUR-S3-BUCKET-REGION-HERE"
},
{
"name": "REGISTRY_STORAGE_S3_BUCKET",
"value": "PLACE-YOUR-S3-BUCKET-NAME-HERE"
},
{
"name": "REGISTRY_AUTH_HTPASSWD_PATH",
"value": "/auth/htpasswd"
}
],
"essential": true,
"entryPoint": [],
"links": [],
"mountPoints": [
{
"containerPath": "/auth",
"sourceVolume": "auth",
"readOnly": true
}
],
"memory": 1000,
"name": "registry",
"cpu": 1024,
"image": "elasticio/docker-registry-ecs:latest"
}
],
"volumes": [
{
"host": {
"sourcePath": "/home/ec2-user/auth"
},
"name": "auth"
}
],
"family": "registry"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.