Skip to content

Instantly share code, notes, and snippets.

@cpanic

cpanic/ubuntu-src-vbox-vm-docker-registry.md

Last active November 4, 2016 14:27
Show Gist options
  • Save cpanic/9163fd43921ee13496b03db5690890cd to your computer and use it in GitHub Desktop.
Save cpanic/9163fd43921ee13496b03db5690890cd to your computer and use it in GitHub Desktop.
Download ZIP
Ubuntu Server 16.04 LTS VirtualBox VM as a Docker Secure Private Registry
Raw
ubuntu-src-vbox-vm-docker-registry.md
Ubuntu Server 16.04 LTS VirtualBox VM

Docker Secure Private Registry

Introduction

I thought it would be a good idea to try my hand at deploying a secure private Docker registry into my development lab. This would make working with my private Kubernetes cluster possible, without having to use the Google Container Registry (gcr.io) or the Docker Hub.

I wrote the document for myself, as a means to replicate my endeavour in the future. It wasn't a simple task, and involved the usual traipsing around the Internet in search of solutions to niggling problems. Sure, the information is out there, but it isn't necessarily in one place, and there are a ton of assumptions about skill level, especially with regard to certificate management.

So, here it is — hopefully it will help someone else out there.

Prerequisites

You will need the following:

Install Docker Engine and Docker Compose

If you haven't done so already, install the required Docker Components.

  1. Start your VM and log into an account with superuser privileges:

    $ sudo su
# apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 \
    --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
# echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" \
    > /etc/apt/sources.list.d/docker.list
# apt update
# apt install -y docker-engine docker-compose
# systemctl start docker
# systemctl enable docker
# exit

  2. Add the administrator to the docker group so that sudo is not required when using the docker command:

    $ sudo usermod -aG docker $(whoami)

  3. Log out and log in again for this last change to become effective.

Adjust Memory and Swap Accounting

If you haven't done so before, you might wish to make a small change to the GRUB bootloader configuration, beneficial on Docker hosts in preventing users from seeing the following warning:

no swap limit support

You should be logged into your administrator account as superuser (sudo su):

  1. Open the /etc/default/grub file with your favourite editor

  2. Set the GRUB_CMDLINE_LINUX value:

    GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

  3. Save and close the file

  4. Update GRUB and restart your system:

    # update-grub
# reboot

Create the docker registry

Create and install the root certificate authority (CA) certificate

If you haven't done so already, start your VM and log into the administrator account.

  1. Generate the root CA key:

    $ mkdir -p ~/registry/certs
$ cd ~/registry/certs
$ openssl genrsa -aes256 -out ca.key 4096
    Generating RSA private key, 4096 bit long modulus
........................................................................................++
.............................................................................................
.............................................................................................
.................................................................++
e is 65537 (0x10001)
Enter pass phrase for ca.key:
Verifying - Enter pass phrase for ca.key:

  2. Using the root CA key, generate the self-signed root CA certificate:

    $ openssl req -new -x509 -days 3650 -key ca.key -sha256 -out ca.pem
    Enter pass phrase for ca.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:UK
State or Province Name (full name) [Some-State]:Hampshire
Locality Name (eg, city) []:Southampton
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Your Name
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:ubuntu01.local
Email Address []:your.email@address.com

    On my network, this machine will be referenced as ubuntu01.local, which is the name I used as the Common Name (e.g. server FQDN or YOUR name). Be sure to supply the correct fully-qualified domain name (FQDN) when generating your own CA certificate.

  3. Install the root CA certficate into the host's /etc/ssl/certs:

    $ sudo cp ca.pem /usr/local/share/ca-certificates/ca.crt
$ sudo update-ca-certificates
    Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

    Beware the target of the cp command has a .crt file extension.

  4. Install the root CA certificate onto any intermediate hosts running a Docker daemon that will use your registry:

    • Using sudo, create the /etc/docker/certs.d directory, if it isn't already there.

    • If you were planning to host the registry at ubuntu01.local on port 5000 (as I am doing here), use sudo to create a /etc/docker/certs.d/ubuntu.local:5000 directory.

    • Using sudo and scp, copy the ca.pem file you created above into this directory as ca.crt.

    • If you ar using /etc/hosts rather than DNS to resolve the address of your registry's machine, you may need to update the etc/hosts file on your docker host, too.

    • You may also need to modify /etc/resolv.conf by adding your DNS (or Gateway if it's your proxy DNS) address to the nameserver record. This may prevent no such host errors later on. I had to do this for my minikube cluster.

    This is an especially important step to follow if your architecture includes a boot2docker host, such as Docker Toolbox for Mac, Docker for Windows and Minikube.

    You may need to perform these tasks from within your docker host using a docker-machine ssh or minikube ssh command to access a superuser-capable account.

    If you are using this type of architecture and fail to to this then your docker machine will have no way to verify your registry certificate's chain of trust, and you will start to see x509: certificate signed by unknown authority error messages.

Create the server certificate

  1. Generate the server key:

    $ openssl genrsa -out ubuntu01.local.key 4096
    Generating RSA private key, 4096 bit long modulus
...........++
..........................++
e is 65537 (0x10001)
Enter pass phrase for ubuntu01.local.key:
Verifying - Enter pass phrase for ubuntu01.local.key:

    Remember to substitute your server details for ubuntu01.local.

    Beware unlike when we generated the root CA key, we do not specify a cipher (e.g. -aes256) for the server key because it must not be encrypted, so do not add one here thinking it was accidentally omitted. Be especially careful if you recalled and modified the previous command, or you could eventually be seeing warnings about unparseable server keys.

  2. Using the server key, generate a certificate signing request (CSR):

    $ openssl req -subj "/CN=ubuntu01.local" -sha256 -new -key ubuntu01.local.key -out ubuntu01.local.csr
    Enter pass phrase for ubuntu01.local.key:

    Supply the same passphrase you gave when generating the server key. Again, remember to substitute your server details for ubuntu01.local.

  3. Create an extensions file containing the server certificate subjectAltName record:

    $ echo subjectAltName = DNS:ubuntu01.local,DNS:ubuntu01,\
    $(ifconfig | grep 'inet addr' | cut -d ':' -f2 | cut -d ' ' -f1 | sed -e 's/^/IP:/' | paste -sd ',' -)\
    | tee > extfile.cnf
    subjectAltName = DNS:ubuntu01.local,DNS:ubuntu01,IP:172.17.0.1,IP:192.168.1.115,IP:127.0.0.1

    It is possible, even likely, that TLS connections will be made using the server's fully-qualified domain name (FQDN), its hostname, or any of the IP addresses at which it may be reached (i.e. via the ethernet, local loopback or docker host interfaces).

    Unless you are certain that only the server FQDN will be used, you should create an extensions file containing this additional information.

  4. Generate the signed server certificate using our self-signed root CA:

    $ openssl x509 -req -days 3650 -sha256 -in ubuntu01.local.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
    -out ubuntu01.local.pem -extfile extfile.cnf
    Signature ok
subject=/CN=ubuntu01.local
Getting CA Private Key
Enter pass phrase for ca.key:

    Supply the same passphrase used when generating the root CA key. And again, remember to substitute your server details for ubuntu01.local.

Remove CSRs, secure private keys and check the server certificate

  1. Check your work:

    $ ls -l
    total 28
-rw-rw-r-- 1 administrator administrator 3326 Nov  2 15:07 ca.key
-rw-rw-r-- 1 administrator administrator 2114 Nov  2 15:11 ca.pem
-rw-rw-r-- 1 administrator administrator   17 Nov  2 16:31 ca.srl
-rw-rw-r-- 1 administrator administrator   74 Nov  2 16:26 extfile.cnf
-rw-rw-r-- 1 administrator administrator 1590 Nov  2 15:31 ubuntu01.local.csr
-rw-rw-r-- 1 administrator administrator 3326 Nov  2 15:28 ubuntu01.local.key
-rw-rw-r-- 1 administrator administrator 1903 Nov  2 16:31 ubuntu01.local.pem

    All being well, you should see output resembling that shown above.

  2. You may now safely remove any certificate signing request .csr files:

    $ rm *.csr

  3. You should secure private key .key files by removing world read privileges:

    $ chmod o-r *.key
$ ls -l
    total 24
-rw-rw---- 1 administrator administrator 3326 Nov  2 15:07 ca.key
-rw-rw-r-- 1 administrator administrator 2114 Nov  2 15:11 ca.pem
-rw-rw-r-- 1 administrator administrator   17 Nov  2 16:31 ca.srl
-rw-rw-r-- 1 administrator administrator   74 Nov  2 16:26 extfile.cnf
-rw-rw---- 1 administrator administrator 3326 Nov  2 15:28 ubuntu01.local.key
-rw-rw-r-- 1 administrator administrator 1903 Nov  2 16:31 ubuntu01.local.pem

  4. You may also wish to check your server certificate:

    $ openssl x509 -in ubuntu01.local.pem -text -noout
    Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12371841798107880252 (0xabb19c91098cdb3c)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=UK, ST=Hampshire, L=Southampton, O=Your Name, CN=ubuntu01.local/emailAddress=your.email@address.com
        Validity
            Not Before: Nov  2 16:31:53 2016 GMT
            Not After : Oct 31 16:31:53 2026 GMT
        Subject: CN=ubuntu01.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:bd:db:49:30:a5:95:aa:5c:e9:5b:a9:ec:3f:8c:
                    •             
                    •             
                    •             
                    db:57:0c:9d:4d:81:d1:59:f4:ae:0c:5d:08:8b:42:
                    fb:08:4d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:ubuntu01, IP Address:172.17.0.1, IP Address:192.168.1.115, IP Address:127.0.0.1
    Signature Algorithm: sha256WithRSAEncryption
         1a:52:0e:a5:b6:bb:c7:da:41:a1:16:fa:d4:5c:9f:86:1e:34:
         •             
         •             
         •             
         3c:65:bd:c1:3f:94:1e:1c:bc:3d:8c:71:e7:c9:b6:12:2e:6a:
         13:e3:3d:41:94:97:bf:d9

    Just ensure that the details presented in the certificate are consistent and correct.

Configure Basic Authentication

  1. Create and change to the directory that will contain Basic Authentication login credentials:

    $ mkdir ~/registry/auth
$ cd ~/registry/auth

  2. Generate a htpasswd file and seed it with some login credentials:

    $ docker run --entrypoint htpasswd registry:2 -Bbn developer passw0rd | tee >> htpasswd
    developer:$2y$05$ZWs.OuoN3x65dNY4HS51jOH4eiw0c887wwUGUPoIew4NYFqw2yB8G

Create the Docker Compose file

  1. Create the directory that your Registry will use as a datastore:

    $ mkdir ~/registry/data

  2. Create and change to the directory that will contain your Registry's Docker Compose file:

    $ cd ~/registry

  3. Use your preferred editor to create a docker-compose.yml file:

    registry:
  restart: always
  image: registry:2
  ports:
    - 5000:5000
  environment:
    REGISTRY_HTTP_SECRET: JuCSrrQN43olZvkOt2z3fepAtNuTebk3
    REGISTRY_HTTP_TLS_CERTIFICATE: /certs/ubuntu01.local.cert
    REGISTRY_HTTP_TLS_KEY: /certs/ubuntu01.local.key
    REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
    REGISTRY_AUTH: htpasswd
    REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
  volumes:
    - ./data:/var/lib/registry
    - ./auth:/auth
    - ./certs:/certs

    You may generate a suitable random secret for REGISTRY_HTTP_SECRET like this:

    $ cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32 && echo
    JuCSrrQN43olZvkOt2z3fepAtNuTebk3

Test the registry

  1. Change to the directory containing the docker-compose.yml file:

    $ cd ~/registry

  2. First, we will bring up the registry in the foreground:

    $ docker-compose up
    Recreating registry_registry_1
Attaching to registry_registry_1
registry_1 | time="2016-11-02T18:08:42Z" level=info msg="redis not configured" go.version=go1.6.3 instance.id=4f0cb91c-5f5b-480d-894b-34cd84e9b421 version=v2.5.1
registry_1 | time="2016-11-02T18:08:42Z" level=info msg="Starting upload purge in 21m0s" go.version=go1.6.3 instance.id=4f0cb91c-5f5b-480d-894b-34cd84e9b421 version=v2.5.1
registry_1 | time="2016-11-02T18:08:42Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.6.3 instance.id=4f0cb91c-5f5b-480d-894b-34cd84e9b421 version=v2.5.1
registry_1 | time="2016-11-02T18:08:42Z" level=info msg="listening on [::]:5000, tls" go.version=go1.6.3 instance.id=4f0cb91c-5f5b-480d-894b-34cd84e9b421 version=v2.5.1

    Provided you got the certificate work done correctly your registry should start without any TLS errors.

  3. Open another terminal window and prepare your shell, if necessary, to use the docker command. I'm using Minikube, so make any necessary changes for your own setup:

    $ eval $(minikube docker-env)
$ docker info
    Containers: 8
 Running: 4
 Paused: 0
 Stopped: 4
Images: 4
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 40
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge null
Swarm:
 NodeID:
 Is Manager: false
 Node Address:
Security Options:
Kernel Version: 4.4.14-boot2docker
Operating System: Boot2Docker 1.11.1 (TCL 7.1); master : 901340f - Fri Jul  1 22:52:19 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.79 GiB
Name: minikube
ID: OEFG:UDIX:O3RP:QJBC:TXHR:NAK5:5HNJ:XQS5:DPOI:SE56:OU5K:6DD2
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 27
 Goroutines: 58
 System Time: 2016-11-02T21:05:56.518053089Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

  4. Log into the registry using the Basic Authentication credentials you set up earlier:

    $ docker login ubuntu01.local:5000
    Username: developer
Password:
Login Succeeded

    Keep half an eye on the docker registry log in your other terminal session. Hopefully, you won't see any errors.

  5. Pull down a nice juicy image from the Docker Hub:

    $ docker pull ubuntu:latest
    latest: Pulling from library/ubuntu
6bbedd9b76a4: Pull complete
fc19d60a83f1: Pull complete
de413bb911fd: Pull complete
2879a7ad3144: Pull complete
668604fde02e: Pull complete
Digest: sha256:2d44ae143feeb36f4c898d32ed2ab2dffeb3a573d2d8928646dfc9cb7deb1315
Status: Downloaded newer image for ubuntu:latest

  6. Tag the image with our secure private registry's details:

    $ docker images
    REPOSITORY                                            TAG                 IMAGE ID            CREATED             SIZE
ubuntu                                                latest              f753707788c5        2 weeks ago         127.2 MB

    $ docker tag f753707788c5 ubuntu01.local:5000/ubuntu:latest

  7. Push the tagged image:

    $ docker push ubuntu01.local:5000/ubuntu:latest
    The push refers to a repository [ubuntu01.local:5000/ubuntu]
0e20f4f8a593: Pushed
1633f88f8c9f: Pushed
46c98490f575: Pushed
8ba4b4ea187c: Pushed
c854e44a1a5a: Pushed
latest: digest: sha256:2d44ae143feeb36f4c898d32ed2ab2dffeb3a573d2d8928646dfc9cb7deb1315 size: 1357

    Again, keeping an eye on the docker registry log, you hopefully saw no errors and the image pushed successfuly.

  8. Go to the session running your registry and press Ctrl+C to gracefully stop it.

  9. You can now check your registry's datastore for evidence that it has indeed been busy:

    $ find ~/data

Launch the registry as a daemon

  1. If you aren't already there, change to the directory containing the docker-compose.yml file:

    $ cd ~/registry

  2. Congratulations! Now you can launch the registry as a daemon:

    $ docker-compose up -d

    Each time the system is rebooted, your registry will be restarted.

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment