EBS and EFS Volumes with Docker For AWS using REX-Ray

README.md

EBS and EFS Volumes with Docker For AWS using REX-Ray

This procedure will deploy Docker For AWS and go through the steps to build REX-Ray containers. This process will have some hiccups because Docker for AWS will provision resources in different availability zones (AZs). Multiple workers/agents will be spread across AZs (not regions) which means a potential host failure will trigger Swarm to restart containers that could spread across an AZ. If a container is restarted in a different AZ, the pre-emption mechanism for REX-Ray will not work because it no longer has access to the volume in the former AZ.

Deploy Docker for AWS.

Launch Stack

SSH into one of your Docker Manager Nodes

ssh -i "my.pem" docker@myhost

EBS

This Dockerfile is being used for testing purposes before an official image is available on Dockerhub.

FROM alpine

MAINTAINER kendrickcoleman@gmail.com

RUN apk update
RUN apk --no-cache add wget ca-certificates openssl curl e2fsprogs
RUN update-ca-certificates 
RUN wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add --allow-untrusted glibc-2.23-r3.apk
RUN rm -rf /var/cache/apk/*
RUN curl -k -sSL https://dl.bintray.com/emccode/rexray/install | INSECURE=1 sh - 
RUN echo $'rexray:\n\
  modules:\n\
    default-docker:\n\
      disabled: true\n\
    ebs-docker:\n\
      type:     docker\n\
      host:     unix:///run/docker/plugins/ebs.sock\n\
      spec:     /etc/docker/plugins/ebs.spec\n\
      libstorage:\n\
        service: ebs\n\
libstorage:\n\
  integration:\n\
    volume:\n\
      operations:\n\
        mount:\n\
          preempt: true\n\
  server:\n\
    services:\n\
      ebs:\n\
        driver: ebs'\
>> /etc/rexray/config.yml

RUN echo -e '#!/bin/sh\n\nexec rexray start -f -l debug' > /docker-entrypoint.sh && chmod +x /docker-entrypoint.sh

ENTRYPOINT ["/docker-entrypoint.sh"]

ENV LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_PATH=/run/libstorage/volumes \
    AWS_ACCESS_KEY_ID="ACCESS KEY MISSING, provide it as environment variable" \
    AWS_SECRET_ACCESS_KEY="SECRET KEY MISSING, provide it as environment variable"
VOLUME ["/dev", "/var/lib/rexray", "/run/libstorage/volumes", "/run/docker/plugins"]

Build the image

docker build -t rexray/ebs .

EBS Runtime on Docker for AWS

docker run -tid --name rexray-ebs --privileged -e AWS_ACCESS_KEY_ID=mykey -e AWS_SECRET_ACCESS_KEY=mysecret -v /dev:/dev -v /var/lib/rexray:/var/lib/rexray -v /run/libstorage/volumes:/run/libstorage/volumes:shared -v /run/docker/plugins:/run/docker/plugins rexray/ebs

Using The EBS Driver

docker volume create -d ebs --name=pg_data
docker run -dit --name pg -e POSTGRES_PASSWORD=mysecretpassword --volume-driver=ebs -v pg_data:/var/lib/postgresql/data postgres

EFS

On each host, install nfs-utils if you want to use EFS/NFS

apk update && apk --no-cache add nfs-utils

Dockerfile with EFS

FROM alpine

MAINTAINER kendrickcoleman@gmail.com

RUN apk update
RUN apk --no-cache add wget ca-certificates openssl curl e2fsprogs nfs-utils
RUN update-ca-certificates 
RUN wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add --allow-untrusted glibc-2.23-r3.apk
RUN rm -rf /var/cache/apk/*
RUN curl -k -sSL https://dl.bintray.com/emccode/rexray/install | INSECURE=1 sh - 
RUN echo $'rexray:\n\
  modules:\n\
    default-docker:\n\
      disabled: true\n\
    efs-docker:\n\
      type:     docker\n\
      host:     unix:///run/docker/plugins/efs.sock\n\
      spec:     /etc/docker/plugins/efs.spec\n\
      libstorage:\n\
        service: efs\n\
libstorage:\n\
  integration:\n\
    volume:\n\
      operations:\n\
        mount:\n\
          preempt: true\n\
  server:\n\
    services:\n\
      efs:\n\
        driver: efs'\
>> /etc/rexray/config.yml

RUN echo -e '#!/bin/sh\n\nexec rexray start -f -l debug' > /docker-entrypoint.sh && chmod +x /docker-entrypoint.sh

ENTRYPOINT ["/docker-entrypoint.sh"]

ENV LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_PATH=/run/libstorage/volumes \
    AWS_ACCESS_KEY_ID="ACCESS KEY MISSING, provide it as environment variable" \
    AWS_SECRET_ACCESS_KEY="SECRET KEY MISSING, provide it as environment variable" \
    EFS_SECURITYGROUPS="SECURITY GROUPS MISSING, provide it as environment variable"
VOLUME ["/dev", "/var/lib/rexray", "/run/libstorage/volumes", "/run/docker/plugins"]

Build the image

docker build -t rexray/efs .

EFS Runtime on Docker for AWS

docker run -tid --name rexray-efs --privileged -e AWS_ACCESS_KEY_ID=mykey -e AWS_SECRET_ACCESS_KEY=mysecrey -e EFS_SECURITYGROUPS=mygroup -v /dev:/dev -v /var/lib/rexray:/var/lib/rexray -v /run/libstorage/volumes:/run/libstorage/volumes:shared -v /run/docker/plugins:/run/docker/plugins rexray/efs

Using The EFS Driver

docker volume create -d efs --name=pg_data
docker run -dit --name pg -e POSTGRES_PASSWORD=mysecretpassword --volume-driver=efs -v pg_data:/var/lib/postgresql/data postgres

Curious if you ever ran into this one with EFS...