dont_fear_the_release.md

Don't fear the release

Releases with Distillery 2.0 and Docker

Contents

1. OTP Release using distillery
   • Create a new phoenix project
   • Building the first release
   • Run your release
   • Move the release anywhere
   • Performing upgrades
2. Building releases with docker
   • Multi-stage docker builds
   • The Dockerfile
   • Use config_providers
3. Automating the build process
   • Create a Makefile
   • Using of docker-compose
4. Ressources

OTP Release using distillery

Create a new phoenix project

➜ mix phx.new --no-ecto my_app

Add distillery

mix.exs

...

defp deps do
    [
     ...,   
     {:distillery, "~> 2.0"}
    ]
  end

Update config/prod.exs

• System.get_env(varname) Returns the value of the given environment variable
• server configures the endpoint to boot the Cowboy application http endpoint on start
• root configures the application root for serving static files
• version ensures that the asset cache will be busted on versioned application upgrades (hot-upgrades)

port = System.get_env("PORT") || 4000
host = System.get_env("HOST") || "localhost"

config :phoenix_distillery, MyAppWeb.Endpoint,
  http: [port: port],
  url: [host: host, port: port],
  cache_static_manifest: "priv/static/cache_manifest.json",
  server: true,
  root: ".",
  version: Application.spec(:phoenix_distillery, :vsn)

endpoint.ex You should set gzip to true if you are running phx.digest when deploying your static files in production.

Building the first release

Fetching deps

• fetch production dependencies from hex
• initialize distillery which creates
  rel/config.exs
• in addition to an empty directory
  rel/plugins/

➜ mix deps.get --only prod
➜ mix release.init

Compile elixir and assets

• mix phx.digest compress and tag your assets for proper caching
• mix release env=prod generate a release for a production environment
• hint: combined mix tasks with MIX_ENV=prod mix do phx.digest, release env=prod

➜ MIX_ENV=prod mix compile
➜ cd assets && webpack --mode production && cd ..
➜ MIX_ENV=prod mix phx.digest
➜ MIX_ENV=prod mix release --env=prod

If you are inside an umbrella application and you got an error regarding Jason
check config/config.exs and ensure the line
config :phoenix, :json_library, Jason
exists

Run the release

➜ _build/prod/rel/my_app/bin/my_app start
➜ _build/prod/rel/my_app/bin/my_app stop
➜ _build/prod/rel/my_app/bin/my_app foreground

Move the release anywhere

• cp _build/prod/rel/my_app/releases/0.0.1/my_app.tar.gz deployment_target/
  Move the release anywhere by copying the release tarball
• cd deployment_target/ && tar xvf my_app.tar.gz
  Extract the tarball at in the target location
• ./bin/my_app start
• ./bin/my_app stop

Performing upgrades

Bump the version in mix.exs and add new features

def project do
   [
    version: "0.2.0",
    ...
   ]
end

Create a new --upgrad release

• MIX_ENV=prod mix release --env=prod --upgrade
  tells Distillery to build an upgrade from the previously built releases in the output directory
• If the upgrade build is not failing a new tarball is created
  _build/prod/rel/phoenix_distillery/releases/0.2.0/my_app.tar.gz
• This tarball can deployed into an existing release, for example in the previous used deployment target

➜ MIX_ENV=prod mix compile
➜ cd assets && webpack --mode production && cd ..
➜ MIX_ENV=prod mix phx.digest
➜ MIX_ENV=prod mix release --env=prod --upgrade

Deploy the new release to the target and run the upgrade

➜ cp _build/prod/rel/phoenix_distillery/releases/0.2.0/my_app.tar.gz  deployment_destination/releases/0.2.0/
➜ ./deployment_target/bin/my_app upgrade

Don't forget to create a new release directory
mkdir deployment_target/release/0.2.0 Depending on your changes, you may have to reload your browser to see the changes

Building a release inside docker

Multi-stage docker builds

With multi-stage builds, you use multiple FROM statements in your Dockerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build. You can selectively copy artifacts from one stage to another, leaving behind everything you don’t want in the final image.

Add a .dockerignore

_build/
deps/
.git/
.gitignore
Dockerfile
Makefile
README*
test/
priv/static/

Add a Dockerfile

#StageOne: Build Container
FROM elixir:1.7.4-alpine AS builder

# The following are build arguments used to change variable parts of the image.
ARG ALPINE_VERSION=3.8
ARG APP_NAME
ARG APP_VSN
ARG MIX_ENV=prod
ARG SKIP_PHOENIX=false
ARG PHOENIX_SUBDIR=.

ENV SKIP_PHOENIX=${SKIP_PHOENIX} \
    APP_NAME=${APP_NAME} \
    APP_VSN=${APP_VSN} \
    MIX_ENV=${MIX_ENV}

WORKDIR /opt/app

# Installs build tools
RUN apk update && \
    apk upgrade --no-cache && \
    apk add --no-cache \
    nodejs \
    npm \
    git \
    build-base && \
    mix local.rebar --force && \
    mix local.hex --force

# Copy our app source code into the build container
COPY . .

RUN mix do deps.get, deps.compile, compile

# Digest assets if we have a phoenix app
RUN if [ ! "$SKIP_PHOENIX" = "true" ]; then \
    cd ${PHOENIX_SUBDIR}/assets && \
    npm ci && \
    npm run deploy && \
    cd .. && \
    mix phx.digest; \
    fi

# Create a release, copy and extract the tarball into a build folder
RUN \
    mkdir -p /opt/built && \
    mix release --verbose && \
    cp _build/${MIX_ENV}/rel/${APP_NAME}/releases/${APP_VSN}/${APP_NAME}.tar.gz /opt/built && \
    cd /opt/built && \
    tar -xzf ${APP_NAME}.tar.gz && \
    rm ${APP_NAME}.tar.gz

# StageTwo: Runtime Container
FROM alpine:${ALPINE_VERSION}

ARG APP_NAME

RUN apk update && \
    apk add --no-cache \
    bash \
    openssl-dev

# OS_VARS will be set in our makefile
ENV REPLACE_OS_VARS=true \
    APP_NAME=${APP_NAME}

WORKDIR /opt/app

# Copy the release from StageOne
COPY --from=builder /opt/built .

CMD trap 'exit' INT; /opt/app/bin/${APP_NAME} foreground

Use config-providers

Configuration providers for loading and persisting runtime configuration during boot.

Compiletime vs. Runtime variables

• http: [port: System.get_env("PORT")]
  Here you would need to provide the PORT environment variable at build time which is when that code would be executed.
• {:system, "PORT"}
  This isn't some magic code to retrieve an environment variable, it is a setting that tells Phoenix to retrieve the port number from the PORT environment variable at runtime

Create/Update rel/config.exs

release :myapp do
  # snip..
  set config_providers: [
    {Mix.Releases.Config.Providers.Elixir, ["${RELEASE_ROOT_DIR}/etc/config.exs"]}
  ]

  # Overlays allow you to modify the contents of the release, you may add/symlink files, create directories, and generate files based on templates.
  set overlays: [
    {:copy, "rel/config/config.exs", "etc/config.exs"}
  ]
end

Create/Update the referenced config file rel/config/config.exs

Use Compile time variables from our docker.env to replace with runtime variables from the original phoenix my_app/config/config.exs

use Mix.Config

config :myapp, MyApp.Repo,
  username: System.get_env("DATABASE_USER"),
  password: System.get_env("DATABASE_PASS"),
  database: System.get_env("DATABASE_NAME"),
  hostname: System.get_env("DATABASE_HOST"),
  pool_size: 15

port = String.to_integer(System.get_env("PORT") || "8080")
config :myapp, MyApp.Endpoint,
  http: [port: port],
  url: [host: System.get_env("HOSTNAME"), port: port],
  root: ".",
  secret_key_base: System.get_env("SECRET_KEY_BASE")

Automating the build process

To help automate building images, it is recommended to use a Makefile or shell script.
Run Makefiles with the make command in your project directory

Create a Makefile

.PHONY: help

APP_NAME ?= `grep 'app:' mix.exs | sed -e 's/\[//g' -e 's/ //g' -e 's/app://' -e 's/[:,]//g'`
APP_VSN ?= `grep 'version:' mix.exs | cut -d '"' -f2`
BUILD ?= `git rev-parse --short HEAD`

help:
    @echo "$(APP_NAME):$(APP_VSN)-$(BUILD)"
    @perl -nle'print $& if m{^[a-zA-Z_-]+:.*?## .*$$}' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

build: ## Build the Docker image
    docker build --build-arg APP_NAME=$(APP_NAME) \
        --build-arg APP_VSN=$(APP_VSN) \
        -t $(APP_NAME):$(APP_VSN)-$(BUILD) \
        -t $(APP_NAME):latest .

run: ## Run the app in Docker
    docker run --env-file config/docker.env \
        --expose 4000 -p 4000:4000 \
        --rm -it $(APP_NAME):latest

If make reports an error mentioning multiple target patterns Ensure that you are using tabs and not spaces in the makefile

Run the build command

➜ make build

Using of docker-compose

Create a config/docker.env

This variables will be exported as system env variables inside our running docker container

HOSTNAME=localhost
SECRET_KEY_BASE="u1QXlca4XEZKb1o3HL/aUlznI1qstCNAQ6yme/lFbFIs0Iqiq/annZ+Ty8JyUCDc"
DATABASE_HOST=db
DATABASE_USER=postgres
DATABASE_PASS=postgres
DATABASE_NAME=myapp_db
PORT=4000
LANG=en_US.UTF-8
REPLACE_OS_VARS=true
ERLANG_COOKIE=myapp

docker-compose up

docker-compose.yml

version: '3.5'

services:
  web:
    image: "myapp:latest"
    ports:
      - "80:4000" # In our .env file above, we chose port 4000
    env_file:
      - config/docker.env

docker-swarm

• docker swarm init --advertise-addr <ip address of droplet> --listen-addr <ip address of droplet>
• docker stack deploy -c docker-compose.yml myapp

version: '3.5'

networks: 
  webnet:
    driver: overlay
    attachable: true # enables running custom commands in the container

services:
  web:
    image: "myapp:latest"

    ports:
      - "80:4000"
    env_file:
      - config/docker.env # exports our system env variables
    networks:
      - webnet

Ressources

Link List

Distillery - github
Distillery - Phoenix Walkthrough
elixir-forum - Compiletime/Runtime variables
Docker multi-stage builds
Distillery - Working with Docker
Slides

Sorry, my charger broke. I'll write up what I had to add.

We actually did not use distillery and multistaged builds (we had each of them in two different project). I tried to merge the two projects' Dockerfiles but left a note from which point on I have not tested it (because it's hard to set up with databases).

# hexdeps just installs dependencies, but caches the dependency getting (which is nice).
FROM elixir:1.6 as hexdeps
ARG mix_env=dev
ENV MIX_ENV=$mix_env
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY ["mix.exs", "mix.lock", "/usr/src/app/"]
RUN mix do local.hex --force, local.rebar --force
RUN mix do deps.get, deps.compile
# precompile most of the plt files (this is because we used to typecheck with CI on the dev build)
RUN ["/bin/bash", "-c", "if [[ $MIX_ENV = dev ]]; then mix dialyzer --plt; fi"]

# buildAssets just builds the frontend
FROM node:carbon as buildAssets
RUN mkdir -p /usr/src/app/assets
WORKDIR /usr/src/app/assets

COPY --from=hexdeps /usr/src/app/deps/phoenix /usr/src/app/deps/phoenix
COPY ["assets/package.json", "assets/package-lock.json", "./"]
RUN npm i
COPY assets/ .
RUN npm run deploy

FROM hexdeps as compiled
ARG mix_env=dev
ENV MIX_ENV=$mix_env
ARG APP_NAME
ARG APP_VSN
# I used to install mysql here, since this was not a distillery project
COPY . /usr/src/app
COPY --from=buildAssets /usr/src/app/priv/static /usr/src/app/priv/static

ENV PORT=4000

RUN mix do compile, compile.app
RUN mix phx.digest
# CMD mix phx.server
# FROM HERE ON EDITED AND NOT TESTED

# extra stage, because for testing, linting and typechecking
# we want to run docker build --target=compiled to get the stage without the release compile

FROM compiled as release
RUN mkdir -p /opt/build
RUN mix release --verbose && \
    tar -C /opt/build xf _build/${MIX_ENV}/rel/${APP_NAME}/releases/${APP_VSN}/${APP_NAME}.tar.gz

FROM debian
ARG APP_NAME
RUN apt-get update && apt-get install -y mysql-client && apt-get clean && apt-get autoremove
COPY --from=release /opt/build ./
CMD trap 'exit' INT; /opt/app/bin/${APP_NAME} foreground

Hi, it's me again,

I actually did a complete deployment with elixir and phoenix. Since distillery got an update I thought I'd post you an updated (and actually working Dockerfile). I only tested production builds for now, but they work fine.

# hexdeps just installs dependencies, but caches the dependency getting (which is nice).
FROM elixir:1.9.1-alpine as hexdeps
RUN apk --no-cache add bash git
RUN addgroup -S appuser && adduser -S appuser -G appuser
WORKDIR /home/appuser
USER appuser
ARG MIX_ENV=prod
ENV MIX_ENV=$MIX_ENV
RUN mix do local.hex --force, local.rebar --force
COPY --chown=appuser ["mix.exs", "mix.lock", "config", "/home/appuser/"]
RUN mix do deps.get, deps.compile
# precompile most of the plt files (this is because we used to typecheck with CI on the dev build)
# RUN ["/bin/bash", "-c", "if [[ $MIX_ENV = dev ]]; then mix dialyzer --plt; fi"]

# buildAssets just builds the frontend
FROM node:dubnium-alpine as buildAssets
RUN addgroup -S appuser && adduser -S appuser -G appuser
RUN install -do appuser  /home/appuser /home/appuser/assets /home/appuser/deps/phoenix_html /home/appuser/deps/phoenix
WORKDIR /home/appuser/assets
USER appuser
COPY ["assets/package.json", "assets/package-lock.json", "./"]
RUN npm i
COPY --from=hexdeps --chown=appuser ["/home/appuser/deps/phoenix/", "/home/appuser/deps/phoenix/"]
COPY --from=hexdeps --chown=appuser ["/home/appuser/deps/phoenix_html/", "/home/appuser/deps/phoenix_html/"]
COPY --chown=appuser assets/ .
RUN npm run deploy

FROM hexdeps as compiled
COPY --chown=appuser . /home/appuser
ARG MIX_ENV=prod
ENV MIX_ENV=$MIX_ENV
RUN mix do compile, compile.app
COPY --from=buildAssets --chown=appuser /home/appuser/priv/static /home/appuser/priv/static
RUN mix phx.digest

# put another stage here, that way we can tell the CI to only run until here and have a compiled container for tests/lints/typecheck etc.
FROM compiled as release
RUN [[ $MIX_ENV = prod ]] && mix distillery.release --env=prod || mix distillery.release
RUN mkdir -p release
# Sadly, ADD does not accept --from so we'll have to unpack here (and we're using bash for $() to work)
RUN echo tar -C release -xzf _build/${MIX_ENV}/rel/$(mix app.config app | tail -1)/releases/$(mix app.config vers
ion | tail -1)/$(mix app.config app | tail -1).tar.gz | bash
RUN cp release/bin/$(mix app.config app | tail -1) release/bin/elixir_program

#IMPORTANT: version number has to be the right one
# `apk info -v | grep musl` in elixir and alpine should ouptut the same versions
FROM alpine:3.9.4
RUN apk --no-cache add bash
RUN addgroup -S appuser && adduser -S appuser -G appuser
EXPOSE 4000
COPY --from=release --chown=appuser /home/appuser/release /home/appuser
USER appuser
WORKDIR /home/appuser
CMD trap 'exit' INT; /home/appuser/bin/elixir_program foreground

The following changes to a bare mix phx.new:

• I've added a mix task to get the version and app-name (that way the Dockerfile could stay generic). It's an alternative to your Makefile approach. this way it stays inside of mix.

lib/mix/tasks/app/config.ex

defmodule Mix.Tasks.App.Config do
  @moduledoc """
  Simply prints some app config variables (useful for CI)
  """

  use Mix.Task

  def run(args) do
    for arg <- args do
      key = String.to_atom(arg)
      Mix.Project.config[key] |> IO.puts
    end
  end
end

• installed distillery and ran mix distillery.init
• I added config-tuples and added it as an config provider in rel/config.exs
• Of course I changed the database configs accordingly.