This guide was written because I don't particularly enjoy deploying Phoenix (or Elixir for that matter) applications. It's not easy. Primarily, I don't have a lot of money to spend on a nice, fancy VPS so compiling my Phoenix apps on my VPS often isn't an option. For that, we have Distillery releases. However, that requires me to either have a separate server for staging to use as a build server, or to keep a particular version of Erlang installed on my VPS, neither of which sound like great options to me and they all have the possibilities of version mismatches with ERTS. In addition to all this, theres a whole lot of configuration which needs to be done to setup a Phoenix app for deployment, and it's hard to remember.
For that reason, I wanted to use Docker so that all of my deployments would be automated and reproducable. In addition, Docker would allow me to have reproducable builds for my releases. I could build my releases on any machine that I wanted in a container which would target the same architecture as the one I used to run my release when I deployed it. For instance, I could build my releases on my MacBook using an Alpine-based Docker container, and then deploy those to my VPS in a different Alpine-based container. This would save me a lot of headache with trying to compile for the correct architecture for my VPS.
In addition, using Docker would allow me to recompile and swap out my app's server without having to mess with the database at all as long as I don't change my models. Since I run apps which need a lot of time to seed the database, this would provide me a lot of granular control about when I need to do that and when I don't.
Finally, this setup would allow me to run an NGINX container to proxy all of my server's traffic to the appropriate app container, since I often run multiple apps on my server.
Overall, the biggest motivation that made me want to create this guide was to setup a VPS in which I have zero dependencies besides Docker, and I wanted to have a reproducable guide for doing that which was compatable with both Phoenix 1.2 and 1.3, since I have apps running both of those versions of Phoenix.
- Run my Phoenix apps inside slim Docker containers.
- Run the database, migration tasks, and server in separate, self-contained containers.
- Use Docker Compose for easy deployments.
- Be able to swap out and upgrade individual components of the app.
- Have each app use a separate database container.
- Run multiple self-contained apps with Docker and maintain them individually.
- Connect all apps to a single NGINX container with individual configs for each active app.
- Run everything with zero dependencies on my VPS besides Docker. No Mix in production.
- Be compatible with the deployment of any other kinds of apps, not just Elixir/Phoenix apps.
- Able to run multiple apps on the same server
- Compatible with deployment of any other kind of app (not just Elixir/Phoenix)
- Compilation completely decoupled from deployment so you can compile anywhere and deploy anywhere else
This guide is written for Phoenix 1.2. However, all necesary changes to work with Phoenix 1.3 will be marked explicitly with "Phoenix 1.3: ...".
For generating releases, we will be using Distillery, which allows us to package a release of our application into a single tarball.
- Add Distillery to your Mix project.
defp deps do
[{:distillery, "~> 1.5", runtime: false}]
end-
Run
mix do deps.get, compileto get and compile Distillery. -
Run
mix release.initto intialize Distillery. -
Edit the generated config file in
rel/config.exsso thatinclude_ertsin the:prodconfig block is set tofalse. We do this because we will run the release inside a container which already has Erlang installed, so we do not need to package the Erlang runtime with the release. If you want to run your release in a standard Alpine container instead of one with Erlang installed, you could keep this option set totrue.
...
environment :prod do
set include_erts: false
...
end
...In order to be able to run migrations and seed the database from the release generated by Distillery, we will need to make a few modifications as generally outlined by the Running Migrations guide. This section assumes that you have database seeding operations defined in the normal location under priv/repo/. If you don't have these, your database will just be migrated and no seeding will be done. Otherwise, you can strip down the release_tasks.ex file as shown below.
-
Create a file called
release_tasks.exwhich will contain the tasks that can be run from the release. You can place this file anywhere in your app's directory structure, but I will place it inlib/myapp. If you are getting errors with themyapp()function as I did when using this file, you should replaceApplication.get_application(__MODULE__)with your OTP app name which looks like:myapp.- Note: if you are seeding your database with certain files, these files must be placed in the
priv/directory of your app to ensure that they are included with the generated release. Otherwise, seeding the database will not work because the files won't exist in the release.
- Note: if you are seeding your database with certain files, these files must be placed in the
defmodule MyApp.ReleaseTasks do
@start_apps [
:crypto,
:ssl,
:postgrex,
:ecto
]
def myapp, do: Application.get_application(__MODULE__)
def repos, do: Application.get_env(myapp(), :ecto_repos, [])
def seed do
me = myapp()
IO.puts "Loading #{me}.."
# Load the code for myapp, but don't start it
:ok = Application.load(me)
IO.puts "Starting dependencies.."
# Start apps necessary for executing migrations
Enum.each(@start_apps, &Application.ensure_all_started/1)
# Start the Repo(s) for myapp
IO.puts "Starting repos.."
Enum.each(repos(), &(&1.start_link(pool_size: 1)))
# Run migrations
migrate()
# Run seed script
Enum.each(repos(), &run_seeds_for/1)
# Signal shutdown
IO.puts "Success!"
:init.stop()
end
def migrate, do: Enum.each(repos(), &run_migrations_for/1)
def priv_dir(app), do: "#{:code.priv_dir(app)}"
defp run_migrations_for(repo) do
app = Keyword.get(repo.config, :otp_app)
IO.puts "Running migrations for #{app}"
Ecto.Migrator.run(repo, migrations_path(repo), :up, all: true)
end
def run_seeds_for(repo) do
# Run the seed script if it exists
seed_script = seeds_path(repo)
if File.exists?(seed_script) do
IO.puts "Running seed script.."
Code.eval_file(seed_script)
end
end
def migrations_path(repo), do: priv_path_for(repo, "migrations")
def seeds_path(repo), do: priv_path_for(repo, "seeds.exs")
def priv_path_for(repo, filename) do
app = Keyword.get(repo.config, :otp_app)
repo_underscore = repo |> Module.split |> List.last |> Macro.underscore
Path.join([priv_dir(app), repo_underscore, filename])
end
endIf your app only needs to run migrations and not seed the database, you can use a stripped-down version of the file. In this version, the seed() function only runs migrations via do_migrate(), it does not seed the database.
defmodule MyApp.ReleaseTasks do
@start_apps [
:crypto,
:ssl,
:postgrex,
:ecto
]
def myapp, do: Application.get_application(__MODULE__)
def repos, do: Application.get_env(myapp(), :ecto_repos, [])
def seed do
me = myapp()
IO.puts "Loading #{me}.."
# Load the code for myapp, but don't start it
:ok = Application.load(me)
IO.puts "Starting dependencies.."
# Start apps necessary for executing migrations
Enum.each(@start_apps, &Application.ensure_all_started/1)
# Start the Repo(s) for myapp
IO.puts "Starting repos.."
Enum.each(repos(), &(&1.start_link(pool_size: 1)))
# Run migrations
migrate()
# Run seed script
Enum.each(repos(), &run_seeds_for/1)
# Signal shutdown
IO.puts "Success!"
:init.stop()
end
def migrate, do: Enum.each(repos(), &run_migrations_for/1)
def priv_dir(app), do: "#{:code.priv_dir(app)}"
defp run_migrations_for(repo) do
app = Keyword.get(repo.config, :otp_app)
IO.puts "Running migrations for #{app}"
Ecto.Migrator.run(repo, migrations_path(repo), :up, all: true)
end
def run_seeds_for(repo) do
# Run the seed script if it exists
seed_script = seeds_path(repo)
if File.exists?(seed_script) do
IO.puts "Running seed script.."
Code.eval_file(seed_script)
end
end
def migrations_path(repo), do: priv_path_for(repo, "migrations")
def seeds_path(repo), do: priv_path_for(repo, "seeds.exs")
def priv_path_for(repo, filename) do
app = Keyword.get(repo.config, :otp_app)
repo_underscore = repo |> Module.split |> List.last |> Macro.underscore
Path.join([priv_dir(app), repo_underscore, filename])
end
end- Create a release command script at
rel/commands/migrate.sh. This script will run theseed()function from the moduleReleaseTasks.
#!/bin/sh
$RELEASE_ROOT_DIR/bin/myapp command Elixir.MyApp.ReleaseTasks seed- Add the command to the list of release commands by appending it to the commands list in the
rel/config.exsconfiguration file.
...
release :myapp do
...
set commands: [
"migrate": "rel/commands/migrate.sh"
]
end
...Now you will be able to run migrations from the release with bin/myapp migrate.
- Add a
.dockerignorefile to your app's root directory to prevent build artifacts and other unecessary files from being copied into the build container. This is necessary to ensure that when we run the container to build the release, fresh compiles the app, installs fresh Node dependencies on its own, etc. so that none of the host machine's build artifacts leak onto the container.
# Git data
.git
# Elixir build artifacts
_build
# Don't ignore the directory where our releases get built
!_build/prod/rel
deps
# Node build artifacts
node_modules
# Tests
test
# Compiled static artifacts
priv/static
Phoenix 1.3: the file needs to be modified slightly for the new location of node_modules.
diff --git a/.dockerignore b/.dockerignore
index 858de0d..c20021c 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -6,7 +6,7 @@ _build
deps
# Node build artifacts
-node_modules
+assets/node_modules
# Tests
test- Add a file called
Dockerfile.buildto the root directory of your app. This file will be used to build the release inside a Docker container with Alpine Linux (the base image we will run the release in) as the target architecture.
FROM bitwalker/alpine-elixir-phoenix:1.6.1
ENV MIX_ENV prod
# Add the files to the image
ADD . .
# Cache Elixir deps
RUN mix deps.get --only prod
RUN mix deps.compile
# Cache Node deps
RUN npm i
# Compile JavaScript
RUN npm run deploy
# Compile app
RUN mix compile
RUN mix phoenix.digest
# Generate release
ENTRYPOINT ["mix"]
CMD ["release", "--env=prod"]
Phoenix 1.3: the file should be modified slightly to accomidate the new assets directory, and the new phx.digest command.
diff --git a/df b/df
index fb7078b..6ea163e 100644
--- a/df
+++ b/df
@@ -9,15 +9,17 @@ ADD . .
RUN mix deps.get --only prod
RUN mix deps.compile
+WORKDIR assets
# Cache Node deps
RUN npm i
# Compile JavaScript
RUN npm run deploy
+WORKDIR ..
# Compile app
RUN mix compile
-RUN mix phoenix.digest
+RUN mix phx.digest
# Generate release
ENTRYPOINT ["mix"]Next we will add a shell script which will build the release. This script first builds the release builder image using Dockerfile.build, and then it runs the container, with a volume connected to the host machine's rel/ directory. This way, when the container is run, the release is built inside of the rel/ directory inside of the container but the release remains on the host machine in the same directory after the container exists. Remember, since the release is built inside the container for the Alpine Linux architecture, you will probably not be able to run this release on your host machine.
- Note: You will most likely have to add execution permissions to this file using
sudo chmod +x build.sh.
#!/bin/sh
# Remove old releases
rm -rf _build/prod/rel/*
# Build the image
docker build --rm -t myapp-build -f Dockerfile.build .
# Run the container
docker run -it --rm --name myapp-build -v $(pwd)/_build/prod/rel:/opt/app/_build/prod/rel myapp-build
Now, you can run the script with ./build.sh and it will build and run the Docker container which will build the release in _build/prod/rel/myapp.
The following diagram depicts the build process:
Next, we need a Dockerfile to run the generated release that Distillery has built for us. In the root directory of your application, create a file called Dockerfile.run. In this file, we have not specified a default command via a CMD instruction for the container. This is because we will set this in the docker-compose.yml file later so that we can change the command there if needed.
- Note: you may have to change the
0.0.1part of the path in this file depending on the version of your OTP app which you have specified.
FROM bitwalker/alpine-erlang:20.2.2
# Set exposed ports
EXPOSE 5000
# Set environment variables
ENV MIX_ENV=prod
# Copy tarball release
ADD _build/prod/rel/myapp/releases/0.0.1/myapp.tar.gz ./
# Set user
USER default
# Set entrypoint
ENTRYPOINT ["./bin/myapp"]
Now, you have a container which includes Erlang to run your release in. You could now run the container as follows:
- Build the image with
docker build -t myapp-release -f Dockerfile.run . - Run the container with
docker run --rm -it --name myapp-server -p 5000:5000 myapp-release foreground
However, this would be quite pointless at this stage because we haven't set up a database for the app to connect to, so if you did this you would just see a bunch of connection refused messages from the app.
Next, we will create a Docker Compose file for our app. This file helps us provision and manage multiple containers so that we don't have to manually start each individual container with many parameters. To do this, create a docker-compose.yml file in the root directory of the app.
- Note: you may notice the network marked as
externalat the bottom of this file. This will be addressed in a coming section.
version: "3"
services:
db:
image: postgres:10.2-alpine
container_name: myapp-db
environment:
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=myapp_prod
networks:
- nginx-network
admin:
image: myapp-release
container_name: myapp-admin
build:
context: .
dockerfile: Dockerfile.run
command: migrate
networks:
- nginx-network
depends_on:
- db
server:
image: myapp-release
container_name: myapp-server
environment:
- PORT=5000
- HOST="myapp.com"
command: foreground
networks:
- nginx-network
depends_on:
- db
- admin
networks:
nginx-network:
external: trueNow your app can be started with a single docker-compose up command. This will start a database container, then a temporary container which seeds and migrates the database with ./myapp migrate, and finally the actual Phoenix server container. However, it still won't be able to connect to the database as we haven't configured it yet. Also, it relies on an external Docker network to have already been created.
- Note: if you don't want to worry about setting up an NGINX container (which is mostly there to proxy traffic from domains to your app), then you can modify the file as follows and you will be able to access your app on port
5000of your host machine.
...
server:
...
depends_on:
...
ports:
- "5000:5000"
...The following diagram depics the process of running the application with Docker Compose:
In order to run our releases inside the Docker environment, we need to change a few Elixir config files. This section is loosely based on the Using Distillery With Phoenix.
- Open up
config/prod.exsand modify it as follows:
...
config :aau_stats, AAUStats.Endpoint,
http: [port: {:system, "PORT"}],
url: [host: {:system, "HOST"}, port: {:system, "PORT"}],
server: true,
root: ".",
version: Application.spec(:fsi, :vsn),
...Phoenix 1.3:, you only have to change the url: line as the http: line is no longer relevant. You should have a line reading load_from_system_env: true, instead.
Now, instead of hard-coded ports and a hard-coded hostname, the app now gets its ports from the runtime environment variables of our container. We define these environment variables in the environment block of our docker-compose.yml. In our configuration, we are using PORT=5000 and HOST="myapp.com", which is the domain that our server will run on.
- Open up
config/prod.secret.exsand modify it as follows:
...
config :myapp, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: "postgres",
password: "postgres",
hostname: "myapp-db"
database: "myapp_prod",
pool_size: 15
...This will ensure that our app is able to communicate with the database container myapp-db. Since our containers will all be on the nginx-network network (more on that in the next section), any container can communicate with another container by it's container name. Therefore, we tell our app to contact the database running on the hostname myapp-db, which is the name of our database container.
Finally, we have everything in place in terms of a container for our database, a container for seeding and migrating the database, and a container for our server. However, we need to set up something to allow our containers to access the outside world. We also need to setup the nginx-network that our containers have been set up to connect to. Without this Docker network, our containers won't be able to start because they won't have a network to connect to.
We will setup a Docker network called nginx-network. Normally, each Docker container started individually joins its own network. When you specify a docker-compose.yml file and start multiple containers with docker-compose up with no netowk specified in the file, your containers will all join a default network created by Docker Compose. When all of your containers are connected to the same network, they are all reachable by their hostnames which are set as their container names. This is what we want.
Then, we will start an NGINX container and provide it with config files for our applications. This way, we can have one central NGINX container which will serve the appropriate traffic to all of our applications.
-
Setup the Docker network with
docker network create nginx-network. Now you should be able to see the new network withdocker network ls. -
Next, create a new directory wherever you want called
nginxwith$ mkdir nginxand$ cd nginxinto it. This directory will be where we keep our NGINX config and Docker files. -
Create a new
docker-compose.ymlfile for your NGINX container. Here, we bind ournginx-servercontainer to port80on the host machine so it can serve all of our application traffic. We also mount a read-only volume to./conf.d, where we will place all of our application configs.
version: "3"
services:
server:
image: nginx:1.13.8-alpine
container_name: nginx-server
ports:
- "80:80"
volumes:
- ./conf.d:/etc/nginx/conf.d:ro
networks:
- nginx-network
networks:
nginx-network:
external: true-
Make a directory for your NGINX application config files with
$ mkdir conf.dand$ cd conf.dinto it. -
Create a new config file for your app in
conf.dcalledmyapp.conf. This config file will be automatically loaded into the NGINX server by the container because we have mounted a volume into the/etc/nginx/conf.ddirectory inside the container. This directory will load any*.conffiles into the NGINX server, so you can have multiple config files for multiple applications.
server {
listen 80;
server_name myapp.com;
location / {
proxy_pass http://myapp-server:5000/;
proxy_set_header Host $host;
proxy_buffering off;
}
}This config file proxies any traffic coming in from the myapp.com domain to the hostname myapp-server on port 5000, which is the HOST and the PORT that we specified in the environment configuration in our app's docker-compose.yml.
Now we have everything set up. Our application will run itself in the appropriate Docker containers while our NGINX container will run a server to proxy all of our application traffic to the appropriate location. Finally, we will run everything.
- Note: ensure that you always start your application before you start your NGINX container. If you don't NGINX will complain about unreachable hosts because your containers won't exist on the Docker network yet.
-
Start your application containers. In your application directory, run
docker-compose up. -
Start your NGINX container. In your
nginxdirectory, rundocker-compose up.
Now, you should be able to access your application on the domain that you specified.
If you want to extend your configuration for multiple applications and domains, it's very easy!
-
For your new application, follow the same application setup process above but change the
PORTin your app'sdocker-compose.ymlso that your new app's port doesn't conflict with the original application's port siince they will both be on the samenginx-network. -
Create a new configuration in
nginx/conf.dfor your new application, specifing the domain and the new port as necessary. -
Next, start your new app by running
docker-compose upin your new app's directory. -
Then, restart your NGINX container by running
docker-compose restart. Make sure your new app's server is started before you restart NGINX.
That's it! Your NGINX container will now proxy traffic appropriately for both of your apps.
The following diagram depicts running multiple apps using a single NGINX container to proxy traffic to the appropriate app containers:
A huge advantage of this deployment strategy is the fact that compiling the releases is completely decoupled from deploying them. That means that you can compile your Distillery release anywhere you want, even on your development machine. It will be compiled for the correct architecture using Docker. Both the release builder image and the deployment (runner) image are based on Alpine.
To compile your release on a different machine than your deployment machine, do the following:
- Run your
build.shscript as described above on whichever machine you want to compile your release. You should now have a release built at_build/prod/rel/myapp. And a tarball release in_build/prod/rel/myapp/releases/0.0.1/myapp.tar.gz(depending on your app version). - Copy the release tarball to your deployment machine. If you already have your app's directory structure on that machine, you can place it in
_build/prod/rel/myapp/releases/0.0.1/myapp.tar.gz(depending on your version).- Otherwise, if you don't have the entire app's directory structure on your deployment machine and are only using the Docker related files which you need to deploy the app, simply make new directories reflecting
_build/prod/rel/myapp/releases/0.0.1/myapp.tar.gzin the root directory where your Docker files (see below) are located and place the tarball there.
- Otherwise, if you don't have the entire app's directory structure on your deployment machine and are only using the Docker related files which you need to deploy the app, simply make new directories reflecting
Note: on your deployment machine, you really only need the following files to deploy the app:
Dockerfile.rundocker-compose.yml- Your release tarball in the correct directory as described above
Then, you can run your releases without having to have the rest of the app's directory structure and code on your deployment machine.