The advantage of containerization like Docker is that the container (nee image) is both the application and its operating environment.
However, since the container has its own operating environment, it also has its own environment variables which are separate from your local (i.e. shell, i.e. native, i.e. calling) environment variables.
Meaning that when you are using Docker, there are TWO sets of separate and independent environment variables to consider...
- Local environment variables in the shell where you are calling the docker/docker-compose commands, like your laptop, CI/CD, or production server)
- Container environment variables in your running containers
🚢 Captain Obvious says that it is important to note that environment variables in your running container are completely isolated and separate from those in your local environment unless you explicitly create and connect them
Having these two separate sets of environment variables can be confusing. In this blog post I hope to remove this confusion.
Here with a simple container that you can build and run yourself, I will show you through examples of sample output the two basic concepts of using environment variables in Docker...
- Environment variable substitution in your docker-compose files
- Setting the container environment variables in docker and docker-compose
And I will show how you can combine these two concepts to configure your containers and their applications (spoiler alert: you use environment variable substitution).
✨ Environment variable substitution is substituting the values of your local environment variables within your docker-compose file settings to configure your running containers
For a succinct summary of using environment variables with Docker and docker-compose, see the Cheat Sheet.
By building a simple image that displays the container's environment variables and running some examples, you can see the different ways environment variables can be used with Docker.
To build your target-app, just...
- Use the official alpine base image since it is small and fast to build
- In the image, simply execute the
envcommand to display the environment variables in our container to see their settings
Like this very basic Dockerfile for these examples...
FROM alpine:latest
CMD env🚢 Captain Obvious says that all the files in these examples should be in the same current directory
Assuming that you are in the directory containing this Dockerfile,
use the following command to build your image and name it target-app...
docker build -t target-app .
✨ By default
dockerbuildwill tag your image aslatest.
Let's look at to how you can configure your docker-compose file settings and containers from your local environment variables using environment variable substitution.
This makes your docker-compose files more flexible and adaptable
and reduces duplication, like using the same docker-compose.yml
file in all of your environments (e.g. local, CI/CD, production).
You can use this docker-compose.yml file which uses the TAG
environment variable to specify the tag to be used for your image...
version: '3.4'
services:
targetapp:
image: target-app:${TAG}If you run this docker-compose config command and don't set the TAG
environment variable...
docker-compose config
You should see something like this...
% docker-compose config
WARNING: The TAG variable is not set. Defaulting to a blank string.
services:
targetapp:
image: 'target-app:'
version: '3.4'
And if you try running this docker-compose up command...
docker-compose up
You should see..
% docker-compose up
WARNING: The TAG variable is not set. Defaulting to a blank string.
ERROR: no such image: target-app:: invalid reference format
🚢 Captain Obvious says that this is assuming that the
TAGenvironment variable is not set in your local environment i.e.unset TAG
Look at how in both commands docker-compose is attempting to substitute
the value of the local environment variable TAG which you did not
set (and causes a warning and then an error).
Now you can try it with setting the local TAG to the value latest.
First let's try it with the docker-compose config command...
TAG=latest docker-compose config
It should look like this...
% TAG=latest docker-compose config
services:
targetapp:
image: target-app:latest
version: '3.4'
By setting the local environment variable TAG, docker-compose
substitutes its value latest.
Now try bringing it up with the docker-compose up command...
TAG=latest docker-compose up
You should see it come up and run and exit...
% TAG=latest docker-compose up
Creating sandbox_targetapp_1 ... done
Attaching to sandbox_targetapp_1
targetapp_1 | HOSTNAME=dfd8a3759fc4
targetapp_1 | SHLVL=1
targetapp_1 | HOME=/root
targetapp_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
targetapp_1 | PWD=/
sandbox_targetapp_1 exited with code 0
Here it performed the environment variable substitution for TAG,
successfully ran your image as a container (which simply outputs the
environment variables in your container), and then exited the
container.
Notice in the output how the TAG environment variable is NOT set
(i.e. not present) in the container.
This shows how environment variables in your running container are completely isolated and separate from those in your local environment.
🚢 Captain Obvious says to be sure to run the corresponding
docker-compose downcommand to tear down and clean up the resources created bydocker-compose up(e.g.TAG=latest docker-compose down)
You can also use a file for environment variable substitution in your docker-compose files. This is useful if you have a lot of environment variables to substitute.
By default, docker-compose will look for a file named .env in the
directory where you are running the docker-compose command.
✨ You can associate (and deploy) docker-compose .env files with specific environments like
.env.local,.env.ci,.env.prod.
Consider this .env file where you will set the TAG environment
variable to the value latest...
TAG=latest
Now when you run the docker-compose config command...
% docker-compose config
services:
targetapp:
image: target-app:latest
version: '3.4'
And when you run the docker-compose up command...
% docker-compose up
Creating network "sandbox_default" with the default driver
Creating sandbox_targetapp_1 ... done
Attaching to sandbox_targetapp_1
targetapp_1 | HOSTNAME=47f9c4c3d66c
targetapp_1 | SHLVL=1
targetapp_1 | HOME=/root
targetapp_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
targetapp_1 | PWD=/
sandbox_targetapp_1 exited with code 0
You see that docker-compose is using the value for TAG set in the
.env file and that again TAG is not present in the container.
🙇 For more information on the
.envfile in docker-compose, see the Docker documentation on the env file
You can also specify a different environment variable file using the
--env-file option with the docker-compose commands. For example
if you rename your .env file to .env.tag...
% docker-compose --env-file .env.tag config
services:
targetapp:
image: target-app:latest
version: '3.4'
🙇 For more information on the
--env-fileoption in docker-compose, see the Docker documentation on the --env-file option
Local environment variables take precedence over those in .env
or specified by --env-file...
Like for this .env...
TAG=latest
If you run TAG= docker-compose config...
% TAG= docker-compose config
services:
targetapp:
image: 'target-app:'
version: '3.4'
Notice how TAG was empty.
Now that you have seen the basics of environment variable substitution in Docker, here are the more advanced features.
- Substitute the exact state of the local environment variable
- Set default values
- Exit with error for required environment variable
🚢 Captain Obvious says that with environment variables, there are three "states" to consider...
- The environment variable is not set (i.e. not present or null)
- The environment variable is set but empty (no value) (
ENVIRONMENTVARIABLE=)- The environment variable is set to a value (
TAG=latest)
If you want the substitution to match the exact state of the local environment variable (i.e. any of the three), just use the raw environment variable on the "left hand side".
For example, with this .env file...
TAG
When TAG is... |
For Instance... | Sample docker-compose config Output |
|---|---|---|
| Not set | unset TAG |
% docker-compose config |
| Set but empty | TAG= |
% TAG= docker-compose config |
| Set to a value | TAG=latest |
% TAG=latest docker-compose config |
Docker supports the typical *nix shell default values syntax:
| This Variable Substitution... | Sets the default value default IF... |
|---|---|
${ENVIRONMENTVARIABLE-default} |
ENVIRONMENTVARIABLE is Not Set |
${ENVIRONMENTVARIABLE:-default} |
ENVIRONMENTVARIABLE is Not Set or Empty |
For example, this sets the tag setting to latest only if the
local TAG environment variable is not set...
TAG=${TAG-latest}For your required environment variables, Docker supports the typical *nix shell variable error syntax:
| This Variable Substitution... | Exits with an Error Message err IF... |
|---|---|
${ENVIRONMENTVARIABLE?err} |
ENVIRONMENTVARIABLE is Not Set |
${ENVIRONMENTVARIABLE:?err} |
ENVIRONMENTVARIABLE is Not Set or Empty |
For Example, if you require a DB_NAME value
DB_NAME=${DB_NAME:?You must specify a DB_NAME}That's about it for the concept of environment variable substitution in docker-compose files. Next, you'll look at setting your container's environment variables.
🙇 I learned this stuff about environment variable substitution from the docker compose documentation on variable substitution
Now that you have seen how to substitute the values of your local environment variables using the concept of environment variable substitution, I will show the next concept on how to set your container's environment variables. I will also show you how you can use what you learned about environment variable substitution to set your container environment variables from your local environment variables, thus connecting them.
Use the docker run -e (or docker run --env) option to set environment
variable in the container, for example here where you set MY_USERNAME
and MY_PASSWORD ...
% docker run --env MY_USERNAME=tomsmith --env MY_PASSWORD=SuperSecretPassword! target-app
HOSTNAME=45ea011433fe
SHLVL=1
HOME=/root
MY_PASSWORD=SuperSecretPassword!
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
MY_USERNAME=tomsmith
🚢 Captain Obvious says that you should probably not pass in secrets like this
And you can use local environment variable substitution too to
set your container environment variables, like here where you use
the value of the local environment variable MY_USERNAME to set
the value of the container environment variable MY_USERNAME...
% MY_USERNAME=tomjones docker run --env MY_USERNAME --env MY_PASSWORD=SuperSecretPassword! target-app
HOSTNAME=dc0e26322c76
SHLVL=1
HOME=/root
MY_PASSWORD=SuperSecretPassword!
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
MY_USERNAME=tomjones
And just like with docker-compose, you can use a file to set the
container's environment variables, although you must specify it with
the --env-file option.
Consider this .env file...
MY_USERNAME=tomjones
MY_PASSWORD
When you run it supplying the MY_PASSWORD as a local
environment variable...
% MY_PASSWORD=SuperSecretPassword! docker run --env-file .env target-app
HOSTNAME=683790b4ce31
SHLVL=1
HOME=/root
MY_PASSWORD=SuperSecretPassword!
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
MY_USERNAME=tomjones
🙇 For more information on setting container environment variables on the command line with
docker run, see the Docker documentation on setting environment variables
You have two ways to set container environment variables in docker-compose:
- Set them on the command line with
docker-compose run -e - Set them in your docker-compose file
Just like with the docker run command, you can set your container
environment variables on the command line using the -e option.
Here is your docker-compose.yml file again...
version: '3.4'
services:
targetapp:
image: target-app:${TAG}🚢 Captain Obvious says remember that
TAGin the lineimage: target-app:${TAG}is using environment variable substitution of the local environment variableTAGfor a docker-compose file setting and is NOT a container environment variable
And assuming that you have a .env file with...
TAG=latest
You can set the environment variables MY_USERNAME and MY_PASSWORD
(using environment variable substitution) in the container with
docker-compose run -e...
% MY_PASSWORD=SuperSecretPassword! docker-compose run -e MY_USERNAME=tomsmith -e MY_PASSWORD targetapp
Creating sandbox_targetapp_run ... done
HOSTNAME=c9a7d953577c
SHLVL=1
HOME=/root
MY_PASSWORD=SuperSecretPassword!
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
MY_USERNAME=tomsmith
Unlike the
docker runcommand, there is no--envlong option, just-efor specifying environment variables
Before moving on, you can really see the difference between using environment variable substitution for docker-compose file settings like the image tag and setting container environment variables.
Use your current docker-compose.yml file but delete your .env
file (i.e. rm .env) that uses environment variable substitution
to set TAG.
Now, if you try and use docker-compose run -e to set TAG, you will
see that it will not work...
% MY_PASSWORD=SuperSecretPassword! docker-compose run -e TAG=latest -e MY_USERNAME=tomsmith -e MY_PASSWORD targetapp
WARNING: The TAG variable is not set. Defaulting to a blank string.
ERROR: no such image: target-app:: invalid reference format
🚢 Captain Obvious says that the
docker-compose run -ecommand is only for setting container environment variables (although you can use local environment variable substitution for their values)
Generally, you will set your container environment variables in your
docker-compose files using the environment key under your service,
for example...
version: '3.4'
services:
targetapp:
image: target-app:${TAG}
environment:
- MY_USERNAME=tomsmith
- MY_PASSWORDAgain, you can use (local) environment variable substitution for the values of your defined container environment variables.
Finally, you can specify an environment file in your docker-compose
files using the env_file key under your service, for example...
version: '3.4'
services:
targetapp:
image: target-app:${TAG}
environment:
- MY_USERNAME=tomsmith
- MY_PASSWORD
env_file: .env.login🚢 Captain Obvious says it is important to note that in docker-compose, "env files" can be used both to set values for any defined environment variables in your containers (e.g.
MY_LOGIN) as well as to configure docker-compose settings (e.g.TAG)
Finally, per the docker-compose documentation, the following is the order of precedence for setting environment variables in multiple ways...
- Environment variable settings in the docker-compose file
- Local environment variables
- Environment file
- Dockerfile
This concludes this deep dive into using environment variables with Docker and I hope that it has cleared up some of the confusion with this complex topic.