Docker Guides

docker-compose.md

Docker Compose

Docker compose is a tool that is used for defining and running multi-container Docker apps in an easy way. It provides docker-compose.yml configuration file that can be used to bring up an app and the suite of services it depends on with just one command.

• docker compose up - start all services from docker-compose.yml
• docker compose up db - start only service db
• docker compose up --build - rebuild all images and run containers
• docker compose -f docker-compose.prod.yml up - start all services from docker-compose.prod.yml
• docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d - run Docker Compose and override config file
• docker compose build db - rebuild db service
• docker compose down
• docker compose down --remove-orphans - remove containers for services not defined in the Compose file.
• docker compose down -v - shutdown containers, remove volumes. Docker stores volumes at /var/lib/docker/volumes
• docker compose logs db | less - show logs of db container
• docker compose ps - show running app's containers
• docker compose exec fpm bash - connect to running fpm container and start bash console in it
• docker compose exec db mysql -u root -ppassword - connect to running db container and start mysql console
• docker compose run --rm php-cli sh - run php-cli container, start sh console. --rm - Docker also removes the anonymous volumes associated with the container when the container is removed.
• docker compose rm db - remove db container
• docker compose rm -fv - remove all containers with volumes
• docker compose config - show docker-compose.yml content after the substitution step has been performed

Proxy settings for Docker

If you see an error:

docker compose up
Pulling db (mysql:)...
ERROR: Get https://registry-1.docker.io/v2/: Proxy Authentication Required

you should configure your Docker's proxy settings (see stackoverflow).

1. Create file /etc/systemd/system/docker.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://kenny:123@277.27.16.233:3118"
Environment="HTTPS_PROXY=http://kenny:123@277.27.16.233:3118"
Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com"

2. Reload docker:

sudo service docker restart

3. Check if it works:

docker pull busybox

Environment variables

Example of docker-compose.yml with env variables:

services:
    database:
        image: postgres:${POSTGRES_VERSION:-15}-alpine
        environment:
            POSTGRES_DB: ${POSTGRES_DB:-app}
            POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-123}
            POSTGRES_USER: ${POSTGRES_USER:-app}
        volumes:
            - database_data:/var/lib/postgresql/data:rw

We can pass environment variables to docker-compose.yml files in these ways:

1. By exporting the variable to the terminal before running docker compose:

   POSTGRES_PASSWORD=123 docker compose config
   

2. By putting the variables inside .env file:

   POSTGRES_PASSWORD=123
   

The env_file option only passes those extra variables to the containers and not the docker-compose.yml file.

docker-compose.yml

build, image

If we use build and image, the image will be built and will then be tagged based on the name given using image:

services:
    some-service:
        build: .
        image: some-service:1.5

healthcheck

services:
    some-service:
        # ...
        healthcheck:
            test: ["CMD", "curl", "-f", "http://localhost:8080/ping"]
            interval: 20s
            timeout: 10s
            retries: 3
            start_period: 5s

Every 20 seconds after an initial period of 5 seconds, a ping to some-service will be issued using curl with a timeout of 10 seconds. In case of failure, there should be 3 retries before marking the instance unhealthy. We use curl with -f option: fail fast with no output on HTTP errors.

Run docker ps to see if container's status is healty or unhealty.

Links:

• Change pre-defined environment variables in Docker Compose.
• Ways to set environment variables in Compose

docker_commands.md

Docker Commands

Installation

We can install Docker using official bash script.

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

System

• docker system prune - clean up any resources - images, containers, volumes, and networks — that are dangling (not associated with a container)
• docker system prune -a - remove ALL stopped containers, unused networks, abandoned images, build cache

Containers

They are created from Docker Images and run the actual app.

• docker run busybox echo 123 - find image, load up the container, run a provided command
• docker run --rm busybox - flag --rm: remove container when it exits
• docker run -it busybox sh - flag --it: attach us to an interactive tty in the container
• docker run --name hello -it busybox sh - param --name hello sets name hello for created container
• docker run -p 8000:80 nginx - flag -p: publish a container's port to the host
• docker run --net foo-network foo-container - flag --net: set network for container
• docker run --rm -it -v $(pwd):/app php:8 bash - create volume for current dir
• docker run --rm -it redis redis-cli -h host.docker.internal -p 6379 - run redis container, enter redis-cli connected to the port 6379
• docker cp nginx-example-nginx-1:/etc/nginx/nginx.conf ./nginx.conf - copy file from working container to host
• docker ps - list running containers
• docker ps -a - list all containers
• docker start foo-container - start existing container by name
• docker start 0a8a89d6b4e5 - ... by container ID
• docker stop foo-container - stop existing container by name
• docker stop 0a8a89d6b4e5 - ... by container ID
• docker rm foo-container - remove container
• docker kill foo-container - kill container
• docker container ls - list contaners
• docker container ls -a - list all containers
• docker container prune - remove all stopped containers
• docker port foo-container - show exposed ports
• docker container logs foo-container - see logs of the container
• docker exec -it foo-container sh - run console into container
• docker logs -f foo-container - show logs of container. Docker's log location: /var/lib/docker/containers/
• docker stats - show statistics for running containers
• docker stats foo-container - show CPU, Memory Usage, Net, Block I/O of container

Images

They are blueprints of our app which form the basis of containers.

• docker build -t kenny/nginx:2.0 . - build an image with repo name kenny/nginx and tag 2.0. And the . lets Docker know where it can find the Dockerfile.
• docker image tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG] - Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE (it's like rename image tag)
• docker image ls - list images
• docker image prune - remove all dangling images (not used by containers)
• docker rmi -f foo-image-name - remove image, -f - forced mode
• docker save nginx > nginx.tar - save image to .tar archive. Size: 121Mb
• docker save nginx:latest | gzip > nginx.tar.gz - save image to .tar.gz archive. Size: 50Mb
• docker load < ./nginx.tar.gz - load image from .tar.gz or .tar archive

Volumes

• docker volume create my-volume - create volume
• docker volume rm my-volume - remove volume
• docker volume ls - list all volumes
• docker volume inspect my-volume - show info about volume
• docker run -it --rm --name example-volume-container --mount source=my-volume,target=/storage bash - mount my-volume to /storage path in the container of bash image
• docker run -it --rm --name read-only-volume-container --mount source=my-volume,target=/storage,readonly bash - mount my-volume to /storage path in readonly mode. We can't write data to this volume.

Volumes location:

• Windows (open in the Explorer): \\wsl$\docker-desktop-data\data\docker\volumes\my-volume

Volume can be located anywhere, the driver will facilitate making the usage of the volume possible.

Volume drivers can enable interactions with a remote filesystem and persist data. Available drivers include: Azure File Storage, NFS, and AWS EFS.

Characteristics of volumes:

• Volume with data can remain after a container has been deleted
• We can attach a volume to another container
• One volume can be used by many containers
• Volumes can be local or remote

Network

• docker network ls - list docker's networks
• docker network inspect bridge - inspect bridge network of docker
• docker network create foo-network - create new bridge network
• docker network rm foo-network - remove network
• docker network prune - remove all custom networks not used by at least one container

Network types

Networks on Docker can come in various forms. The most used formats include the following:

Bridge

It's the default network driver when using Docker. A bridge network on Docker is a software layer providing connectivity between containers that are connected to the same bridge network. Also, those containers are isolated from containers that are not connected to that network.

• Run nginx container: docker run --rm -p 8080:80 --name nginx-my nginx
• Check the network that the container is attached to: docker inspect nginx-my. Find value Networks.bridge.NetworkID
• Ensure that we have bridge network with the same id: docker network ls

Host

By using the Docker host, the container is not isolated from it. The container does not have its own IP address. The container will be accessed using the host IP.

• Run nginx instance using the host network: docker run --rm --network host --name nginx-my nginx
• Ensure that the container doesn't have an IP assigned. Check section NetworkSettings.Networks: docker inspect nginx-my.
• Run bash container: docker run -it --network host --rm --name nginx-wget bash
• Try to get access to localhost: wget -O- localhost

Here, instead of attaching the containers to a network, we mounted them to the host network. As expected, the container doesn't have an IP assigned; therefore, any requests to localhost are mapped to the host.

Overlay

Bridge, host networks centered on Docker Engine installed on one host. Using Docker on a production with an orchestration engine such as Swarm requires the use of multiple hosts. The overlay network creates a distributed network among those hosts. An overlay network will transparently join the host networks and create a unified network on top of them.

Default address pools

By default, docker creates networks in these pools:

• 172.17.0.0/16 - 172.31.0.0/16
• 192.168.0.0/20 - 192.168.240.0/20

These IP addresses may conflict with server's IP address. You may lose SSH connection trying to docker-compose up. In this case create file /etc/docker/daemon.json:

{
  "default-address-pools" : [
    {
      "base" : "172.31.0.0/16",
      "size" : 24
    }
  ]
}

In this case created networks will have Subnet: 172.31.1.0/24, 172.31.2.0/24, .., 172.31.255.0/24.

Useful commands

Copy config from working container

docker run -d httpd:2.4
docker ps
docker cp [ID of container]:/usr/local/apache2/conf/httpd.conf ./httpd.conf

docker-compose run php bash
docker ps # See Container ID
docker cp 164b0441bb1c:/usr/local/etc/php/php.ini-development ./

docker_swarm.md

Docker Swarm

• docker node ls - show virtual machines in the cluster
• docker service ls - show services in cluster
• docker node inspect jenkins - show detail info for node
• docker node update jenkins --label-add db - add label db for node jenkins
• docker service ps site_api-php - show replicas' info for one serivce

dockerfile.md

Dockerfile

See official Dockerfile reference

• Each image has base image layer, e.g. FROM debian:12-slim
• Each instruction in Dockerfile creates Instruction Layer, e.g. RUN apt update && apt install iputils-ping -y
• On subsequent image build Docker use cached layers. You'll see in logs:

  #5 [2/2] RUN apt update && apt install iputils-ping -y
  #5 CACHED
  

• Put frequently changed commands at the bottom of Dockerfile to get maximum benefit from cached layers.

To build image run this command in the same directory where Dockerfile is located:

docker build -t myimage .

Run container based on our new image:

docker run -it --rm myimage

Example 1

# base image
FROM python:3-onbuild

# This port the container should expose
EXPOSE 5000

# Tell the container which command it should run when it's started
CMD ['python', './app.py']

Example 2

FROM nginx:latest

ADD ./nginx/default.conf /etc/nginx/conf.d/default.conf
WORKDIR /var/www/frontend

Example 3

FROM debian:12-slim

RUN apt update && apt install iputils-ping -y

CMD ["echo", "Hello!"]

Example 4. Python app

File requirements.txt:

requests==2.31.0

File app.py:

print("Hello!")

File Dockerfile:

FROM python:3.9-alpine

ENV APP_NAME="Hey Python!"

WORKDIR /app

RUN apk update

COPY requirements.txt requirements.txt

COPY app.py app.py

RUN pip install -r requirements.txt

CMD python app.py

Dockerfile commands

• RUN - executes command(s) in a new layer and creates a new image. E.g., it is often used for installing software packages.
• CMD ["echo", "Hi"] (exec form, preferred), CMD echo "Hi" (shell form) - sets default command and/or parameters, which can be overwritten from command line when docker container runs.
• ENTRYPOINT ["python", "app.py"] (exec form, preferred), ENTRYPOINT python app.py (shell form) - configures a container that will run as an executable.
• COPY ./index.html /var/www/html/index.html - copy file from current directory to a container

Debug Dockerfile

Place this code in the Dockerfile to prevent further build process:

# ...
RUN exit 1;

# .. Instructions below will not be executed

Look at the last console output:

/usr/local/share/doc/openssl/html/man7/x509.html
Removing intermediate container e535156b723f
 ---> 3402f3207c06
Step 8/21 : RUN exit 1;
 ---> Running in 0c00f6ae64fb
The command '/bin/sh -c exit 1;' returned a non-zero code: 1

You can run saved contaner 3402f3207c06 before an error occured and try to find out what happened:

docker run -it 3402f3207c06 bash

errors.md

Docker Errors

ERROR: network site_default is ambiguous

docker network ls
docker network rm e2d790be0302