davecan/docker-cheatsheet.md

## docker-cheatsheet.md

      
    Raw
  

              docker-cheatsheet.md
            
          
    Super simple docker cheatsheet.
Build & deploy an app

First build the image:
docker build -t <desired_image_name> .

Example from when I built a container having a TCP/IP client script and TCP/IP server script, so I wanted a generic name:
docker build -t tcpnode .

This may take a while the first time.
Once the image is built it can be run:
docker run -it -p 5000:5000 --name myserver tcpnode

-p      exposes docker client port to the host
-it     runs in interactive terminal mode
-d      runs in detached (headless) mode, manage w/ ps -a, start/stop/kill etc
-v      mounts a volume for shared storage
        using COPY/ADD in the Dockerfile only copies the folder contents to the container
        using -v actually mounts the specified folder as a folder in the container
        so writes in the host can be seen immediately in the container and vice-versa
        this can be useful for writing code in the host but running tests/devops/etc in a container
--name  gives this running instance a name 
        so e.g. we could run myserver 1, myserver 2, etc all based on the tcpserver base image

In that example, because it is run with -it we can run arbitrary commands in it from the command prompt.
To run a script/program automatically when the container starts we can use a CMD command in the Dockerfile.
Alternately, if we want to run multiple containers, each based on the same image but each running a different script, we do not have to create a separate Dockerfile for each.
Instead we can pass the script/app startup command as a parameter:
docker run -it --name myserver tcpnode python tcpServer.py
docker run -it --name myclient tcpnode python tcpClient.py

That starts up two containers, one for the client and one for the server, both from the same tcpnode image built previously.
Once the image is no longer needed, shut it down (logout, stop/kill, etc) and then remove it:
docker ps -a   # if we want to see what is running before we kill it
docker rm tcpnode

More on docker run here: https://blog.codeship.com/the-basics-of-the-docker-run-command/
How to properly use apt-get in a Dockerfile

Put both apt-get update and apt-get install in the SAME RUN COMMAND.
RUN apt-get update && apt-get install -y \
  bzr \
  cvs \
  git \
  mercurial \
  subversion


This is done to make sure that the latest packages will be installed. If apt-get install were in a separate RUN instruction, then it would reuse a layer added by apt-get update, which could had been created a long time ago.

RUN creates and commits new layer each time it executes.
RUN vs CMD vs ENTRYPOINT

http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/

Use RUN to alter the image by creating and commiting a new layer - ex: running apt-get to configure the container.
Use CMD to set a default command which can be overridden by passing a command to docker run on the command line.

Only the last CMD in the Dockerfile is actually executed, and only if there is no command passed to docker run.


Use ENTRYPOINT to configure the container to run as an executable.

ENTRYPOINT commands are not ignored when a command is passed to docker run -- unlike CMD


Prefer ENTRYPOINT to CMD when building executable Docker image and you need a command always to be executed.

There are two forms for the commands: Shell form and Exec form.
Shell form:
<instruction> <command>

RUN apt-get install python3
CMD echo "Hello world"
ENTRYPOINT echo "Hello world"

Shell form executes /bin/sh -c <command> behind the scenes, so the RUN command above is actually:
/bin/sh -c apt-get install python3

To use environment variables in a script run from within a Dockerfile, use shell form.
Exec form:
<instruction> ["executable", "param1", "param2", ...]

RUN ["apt-get", "install", "python3"]
CMD ["/bin/echo", "Hello world"]
ENTRYPOINT ["/bin/echo", "Hello world"]

Shell processing does not occur in exec form. So for example environment variables are not accessible from scripts run from within the Dockerfile.
Shell form is preferred for CMD and ENTRYPOINT.
Either form is ok for RUN.
To run /bin/bash from within a Dockerfile: Use the exec form.
ENV name John Dow
ENTRYPOINT ["/bin/bash", "-c", "echo Hello, $name"]

Quickly testing if booted services work correctly

From the takacsmark tutorial this technique can be helpful for testing a web app where we haven't yet created a gui interface. His example is a Flask app container that stores data in a Redis data store container.
# add a name to the data store
$ curl --header "Content-Type: application/json" \
--request POST \
--data '{"name":"Kumar"}' \
localhost:5000

{
  "name": "Kumar"
}

# retrieve list of names from the data store
$ curl localhost:5000

[
  "{'name': 'Kumar'}"
]

This is a quick and easy way to validate the two containers are connected without having to build a gui.
Manage images

To see what images are installed locally:
docker image ls

To remove a local image:
docker image rm imagename:tagname

After a while a bunch of anonymous images may be in the list. To remove all unused images:   (will show warning)
docker image prune

Manage networks

To view all current networks:
docker network ls

For more details:
docker network inspect <network_name>

ex:  docker network inspect bridge

inspect returns a JSON object with the configuration details.
To specify which network a container should use when we run it:
docker run <container_name> --net=<network_name>

Container Networking Overview

Good resources:

https://runnable.com/docker/basic-docker-networking
https://mesosphere.com/blog/networking-docker-containers/  (excellent diagrams!)
https://docs.docker.com/docker-for-mac/networking/

(quotes come from a variety of sources including the above)
IMPORTANT: Docker for Mac network limitations

(src: https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds)

We cannot see the docker0 interface on the Docker for Mac host as it is contained entirely in the virtual machine running silently in the background.
Docker for Mac cannot route traffic from the host to the containers, so we can't ping a container.

To connect FROM a container TO a service on the Mac host:

The host has a changing IP address (or none if you have no network access). From 18.03 onwards our recommendation is to connect to the special DNS name host.docker.internal, which resolves to the internal IP address used by the host. This is for development purpose and will not work in a production environment outside of Docker for Mac.
The gateway is also reachable as gateway.docker.internal.

Remember: gateway == network border router.
To connect TO a container FROM the Mac host:

Port forwarding works for localhost; --publish, -p, or -P all work. Ports exposed from Linux are forwarded to the host.
Our current recommendation is to publish a port, or to connect from another container. This is what you need to do even on Linux if the container is on an overlay network, not a bridge network, as these are not routed.


Important: Per the takacsmark tutorial and video below, containers in the same network can refer to each other using the service name (specified in docker-compose.yml) as host names. It can be useful when experimenting with new services to find out what it expects other services to be named by default and use that default as the corresponding service name. In his example, he checked inside the Wordpress code and instructions to discover that the default name for the backend MySQL database is mysql so he gave the service the name mysql in the docker-compose.yml file. That way it just works out of the box which is what we want for experimentation.
Docker can manage multiple networks which is useful when spinning up containers we want to network together. Each container can be assigned to a network, so we can have multiple containers running on different networks on the same host.
Docker creates 3 networks automatically: bridge, host, and none.
Note: It is best to create a custom network instead of using one of the builtins.
Bridge network

This is the default network and the simplest to use. All containers connect here unless explicitly connected to another network.

Docker automatically creates a subnet and gateway for the bridge network, and docker run automatically adds containers to it. Any containers on the same network may communicate with one another via IP addresses (ONLY). Docker does not support automatic service discovery on bridge. You must connect containers together with the --link option in your docker run command.
The Docker bridge supports port mappings and docker run --link allowing communications between containers on the docker0 (en0 etc) network. However, these error-prone techniques require unnecessary complexity. Just because you can use them, does not mean you should. It’s better to define your own networks instead.

Note: I read elsewhere that creating our own network is the best practice for a litany of reasons, including the fact that adopting the basic bridge network requires exposing all ports to the world or none. A custom network allows us to expose only what we want. (so creating a custom network is similar to creating a firewall?)
None network


This offers a container-specific network stack that lacks a network interface. This container only has a local loopback interface (i.e., no external network interface).

So in other words a container using the none network has no network connectivity with any other container nor with the host.
Host network


This enables a container to attach to your host’s network (meaning the configuration inside the container matches the configuration outside the container).

This is essentially allowing the container to run on our host's network. So running a container here should make it reachable from other computers on our network.
Network connectivity limitations


You can create multiple networks with Docker and add containers to one or more networks.

So a single container can be hooked to multiple networks, this is important to understand because:

Containers can communicate within networks but not across networks.

So to have two containers communicate they must be attached to the same network.

A container with attachments to multiple networks can connect with all of the containers on all of those networks.

This part is interesting:

This lets you build a “hub” of sorts to connect to multiple networks and separate concerns.

Port mapping

To forward a host port to a container port:
docker run -p host_port_num:container_port_num ...

So to run a container having a web server with connections from the host on port 8000 forwarded to port 80 on the container:
docker run -p 8000:80 ...

Custom networks

Building a new network based on the default bridge network is the easiest way to set up an isolated network for a set of containers.
To create a custom network based on the default bridge network type:
docker network create --driver bridge my_network

To view the network details:
docker network inspect my_network

Inspecting will also show the containers that are currently wired into the network.
There is a command that can attach/detach containers to a network after the containers are started but I can't recall offhand what that is.
Service names as local references in networks

All containers in the same docker network can reference each other using service names as if they were machine names.
For example from the takacsmark tutorial there is a redis service in the docker-compose.yml file, and the app.py app points to the redis "server" by using this line:
redis = Redis(host="**redis**", db=0, socket_timeout=5, charset="utf-8", decode_responses=True)

If we rename the service in the docker-compose.yml file we must rename it in all references in the app as well.

Docker Compose

Docs: https://docs.docker.com/compose/compose-file/
Orchestrates a cluster of containers.

Best Tutorial: https://takacsmark.com/docker-compose-tutorial-beginners-by-example-basics/

His video walkthrough: https://www.youtube.com/watch?v=4EqysCR3mjo


Docker Compose file reference: https://docs.docker.com/compose/compose-file/

Important: This is for both Docker Compose and Docker Swarm! Some features here (e.g. secret, deploy) only apply to Docker Swarm while others (e.g. build) only apply to Docker Compose. (Swarm only accepts pre-built images) The documentation includes both but not everything is applicable to both.


Limitation: Docker Compose only composes clusters on a single machine. It cannot manage containers across multiple machines. To manage containers across machines use Docker Swarm or Kubernetes. Docker Swarm uses the same docker-compose.yml file, so we can develop locally with Docker Compose and deploy to a cloud environment with Docker Swarm.
Desired vs actual state

docker-compose.yml describes the desired state, but during execution the system's actual state may deviate.
Built into the system is an ability to bring the entire stack back into the desired state:
docker-compose up

or for detached mode of course:
docker-compose up -d


if an image is already running and if it is in the desired state then it is not touched
if an image is already running and has deviated from the desired state then it is destroyed and recreated
if an image from the docker-compose.yml file does not exist in the stack it is created

Managing orchestrated services

Many commands are very similar to their counterparts in docker except they work across the entire stack unless a specific service is named in the command.
Use docker-compose up and docker-compose down to spin up the stack and to teardown and destroy it, respectively. It can setup and teardown containers, networks, images, and volumes.
Use docker-compose ps to view a list of containers (started & stopped) in our stack, the commands they are running, and their port mappings.
Use docker-compose top to list the top process for each service in the stack.
Use docker-compose logs -f to follow the logs from both services.
Use docker-compose logs -f <service_name> to follow the logs from only one service.
Use docker-compose stop to stop all containers, docker-compose start to start them again, or docker-compose restart to do both together.
Use docker-compose exec <service_name> <command> similar to docker exec, to execute <command> against the running service.
There are other useful commands such as kill, rm, etc.
To "restart" a stack that has been modified

docker-compose down will tear down the entire stack and dispose of them.
docker-compose up --build will do a rebuild first if the Dockerfile or image have changed before spinning the stack up.
docker-compose build --no-cache will force a rebuild regardless of change status, and we can follow it up with a normal docker-compose up -d command.
Defining networks and volumes

We can define networks and volumes in the docker-compose.yml file using the networks and volumes keywords.
Using named volumes gives us the flexibility of starting and destroying containers that use persistent data volumes, so we can share state between containers (e.g. in a DevOps pipeline) or between instances of the same container image. (e.g. spawning multiple Redis data stores to store state, with state shared via persistent named volume)
run vs exec


exec runs the command in a running container
run starts up a completely new container to run the command

Enter a shell in a specific container

docker-compose exec <service_name> /bin/bash
To enter a shell in a new container based on a service (from docker-compose.yml) use docker-compose run <service_name> /bin/bash.
Using a custom Docker image in a service

Use the build section under the service definition in docker-compose.yml. It defines the build context, the build args, the dockerfile location and even the target build stage in a multi-stage build.
Automating docker run

Docker Compose can automate docker run because we can specify runtime paraneters in a separate Config file.
Passing build arguments to a service

Use the args element under build. Arguments defined in args are passed to the Dockerfile. Inside the Dockerfile the argument must be defined in the ARG instruction, after which it can be used by prepending a dollar sign.
This makes the Dockerfile more generic and allows us to drive the configuration from the docker-compose.yml file.
Ex from takacsmark:
**In docker-compose.yml**

...
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - PYTHON_VERSION=3.7.0-alpine3.8
    ...

**In Dockerfile**

ARG PYTHON_VERSION
FROM python:$PYTHON_VERSION

Put variables in a separate file

It is of course a bad idea to put sensitive info in the Dockerfile or docker-compose.yml file.
The better way is to point docker-compose.yml to a file containing the environment variable values, that way we can store them separately and securely.
Use the env_file directive inside of a service definition in docker-compose.yml to specify the name of the file to use:
env_file:
  - .env.txt

This further abstracts out the variables from the docker-compose.yml just like putting them in docker-compose.yml abstracts them out from the Dockerfile.
Likewise if we create a file named .env in the same folder as docker-compose.yml it should automatically be picked up by Docker Compose and can be used to define variables within docker-compose.yml itself.
Example:
(in docker-compose.yml)
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - PYTHON_VERSION=${PYTHON_VERSION}   # notice the braces! 
      ...
  redis:
    image: redis:${REDIS_VERSION}

And the corresponding .env file:
PYTHON_VERSION=3.7.0-alpine3.8
REDIS_VERSION=4.0.11-alpine

Now when docker-compose up is run, the environment variables will be read from the .env file, preprocessed into docker-compose.yml, and then docker-compose.yml will be used to orchestrate the spinup of the stack. This means the variables defined in .env will be passed through to the Dockerfile, and the resulting containers will be configured as specified in the .env file.
Easily scaling up and down

To run N copies of a single container:
docker-compose up --scale <service_name>=N

Ex:  docker-compose up --scale app=3    # for service named 'app' in docker-compose.yml

Note: If we specified a custom container_name then we cannot scale up past one container!
Note: If we map ports in a Dockerfile we will get errors on the above because we can't spawn multiple copies that map the same port.
This can be useful to have one service that listens on a port and then have multiple worker containers in the background. The frontend container listening on the port then manages distributing the workload among the worker containers as needed.
For example, we have a web app container that allows a user to upload an image to be resized, and it passes off the image to a worker container behind the scenes.
To increase from 3 worker containers (based on a single service named worker in docker-compose.yml) to 6 in response to increased load we can do this:
docker-compose up --scale worker=6

Then Docker Compose will spin up 6 workers.
Presumably we need a means to register the workers with the web service so the service can hand off work to the new workers, and a way to monitor worker load.
Starting dependencies in order

Specify service dependencies with the depends_on directive in docker-compose.yml and those service containers will be started in the specified order before the enclosing service.
Note: Docker does not guarantee the dependency services are functional and ready to be used only that the containers have started.
Overriding docker-compose.yml directives

Put a file named docker-compose.override.yml in the same folder as docker-compose.yml.
When we run docker-compose up, any directives in docker-compose.override.yml will override their corresponding directives in docker-compose.yml.
Caveat: items in YAML lists are appended, not necessarily overridden.
Use multiple docker-compose.yml files as an Architecture Pattern

We can stack files to set up various environments, e.g. docker-compose.dev.yml, docker-compose.test.yml etc.
Start a stack based on a specific docker-compose.yml file

If we have multiple docker-compose.yml files we can opt to use only one of them during a stack deployment:
docker-compose -f docker-compose.yml up -d

We can apply multiple files in sequence to configure our stack precisely to our environment:
docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.dev.yml up -d

This will spin up the stack by running each of the files in sequence to give us our desired dev environment.
Copying a stack across multiple computers

We can't orchestrate a stack across multiple computers, but we can copy the stack to other computers.
Use docker-compose push to push the stack images to the selected central repository (e.g. Docker Hub) and docker-compose pull to pull it down to another computer. Then use docker-compose up etc to run it on the other computer.