Skip to content

Instantly share code, notes, and snippets.

@spkane

spkane/class-docker-4-weeks.txt Secret

Last active June 14, 2024 18:25
Show Gist options
  • Save spkane/686ae98467a840d73cd3e181da9b1194 to your computer and use it in GitHub Desktop.
Save spkane/686ae98467a840d73cd3e181da9b1194 to your computer and use it in GitHub Desktop.
Download ZIP
Docker in 4 Weeks - DevOps
Raw
class-docker-4-weeks.txt
------------------------------------------------------------------------------------------------------------------------
marp: true theme: techlabs
Docker: Fundamentals
Docker Images
Presented by Sean P. Kane
https://techlabs.sh/ [https://techlabs.sh/]
Release %BUILD_RELEASE%
------------------------------------------------------------------------------------------------------------------------
Instructor
Sean P. Kane [https://www.linkedin.com/in/spkane/]
@spkane [https://twitter.com/spkane]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
https://techlabs.sh [https://techlabs.sh]
!bg right [marp-themes/images/skane-2018-side-black-cropped.jpg]
------------------------------------------------------------------------------------------------------------------------
Follow Along Guide [https://gist.github.com/spkane/a15467f50545483282496624629bc62e]
Textual Slides
------------------------------------------------------------------------------------------------------------------------
O'Reilly Online Sandbox VM [https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469
[https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
NOTE: You must be logged into your O'Reilly account and the VM sessions will expire after 60 minutes!
------------------------------------------------------------------------------------------------------------------------
Prerequisites (1 of 3)
NOTE: You MUST have open access to Github if you want to participate in the hands-on portion of class from your local
computer system.
------------------------------------------------------------------------------------------------------------------------
Prerequisites (2 of 3)
* A recent computer and OS
* Recent/Stable Linux, macOS, or Windows 10+
* root/admin rights
* Sufficient resources to run one 2 CPU VM
* CPU Virtualization MUST be enabled in your BIOS/EFI
* Reliable and fast internet connectivity
* Docker Community/Desktop Edition (on Linux also install Docker Compose V2)
------------------------------------------------------------------------------------------------------------------------
Prerequisites (3 of 3)
* A graphical web browser
* A text editor
* A software package manager
* git client
* General comfort with the command line will be helpful.
* [optional] tar, wget, curl, jq, ssh client
* curl.exe for Windows - https://curl.se/windows/ [https://curl.se/windows/]
------------------------------------------------------------------------------------------------------------------------
A Note for Powershell Users
Terminal commands reflect the Unix bash shell. PowerShell users will need to adjust the commands.
* Unix Shell Variables
* export MY_VAR='test'
* echo ${MY_VAR}
* PowerShell Variables
* $env:my_var = "test"
* Get-ChildItem Env:my_var
* Remove-Item Env:\my_var
------------------------------------------------------------------------------------------------------------------------
Translation Key
\ - Unix Shell Line Continuation ` - Powershell Line Continuation (sort of)
${MY_VAR} - Is generally a placeholder in the slides.
------------------------------------------------------------------------------------------------------------------------
Linux Container Mode
* On the Windows platform make sure that you are running in Linux Container mode
[https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers].
* WARNING: On Windows, avoid using Git Bash. WSL2 is highly recommended instead.
!bg right [marp-themes/images/linux-container-mode.png]
------------------------------------------------------------------------------------------------------------------------
A Note About Proxies & VPNs
Proxies can interfere with some Docker activities if they are not configured correctly and VPNs can increase audio and
video streaming issues in class.
* Configure HTTP_PROXY, HTTPS_PROXY, and your web browser.
If required, you can configure a proxy in Docker Desktop via the preferences.
* Docker [https://dev.to/mcastellin/use-docker-with-proxy-servers-tutorial-10gg]
* Docker-Compose [https://stackoverflow.com/questions/34990458/docker-compose-build-and-http-proxy]
------------------------------------------------------------------------------------------------------------------------
Instructor Environment
* Operating System: macOS (v14.X.X+)
* Terminal: iTerm2 (Build 3.X.X+) - https://www.iterm2.com/ [https://www.iterm2.com/]
* Shell Prompt Theme: Starship - https://starship.rs/ [https://starship.rs/]
* Shell Prompt Font: Fira Code - https://github.com/tonsky/FiraCode [https://github.com/tonsky/FiraCode]
* Text Editor: Visual Studio Code (v1.X.X+) - https://code.visualstudio.com/ [https://code.visualstudio.com/]
* export BUILDKIT_COLORS=run=green:warning=yellow:error=red:cancel=cyan
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker client
* The docker command used to control most of the Docker workflow and talk to remote Docker servers.
* Docker server
* The dockerd command used to launch the Docker daemon. This turns a Linux system into a Docker server that can have
containers deployed, launched, and torn down via a remote client.
------------------------------------------------------------------------------------------------------------------------
The Docker Server
* Virtual Machine or Bare Metal
* In general, the Docker server can only be run on Linux. If you are running Linux on your system, you may have the
server running natively on bare metal. On Windows and macOS, it is common to utilize a Linux virtual machine to run
the server and Docker Desktop Edition takes care of this for you.
* In class, it makes no difference. If I refer to the "virtual machine" or the "docker server" know that these terms
are interchangeable for class and it does not really matter how you are setup locally.
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker/OCI images
* Docker images consist of one or more filesystem layers and some important metadata that represent all the files
required to run a Dockerized application. A single Docker image can be copied to numerous hosts. A container will
typically have both a name and a tag. The tag is generally used to identify a particular release of an image.
* OCI - Open Container Initiative
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Linux Containers
* A Linux Container is a single instantiation of a Docker or OCI-standard image. A specific container can only exist
once; however, you can easily create multiple containers from the same image.
------------------------------------------------------------------------------------------------------------------------
Testing the Docker Setup
$ docker image ls
$ docker container run -d --rm --name quantum \
--publish mode=ingress,published=18080,target=8080 \
docker.io/spkane/quantum-game:latest
$ docker container ls
* In a web browser, navigate to port 18080 on your Docker server.
$ docker container stop quantum
$ docker container ls
------------------------------------------------------------------------------------------------------------------------
Exploring the Dockerfile
$ cd ${HOME}
$ mkdir class
$ cd ${HOME}/class
$ git clone https://github.com/spkane/balance_game.git
$ cd balance_game
* Open & explore Dockerfile in your text editor
Full Documentation: https://docs.docker.com/engine/reference/builder/
[https://docs.docker.com/engine/reference/builder/]
------------------------------------------------------------------------------------------------------------------------
Registering with Docker Hub
Create an account at: https://hub.docker.com/ [https://hub.docker.com/]
------------------------------------------------------------------------------------------------------------------------
Create Your Image Repository
* Login: https://hub.docker.com/
* Click: Create Repository+
* Enter name: balance_game
* Set visibility: public
* Click: Create
------------------------------------------------------------------------------------------------------------------------
Docker Login
$ docker login
$ cat ~/.docker/config.json
------------------------------------------------------------------------------------------------------------------------
Registry Authentication
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "q378t348q7tb78bfs387b==",
"email": "me@example.com"
}
}
}
------------------------------------------------------------------------------------------------------------------------
Building Your First Image
$ export HUB_USER=${USER}
$ export BUILDKIT_COLORS=run=green:warning=yellow:error=red:cancel=cyan
$ docker image build -t docker.io/${HUB_USER}/balance_game:test .
------------------------------------------------------------------------------------------------------------------------
Testing and Uploading the Image
$ docker container run -d --rm --name balance_game \
--publish mode=ingress,published=18081,target=80 \
docker.io/${HUB_USER}/balance_game:test
$ docker container stop balance_game
$ docker image push docker.io/${HUB_USER}/balance_game:test
$ docker search ${HUB_USER}
------------------------------------------------------------------------------------------------------------------------
Docker Hub API Examples
$ curl -s -S \
"https://registry.hub.docker.com/v2/repositories/library/alpine/tags/" \
| jq '."results"[]["name"]' | sort
------------------------------------------------------------------------------------------------------------------------
Launch a Typical Build Process
$ docker image build --progress=plain --no-cache \
-t docker.io/${HUB_USER}/balance_game:test .
------------------------------------------------------------------------------------------------------------------------
A Typical Build Process - Simplified (1 of 2)
1. The client loads the build definition from Dockerfile.
2. The client loads the .dockerignore file.
3. The client transfers the build context to the server.
4. The client loads metadata for the base image.
5. The client authenticates with the registry.
6. The client pulls the base image.
------------------------------------------------------------------------------------------------------------------------
A Typical Build Process - Simplified (2 of 2)
7. New intermediate container created from the base image.
* or an empty container created if using FROM scratch.
8. Command executed inside intermediate container.
9. New image/layer created from the intermediate container snapshot.
10. If there is another step, a new intermediate container is created from the last step, and then the build goes back
to step 7.
------------------------------------------------------------------------------------------------------------------------
Advanced Dockerfile Techniques
------------------------------------------------------------------------------------------------------------------------
Keep it Small
* Every layer’s size matters
* Don’t install unnecessary files
$ cd ${HOME}/class
$ docker container run --rm -d --name outyet-small \
--publish mode=ingress,published=8090,target=8080 \
docker.io/spkane/outyet:latest-small
$ docker container export outyet-small -o export.tar
$ tar -tvf export.tar
$ docker container stop outyet-small
$ rm export.tar
------------------------------------------------------------------------------------------------------------------------
Debugging an Image
* If your image has a shell installed, you can access it using a command like this:
$ docker container run --rm -ti docker.io/spkane/outyet:latest \
/bin/sh
------------------------------------------------------------------------------------------------------------------------
Debugging an Image
* But without a shell in the image, this will fail.
$ docker image ls
$ docker container run --rm -ti docker.io/spkane/outyet:latest-small \
/bin/sh
------------------------------------------------------------------------------------------------------------------------
Debugging an Image
So, let's fix this:
$ git clone https://github.com/spkane/outyet.git
$ cd outyet
------------------------------------------------------------------------------------------------------------------------
Multi-Stage Images
FROM golang:1.9.4 as builder
COPY . /go/src/outyet
WORKDIR /go/src/outyet
ENV CGO_ENABLED=0
ENV GOOS=linux
RUN go get -v -d && \
go install -v && \
go test -v && \
go build -ldflags "-s" -a -installsuffix cgo -o outyet .
------------------------------------------------------------------------------------------------------------------------
Multi-Stage Images
# To support debugging, let's use alpine instead of scratch
FROM alpine:latest as deploy
# Since we are using alpine we can simply install these
RUN apk --no-cache add ca-certificates
WORKDIR /
COPY --from=builder /go/src/outyet/outyet .
# (or) COPY --from=0 /go/src/outyet/outyet .
EXPOSE 8080
CMD ["/outyet", "-version", "1.9.4", "-poll", "600s", "-http", ":8080"]
------------------------------------------------------------------------------------------------------------------------
Building the Improved Image
$ docker image build -f Dockerfile -t outyet:local .
------------------------------------------------------------------------------------------------------------------------
Debugging an Image
* Now that we have a shell, let's try this again:
$ docker image ls
$ docker container run --rm -ti outyet:local /bin/sh
* Once inside the new container:
$ ls -lFa /outyet
$ exit
------------------------------------------------------------------------------------------------------------------------
Debugging a Broken Build (1 of 4)
* Break the Dockerfile and then try building it again.
RUN go getit -v -d && \
go install -v && \
go test -v && \
go builder -ldflags "-s" -a -installsuffix cgo -o outyet .
$ docker image build -t outyet:debug --no-cache .
------------------------------------------------------------------------------------------------------------------------
Debugging a Broken Build (2 of 4)
* Add another stage and build the known working stages.
FROM builder as debug
RUN go getit -v -d && \
go install -v && \
go test -v && \
go builder -ldflags "-s" -a -installsuffix cgo -o outyet .
$ docker image build -t outyet:debug --target builder --no-cache .
------------------------------------------------------------------------------------------------------------------------
Debugging a Broken Build (3 of 4)
* Let's debug the last successful image in that build
$ docker container run --rm -ti outyet:debug
* Once inside the new container:
$ go getit -v -d
$ go get -v -d
$ exit
------------------------------------------------------------------------------------------------------------------------
Debugging a Broken Build (4 of 4)
* Important: Always, remember to apply your fixes to the Dockerfile and remove the extra stage that you added before
committing the changes.
RUN go get -v -d && \
go install -v && \
go test -v && \
go builder -ldflags "-s" -a -installsuffix cgo -o outyet .
------------------------------------------------------------------------------------------------------------------------
Smart Layering
* Each and every layer’s size matters
* Clean up, inside of each step.
$ cd ${HOME}/class
$ cd balance_game
$ docker image build -f Dockerfile.fedora .
$ docker image tag ${IMAGE_ID} size${#}
$ docker image history size${#}
------------------------------------------------------------------------------------------------------------------------
Smart Layering
* edit Dockerfile.fedora, build, and re-examine size
RUN dnf install -y httpd
RUN dnf clean all
RUN dnf install -y httpd && \
dnf clean all
------------------------------------------------------------------------------------------------------------------------
Timing commands in Windows
* In the next exercise we will be timing commands using a Unix utility. If you are on Windows and want to try to time
these commands locally, you can try something like this in Powershell.
PS C:\> $t = Measure-Command { docker image build --no-cache . }
PS C:\> Write-Host That command took $t.TotalSeconds to complete.
------------------------------------------------------------------------------------------------------------------------
Order Matters
* Keep commands that change towards the end of your Dockerfile.
$ cd ${HOME}/class
$ cd balance_game
$ time docker image build --no-cache .
$ time docker image build .
------------------------------------------------------------------------------------------------------------------------
Order Matters
* edit start.sh
$ time docker image build .
------------------------------------------------------------------------------------------------------------------------
Order Matters
* Add to the top of Dockerfile:
ADD start.sh /
------------------------------------------------------------------------------------------------------------------------
Order Matters
* Then, remove the same line from the bottom of the Dockerfile.
$ time docker image build --no-cache .
$ time docker image build .
$ vi start.sh
$ time docker image build .
------------------------------------------------------------------------------------------------------------------------
What We Have Learned
* What a Dockerfile is
* Building a Docker image
* Using Docker Hub
* Keeping Images Small
* Keeping Builds Fast
* Multi-Stage Dockerfiles
* Debugging Images
------------------------------------------------------------------------------------------------------------------------
Additional Reading
* The 12-Factor App
* http://12factor.net/ [http://12factor.net/]
* Official Docker Documentation
* https://docs.docker.com/ [https://docs.docker.com/]
* Docker: Up and Running
* https://dockerupandrunning.com/ [https://dockerupandrunning.com/]
!bg right:40% [marp-themes/images/docker-up-and-running-cover-3e-medium.jpg]
------------------------------------------------------------------------------------------------------------------------
Additional Learning Resources
https://learning.oreilly.com/ [https://learning.oreilly.com/]
------------------------------------------------------------------------------------------------------------------------
Student Survey
Please take a moment to fill out the class survey linked to from the bottom of the ON24 audience screen.
O’Reilly and I value your comments about the class.
Thank you!
------------------------------------------------------------------------------------------------------------------------
Any Questions?
Sean P. Kane [https://www.linkedin.com/in/spkane/]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
Hands-on technical training and engineering
https://techlabs.sh/ [https://techlabs.sh/]
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
marp: true theme: techlabs
Docker: Fundamentals
Linux Containers
Presented by Sean P. Kane
https://techlabs.sh/ [https://techlabs.sh/]
Release %BUILD_RELEASE%
------------------------------------------------------------------------------------------------------------------------
Instructor
Sean P. Kane [https://www.linkedin.com/in/spkane/]
@spkane [https://twitter.com/spkane]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
https://techlabs.sh [https://techlabs.sh]
!bg right [marp-themes/images/skane-2018-side-black-cropped.jpg]
------------------------------------------------------------------------------------------------------------------------
Follow Along Guide [https://gist.github.com/spkane/17aee9422d8235c02417d53d11f42dfa]
Textual Slides
------------------------------------------------------------------------------------------------------------------------
O'Reilly Online Sandbox VM [https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469
[https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
NOTE: You must be logged into your O'Reilly account and the VM sessions will expire after 60 minutes!
------------------------------------------------------------------------------------------------------------------------
Prerequisites (1 of 3)
NOTE: You MUST have open access to Github if you want to participate in the hands-on portion of class from your local
computer system.
------------------------------------------------------------------------------------------------------------------------
Prerequisites (2 of 3)
* A recent computer and OS
* Recent/Stable Linux, macOS, or Windows 10+
* root/admin rights
* Sufficient resources to run one 2 CPU VM
* CPU Virtualization MUST be enabled in your BIOS/EFI
* Reliable and fast internet connectivity
* Docker Community/Desktop Edition (on Linux also install Docker Compose V2)
------------------------------------------------------------------------------------------------------------------------
Prerequisites (3 of 3)
* A graphical web browser
* A text editor
* A software package manager
* git client
* General comfort with the command line will be helpful.
* [optional] tar, wget, curl, jq, ssh client
* curl.exe for Windows - https://curl.se/windows/ [https://curl.se/windows/]
------------------------------------------------------------------------------------------------------------------------
A Note for Powershell Users
Terminal commands reflect the Unix bash shell. PowerShell users will need to adjust the commands.
* Unix Shell Variables
* export MY_VAR='test'
* echo ${MY_VAR}
* PowerShell Variables
* $env:my_var = "test"
* Get-ChildItem Env:my_var
* Remove-Item Env:\my_var
------------------------------------------------------------------------------------------------------------------------
Translation Key
\ - Unix Shell Line Continuation ` - Powershell Line Continuation (sort of)
${MY_VAR} - Is generally a placeholder in the slides.
------------------------------------------------------------------------------------------------------------------------
Linux Container Mode
* On the Windows platform make sure that you are running in Linux Container mode
[https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers].
* WARNING: On Windows, avoid using Git Bash. WSL2 is highly recommended instead.
!bg right [marp-themes/images/linux-container-mode.png]
------------------------------------------------------------------------------------------------------------------------
A Note About Proxies & VPNs
Proxies can interfere with some Docker activities if they are not configured correctly and VPNs can increase audio and
video streaming issues in class.
* Configure HTTP_PROXY, HTTPS_PROXY, and your web browser.
If required, you can configure a proxy in Docker Desktop via the preferences.
* Docker [https://dev.to/mcastellin/use-docker-with-proxy-servers-tutorial-10gg]
* Docker-Compose [https://stackoverflow.com/questions/34990458/docker-compose-build-and-http-proxy]
------------------------------------------------------------------------------------------------------------------------
Instructor Environment
* Operating System: macOS (v14.X.X+)
* Terminal: iTerm2 (Build 3.X.X+) - https://www.iterm2.com/ [https://www.iterm2.com/]
* Shell Prompt Theme: Starship - https://starship.rs/ [https://starship.rs/]
* Shell Prompt Font: Fira Code - https://github.com/tonsky/FiraCode [https://github.com/tonsky/FiraCode]
* Text Editor: Visual Studio Code (v1.X.X+) - https://code.visualstudio.com/ [https://code.visualstudio.com/]
* export BUILDKIT_COLORS=run=green:warning=yellow:error=red:cancel=cyan
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker client
* The docker command used to control most of the Docker workflow and talk to remote Docker servers.
* Docker server
* The dockerd command used to launch the Docker daemon. This turns a Linux system into a Docker server that can have
containers deployed, launched, and torn down via a remote client.
------------------------------------------------------------------------------------------------------------------------
The Docker Server
* Virtual Machine or Bare Metal
* In general, the Docker server can only be run on Linux. If you are running Linux on your system, you may have the
server running natively on bare metal. On Windows and macOS, it is common to utilize a Linux virtual machine to run
the server and Docker Desktop Edition takes care of this for you.
* In class, it makes no difference. If I refer to the "virtual machine" or the "docker server" know that these terms
are interchangeable for class and it does not really matter how you are setup locally.
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker/OCI images
* Docker images consist of one or more filesystem layers and some important metadata that represent all the files
required to run a Dockerized application. A single Docker image can be copied to numerous hosts. A container will
typically have both a name and a tag. The tag is generally used to identify a particular release of an image.
* OCI - Open Container Initiative
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Linux Containers
* A Linux Container is a single instantiation of a Docker or OCI-standard image. A specific container can only exist
once; however, you can easily create multiple containers from the same image.
------------------------------------------------------------------------------------------------------------------------
The Slow Rise Of Linux Containers
* chroot system call (1979)
* Version 7 Unix
* jail (2000)
* FreeBSD 4.0
* Solaris Zones (2004)
* Solaris 10
* Linux Containers - LXC (2008)
* version 2.6.24 of the Linux kernel
------------------------------------------------------------------------------------------------------------------------
The Quick Rise of Docker
* Heroku published 'The 12-Factor App' in 2011
* http://12factor.net/ [http://12factor.net/]
* Docker Engine - Announced March 15, 2013
* Provided:
* Application packaging
* App & dependencies packed together
* API
* Abstract software from hardware
------------------------------------------------------------------------------------------------------------------------
Docker Engine isn’t a…
* virtualization platform (VMware, KVM, etc.)
* cloud platform (AWS, Azure, etc.)
* configuration management tool (Chef, Puppet, etc.)
* deployment framework (Capistrano, etc.)
* development environment (Vagrant, etc.)
* workload management tool (Mesos, Kubernetes, etc.)
------------------------------------------------------------------------------------------------------------------------
Testing the Docker Setup
$ docker image ls
$ docker container run -d --rm --name quantum \
--publish mode=ingress,published=18080,target=8080 \
docker.io/spkane/quantum-game:latest
$ docker container ls
* In a web browser, navigate to port 18080 on your Docker server.
$ docker container stop quantum
$ docker container ls
------------------------------------------------------------------------------------------------------------------------
Network Ports
$ docker container run -d --rm --name quantum \
--publish mode=ingress,published=18090,target=8080 \
docker.io/spkane/quantum-game:latest
* In a web browser, navigate to port 18090 on your Docker server.
* (e.g.) http://127.0.0.1:18090/ [http://127.0.0.1:18090/]
$ docker container stop quantum
------------------------------------------------------------------------------------------------------------------------
Re-Launching Your Container
$ docker container run -d --rm --name quantum \
--publish mode=ingress,published=18080,target=8080 \
docker.io/spkane/quantum-game:latest
------------------------------------------------------------------------------------------------------------------------
Configuration
* The default is often to use environment variables.
* See: https://github.com/spkane/balance_game/blob/main/start.sh
[https://github.com/spkane/balance_game/blob/main/start.sh]
$ docker container run --rm --name balance_game \
--publish mode=ingress,published=18090,target=80 \
docker.io/spkane/balance_game:latest
* http://127.0.0.1:18090/ [http://127.0.0.1:18090/]
------------------------------------------------------------------------------------------------------------------------
Configuration
$ docker container stop balance_game
$ docker container run -d --rm --name balance_game \
--publish mode=ingress,published=18090,target=80 \
-e THEME=2 \
docker.io/spkane/balance_game:latest
* http://127.0.0.1:18090/ [http://127.0.0.1:18090/] (new private/incognito window)
$ docker container stop balance_game
------------------------------------------------------------------------------------------------------------------------
Logs: Accessing STDOUT/STDERR
* Log everything to STDOUT / STDERR
* Avoid writing logs into a file
$ docker container logs --help
$ docker container logs -f quantum
* Reload the game in your web browser.
* (e.g.) http://127.0.0.1:18080/ [http://127.0.0.1:18080/]
$ docker container logs -t quantum
------------------------------------------------------------------------------------------------------------------------
Logs: Foreground Container
$ docker container run --rm --name quantum \
--publish mode=ingress,published=18099,target=8080 \
docker.io/spkane/quantum-game:latest
* [Control-C] to exit.
------------------------------------------------------------------------------------------------------------------------
Statistics
$ docker container stats --help
$ docker container run --rm -d --name stress \
docker.io/spkane/train-os:latest \
stress -v --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 360s
$ docker container stats stress
$ docker container stats --no-stream stress
$ docker container stats
$ docker container top stress
------------------------------------------------------------------------------------------------------------------------
Statistics API
* Try the API:
$ curl --no-buffer -XGET --unix-socket /var/run/docker.sock \
http://docker/containers/stress/stats
------------------------------------------------------------------------------------------------------------------------
Events
$ docker container stop stress
$ docker system events --help
$ docker system events --since $(date +%Y%m%d)
$ docker container run debian sleep 5
* Try the API:
$ curl --no-buffer -XGET --unix-socket /var/run/docker.sock \
http://docker/events
------------------------------------------------------------------------------------------------------------------------
Debugging a Live Container
* If your container has a shell installed you can access it using a command like this:
$ docker container ls
$ docker container exec -ti quantum /bin/sh
# ps auxwww
# exit
$ docker container top quantum
------------------------------------------------------------------------------------------------------------------------
Docker Debug
* https://docs.docker.com/reference/cli/docker/debug/ [https://docs.docker.com/reference/cli/docker/debug/]
* Beta and might eventually require a license.
$ docker run --name hello hello-world
$ docker debug hello
docker > ls /
docker > install ripgrep
docker > uninstall ripgrep
docker > entrypoint
docker > builtins
docker > exit
------------------------------------------------------------------------------------------------------------------------
System Cleanup
$ docker system prune
$ docker system prune -a
------------------------------------------------------------------------------------------------------------------------
Stopping Quantum
$ docker container stop quantum
------------------------------------------------------------------------------------------------------------------------
Local Host Storage 1/4
$ git clone https://github.com/spkane/docker-volumes-example.git
$ cd docker-volumes-example
$ docker image build -t volumes .
$ docker container run --rm --name volumes \
--publish mode=ingress,published=18080,target=80 volumes
* Browse to: http://127.0.0.1:18080 [http://127.0.0.1:18080]
* private/incognito window is best
------------------------------------------------------------------------------------------------------------------------
Local Host Storage 2/4
$ docker container stop volumes
$ docker container run --rm --name volumes \
--publish mode=ingress,published=18080,target=80 \
--mount type=bind,\
source="${PWD}/volumes/single-file/1.jpg",\
target=/usr/local/apache2/htdocs/images/1.jpg \
volumes
* Browse to: http://127.0.0.1:18080 [http://127.0.0.1:18080]
* private/incognito window is best
* Note that the #1 image has changed.
------------------------------------------------------------------------------------------------------------------------
Local Host Storage 3/4
$ docker container stop volumes
$ docker container run --rm --name volumes \
--publish mode=ingress,published=18080,target=80 \
--mount type=bind,\
source="${PWD}/volumes/directory",\
target=/usr/local/apache2/htdocs/images \
volumes
* Browse to: http://127.0.0.1:18080 [http://127.0.0.1:18080]
* private/incognito window is best
* Note that the number images are now letters.
------------------------------------------------------------------------------------------------------------------------
Local Host Storage 4/4
$ cp ${PWD}/volumes/single-file/1.jpg ${PWD}/volumes/directory/1.jpg
* Refresh the last browser window and see the image change.
* Note that the first image is now a number again.
$ docker container stop volumes
------------------------------------------------------------------------------------------------------------------------
Linux Namespaces
* Mount (filesystem resources)
* UTS (host & domain name)
* IPC (shared memory, semaphores)
* PID (process tree)
* Network (network layer)
* User (user and group IDs)
* Time (clock offsets)
* Linux 5.6+ (2020)
------------------------------------------------------------------------------------------------------------------------
Control Groups (cgroups)
* Resource limiting
* Prioritization
* Accounting
* Control
------------------------------------------------------------------------------------------------------------------------
Stressing the System
$ docker container run --rm -ti docker.io/spkane/train-os:latest stress \
-v --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 360s
$ docker container run docker.io/spkane/alpine-base:latest ps auxwww
$ docker container run -it --pid=host \
docker.io/spkane/alpine-base:latest sh
# ps auxwww
# htop -p $(pgrep stress | tr '\n' ',')
# exit
------------------------------------------------------------------------------------------------------------------------
CPU Quotas
$ docker container run -d --cpus="2" docker.io/spkane/train-os:latest \
stress -v --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 60s
$ docker container run -d --cpus=".25" docker.io/spkane/train-os:latest \
stress -v --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 60s
$ docker container run -d --cpus="10" docker.io/spkane/train-os:latest \
stress -v --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 60s
------------------------------------------------------------------------------------------------------------------------
Memory Quotas
* On the O'Reilly VM sandbox, you may need to run swapoff -a before running these commands, and swapon -a when you are
done.
$ docker container run --rm -ti --memory="512m" \
docker.io/spkane/train-os:latest \
stress -v --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 10s
$ docker container run --rm -ti --memory="100m" \
docker.io/spkane/train-os:latest \
stress -v --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 10s
------------------------------------------------------------------------------------------------------------------------
Kernel Debug logs
$ docker container run -it debian \
nsenter -t 1 -m -u -n -i sh
$ docker container run -it --pid=host debian \
nsenter -t 1 -m -u -n -i sh
$ docker container run -it --privileged --pid=host debian \
nsenter -t 1 -m -u -n -i sh
# dmesg
# exit
------------------------------------------------------------------------------------------------------------------------
Timing commands in Windows
* In the next exercise we will be timing commands using a Unix utility. If you are on Windows and want to try to time
these commands locally, you can try something like this in Powershell.
PS C:\> $t = Measure-Command `
{ docker container run -ti --rm docker.io/spkane/train-os:latest `
bonnie++ -u 500:500 -d /tmp -r 1024 -s 2048 -x 1 }
PS C:\> Write-Host That command took $t.TotalSeconds to complete.
------------------------------------------------------------------------------------------------------------------------
I/O Quotas
$ docker image pull docker.io/spkane/train-os:latest
$ time docker container run -ti --rm docker.io/spkane/train-os:latest \
bonnie++ -u 500:500 -d /tmp -r 1024 -s 2048 -x 1
$ time docker container run -ti --rm --device-write-iops /dev/vda:256 \
docker.io/spkane/train-os:latest bonnie++ \
-u 500:500 -d /tmp -r 1024 -s 2048 -x 1
$ time docker container run -ti --rm --device-write-bps /dev/vda:100mb \
docker.io/spkane/train-os:latest bonnie++ \
-u 500:500 -d /tmp -r 1024 -s 2048 -x 1
------------------------------------------------------------------------------------------------------------------------
Container UID
$ docker container run docker.io/spkane/train-os:latest id
$ docker container run docker.io/spkane/train-os:latest grep 500 /etc/passwd
$ docker container run -u 500:1000 docker.io/spkane/train-os:latest id
$ docker container run -u 65534:65534 docker.io/spkane/train-os:latest id
$ docker container run docker.io/spkane/train-os:latest cat /etc/shadow
$ docker container run -u 65534:65534 docker.io/spkane/train-os:latest \
cat /etc/shadow
* Rootless mode
* https://docs.docker.com/engine/security/rootless/ [https://docs.docker.com/engine/security/rootless/]
------------------------------------------------------------------------------------------------------------------------
Dockerfile UID
Set a default non-root UID in your Dockerfiles.
* USER <UID>[:<GID>] or
* USER <user>[:<group>]
e.g. USER myuser:mygroup
------------------------------------------------------------------------------------------------------------------------
The Dangers of Privileged Containers
$ docker container run -ti --rm docker.io/spkane/train-os:latest \
ntpdate pool.ntp.org
$ docker container run -ti --rm --privileged=true \
docker.io/spkane/train-os:latest ntpdate pool.ntp.org
$ docker container run -ti --rm --privileged=true \
docker.io/spkane/train-os:latest bash -c \
"mount /dev/vda1 /mnt && ls -F /mnt/docker/volumes | head -n 10"
------------------------------------------------------------------------------------------------------------------------
Capabilities & SECCOMP Mode
* Linux Kernel Capabilities
* http://man7.org/linux/man-pages/man7/capabilities.7.html [http://man7.org/linux/man-pages/man7/capabilities.7.html]
* Secure Computing Mode
* https://docs.docker.com/engine/security/seccomp/ [https://docs.docker.com/engine/security/seccomp/]
------------------------------------------------------------------------------------------------------------------------
Controlling Capabilities
$ docker container run -ti --rm --cap-add=SYS_TIME \
docker.io/spkane/train-os:latest ntpdate pool.ntp.org
$ docker container run -ti --rm --cap-add=SYS_TIME \
docker.io/spkane/train-os:latest bash -c \
"mount /dev/vda1 /mnt && ls -F /mnt/docker/volumes | head -n 10"
$ docker container run -ti --rm docker.io/spkane/train-os:latest \
tcpdump -i eth0
$ docker container run -ti --rm --cap-drop=NET_RAW \
docker.io/spkane/train-os:latest tcpdump -i eth0
------------------------------------------------------------------------------------------------------------------------
Secure Computing Mode
$ docker container run -ti --rm --cap-drop=NET_RAW \
docker.io/spkane/train-os:latest strace tcpdump -i eth0 \
| grep "Operation not permitted"
$ wget \
https://raw.githubusercontent.com/spkane/train-os/master/default.json
$ wget \
https://raw.githubusercontent.com/spkane/train-os/master/tcpdump.json
$ diff -u default.json tcpdump.json
$ docker container run -ti --rm --security-opt=seccomp:tcpdump.json \
docker.io/spkane/train-os:latest tcpdump -i eth0
------------------------------------------------------------------------------------------------------------------------
What We Have Learned
* History of Containers
* Container Logging
* Container Statistics
* Container Events
* Debugging Containers
* Kernel Functionality
* Controlling Resource Usage
* Improving Container Security
------------------------------------------------------------------------------------------------------------------------
Additional Reading
* The 12-Factor App
* http://12factor.net/ [http://12factor.net/]
* Official Docker Documentation
* https://docs.docker.com/ [https://docs.docker.com/]
* Docker: Up and Running
* https://dockerupandrunning.com/ [https://dockerupandrunning.com/]
!bg right:40% [marp-themes/images/docker-up-and-running-cover-3e-medium.jpg]
------------------------------------------------------------------------------------------------------------------------
Additional Learning Resources
https://learning.oreilly.com/ [https://learning.oreilly.com/]
------------------------------------------------------------------------------------------------------------------------
Student Survey
Please take a moment to fill out the class survey linked to from the bottom of the ON24 audience screen.
O’Reilly and I value your comments about the class.
Thank you!
------------------------------------------------------------------------------------------------------------------------
Any Questions?
Sean P. Kane [https://www.linkedin.com/in/spkane/]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
Hands-on technical training and engineering
https://techlabs.sh/ [https://techlabs.sh/]
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
marp: true theme: techlabs
Docker: Up and Running
Docker Compose
Presented by Sean P. Kane
https://techlabs.sh/ [https://techlabs.sh/]
Release %BUILD_RELEASE%
------------------------------------------------------------------------------------------------------------------------
Instructor
Sean P. Kane [https://www.linkedin.com/in/spkane/]
@spkane [https://twitter.com/spkane]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
https://techlabs.sh [https://techlabs.sh]
!bg right [marp-themes/images/skane-2018-side-black-cropped.jpg]
------------------------------------------------------------------------------------------------------------------------
Follow Along Guide [https://gist.github.com/spkane/f5cf4b97b13d372ef9622346952e990f]
Textual Slides
------------------------------------------------------------------------------------------------------------------------
O'Reilly Online Sandbox VM [https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469
[https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
NOTE: You must be logged into your O'Reilly account and the VM sessions will expire after 60 minutes!
------------------------------------------------------------------------------------------------------------------------
O'Reilly Online Sandbox VM
* If your VM session disconnects mid-session, you can use this command block to get back into a working state.
git clone https://github.com/spkane/rocketchat-hubot-demo.git && \
cd rocketchat-hubot-demo/compose/ && \
docker compose build && cd ../mongodb/data_volume_image && \
docker volume create mongodb-rocketchat && \
docker run --rm -v mongodb-rocketchat:/bitnami/mongodb \
-v ${PWD}:/backup \
busybox tar -xzvf /backup/mongodb-rocketchat.tgz \
-C /bitnami/mongodb && \
cd ../../compose/ && docker compose up -d --build
------------------------------------------------------------------------------------------------------------------------
Prerequisites (1 of 3)
NOTE: You MUST have open access to Github if you want to participate in the hands-on portion of class from your local
computer system.
------------------------------------------------------------------------------------------------------------------------
Prerequisites (2 of 3)
* A recent computer and OS
* Recent/Stable Linux, macOS, or Windows 10+
* root/admin rights
* Sufficient resources to run one 2 CPU VM
* CPU Virtualization MUST be enabled in your BIOS/EFI
* Reliable and fast internet connectivity
* Docker Community/Desktop Edition (on Linux also install Docker Compose V2)
------------------------------------------------------------------------------------------------------------------------
Prerequisites (3 of 3)
* A graphical web browser
* A text editor
* A software package manager
* git client
* General comfort with the command line will be helpful.
* [optional] tar, wget, curl, jq, ssh client
* curl.exe for Windows - https://curl.se/windows/ [https://curl.se/windows/]
------------------------------------------------------------------------------------------------------------------------
A Note for Powershell Users
Terminal commands reflect the Unix bash shell. PowerShell users will need to adjust the commands.
* Unix Shell Variables
* export MY_VAR='test'
* echo ${MY_VAR}
* PowerShell Variables
* $env:my_var = "test"
* Get-ChildItem Env:my_var
* Remove-Item Env:\my_var
------------------------------------------------------------------------------------------------------------------------
Translation Key
\ - Unix Shell Line Continuation ` - Powershell Line Continuation (sort of)
${MY_VAR} - Is generally a placeholder in the slides.
------------------------------------------------------------------------------------------------------------------------
Linux Container Mode
* On the Windows platform make sure that you are running in Linux Container mode
[https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers].
* WARNING: On Windows, avoid using Git Bash. WSL2 is highly recommended instead.
!bg right [marp-themes/images/linux-container-mode.png]
------------------------------------------------------------------------------------------------------------------------
A Note About Proxies & VPNs
Proxies can interfere with some Docker activities if they are not configured correctly and VPNs can increase audio and
video streaming issues in class.
* Configure HTTP_PROXY, HTTPS_PROXY, and your web browser.
If required, you can configure a proxy in Docker Desktop via the preferences.
* Docker [https://dev.to/mcastellin/use-docker-with-proxy-servers-tutorial-10gg]
* Docker-Compose [https://stackoverflow.com/questions/34990458/docker-compose-build-and-http-proxy]
------------------------------------------------------------------------------------------------------------------------
Instructor Environment
* Operating System: macOS (v14.X.X+)
* Terminal: iTerm2 (Build 3.X.X+) - https://www.iterm2.com/ [https://www.iterm2.com/]
* Shell Prompt Theme: Starship - https://starship.rs/ [https://starship.rs/]
* Shell Prompt Font: Fira Code - https://github.com/tonsky/FiraCode [https://github.com/tonsky/FiraCode]
* Text Editor: Visual Studio Code (v1.X.X+) - https://code.visualstudio.com/ [https://code.visualstudio.com/]
* export BUILDKIT_COLORS=run=green:warning=yellow:error=red:cancel=cyan
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker client
* The docker command used to control most of the Docker workflow and talk to remote Docker servers.
* Docker server
* The dockerd command used to launch the Docker daemon. This turns a Linux system into a Docker server that can have
containers deployed, launched, and torn down via a remote client.
------------------------------------------------------------------------------------------------------------------------
The Docker Server
* Virtual Machine or Bare Metal
* In general, the Docker server can only be run on Linux. If you are running Linux on your system, you may have the
server running natively on bare metal. On Windows and macOS, it is common to utilize a Linux virtual machine to run
the server and Docker Desktop Edition takes care of this for you.
* In class, it makes no difference. If I refer to the "virtual machine" or the "docker server" know that these terms
are interchangeable for class and it does not really matter how you are setup locally.
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker/OCI images
* Docker images consist of one or more filesystem layers and some important metadata that represent all the files
required to run a Dockerized application. A single Docker image can be copied to numerous hosts. A container will
typically have both a name and a tag. The tag is generally used to identify a particular release of an image.
* OCI - Open Container Initiative
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Linux Containers
* A Linux Container is a single instantiation of a Docker or OCI-standard image. A specific container can only exist
once; however, you can easily create multiple containers from the same image.
------------------------------------------------------------------------------------------------------------------------
Testing the Docker Setup
$ docker image ls
$ docker container run -d --rm --name quantum \
--publish mode=ingress,published=18080,target=8080 \
docker.io/spkane/quantum-game:latest
$ docker container ls
* In a web browser, navigate to port 18080 on your Docker server.
$ docker container stop quantum
$ docker container ls
------------------------------------------------------------------------------------------------------------------------
Checkout the Code
$ mkdir ${HOME}/class
$ cd ${HOME}/class
$ git clone https://github.com/spkane/rocketchat-hubot-demo.git
$ cd rocketchat-hubot-demo
------------------------------------------------------------------------------------------------------------------------
ChatOps
* MongoDB [https://www.mongodb.com/]
* rocket.chat [https://www.rocket.chat/]
* Hubot [https://hubot.github.com/]
* Z-machine [https://en.wikipedia.org/wiki/Z-machine]
------------------------------------------------------------------------------------------------------------------------
The Script Approach
$ cat scripts/shell_deploy.sh
------------------------------------------------------------------------------------------------------------------------
The Compose Approach
unix:
$ cat compose/docker-compose.yaml
Windows:
PS C:\> Get-Content compose\docker-compose.yaml
Full Documentation: https://docs.docker.com/compose/compose-file [https://docs.docker.com/compose/compose-file]
------------------------------------------------------------------------------------------------------------------------
Preparing the Composition
$ cd compose
$ docker compose config
$ docker compose build
* If you get a failed to solve with frontend dockerfile.v0: failed to create LLB definition error:
* Try docker image pull docker.io/zcube/bitnami-compat-mongodb:5.0 and then rebuild.
------------------------------------------------------------------------------------------------------------------------
Important Note For ARM64 Users
* This compose project uses AMD64 images, because there no ARM64 images available yet.
* QEMU emulation is slow.
* The containers might take 2 minutes after each launch to be running and available via their network ports.
* The beta Rosetta emulation for macOS does not work for all of these images.
* If you enabled it and are following along, please disable it now.
------------------------------------------------------------------------------------------------------------------------
Important Note For Windows Users
* In the next section, you might see:
* a Windows Security Alert for vpnkit.exe, be sure and select Allow access.
* Multiple Docker for Windows - Share drive alerts. Be sure and select Share it for each prompt.
* With Windows PowerShell, you may need to set: $Env:COMPOSE_CONVERT_WINDOWS_PATHS=1
------------------------------------------------------------------------------------------------------------------------
Create a Data Volume for MongoDB
$ cd ../mongodb/data_volume_image
$ docker volume create mongodb-rocketchat
$ docker run --rm \
-v mongodb-rocketchat:/bitnami/mongodb \
-v ${PWD}:/backup busybox \
tar -xzvf /backup/mongodb-rocketchat.tgz -C /bitnami/mongodb
$ cd ../../compose
* No such file or directory error? You are likely in the wrong directory.
------------------------------------------------------------------------------------------------------------------------
Launching the Composition
$ docker compose up -d --build
------------------------------------------------------------------------------------------------------------------------
Checking Compose Status
$ docker compose ps
$ docker compose logs -f
$ docker compose logs rocketchat
Note: If you did not use the container volume for the database setup, you may notice that hubot is restarting. That is
fine at this point.
------------------------------------------------------------------------------------------------------------------------
Exploring RocketChat
* Open your web browser and navigate to:
* http://127.0.0.1:3000 [http://127.0.0.1:3000]
------------------------------------------------------------------------------------------------------------------------
SETUP: Notice
NOTE: The following SETUP slides are only required when you did NOT import the existing MongoDB data. However, you
should note the username and password for login!
------------------------------------------------------------------------------------------------------------------------
SETUP: Admin Info screen
* Fill out the following form to create the admin user.
Full Name: student Username: student Email: student@example.com Password: student-pw!
Then click the Next button.
------------------------------------------------------------------------------------------------------------------------
SETUP: Organization Info screen
* Fill out the form. The specifics are not critical, but you can fill it in something like this:
Organization name: training Organization industry: Education Organization size: 1-10 people Country: United States
Then click the blue button labeled Next.
------------------------------------------------------------------------------------------------------------------------
SETUP: Register Your Server screen
You MUST provide a valid email address and confirm it.
Once confirmed via email, this page should refresh and log in.
------------------------------------------------------------------------------------------------------------------------
SETUP: Create a Bot User
* In the middle of the screen, click the Add users button. _ Then, in the upper right, click New user
* And fill out the user form:
Name: hubot Username:hubot Email: hubot@example.com Click: Verified (Blue) Password:bot-pw! Roles: remove user and add
bot Click: Save
------------------------------------------------------------------------------------------------------------------------
SETUP: Disable Two Factor Authentication
To ensure that the bot can log in, we also need to disable two-factor authentication.
Click: Settings on the Administration sidebar Click: Open under the box marked Accounts Scroll: to the section titled
Two Factor Authentication Open: Two Factor Authentication section Deselect: Enable Two Factor Authentication option
Click: Save changes
------------------------------------------------------------------------------------------------------------------------
SETUP: Close Administration Panel
* At the top of the left side Administration panel, click the X to close the panel
------------------------------------------------------------------------------------------------------------------------
SETUP: Verify Hubot Connection
$ docker compose ps
* The hubot container should be up. If not, try restarting the container:
$ docker compose restart hubot
------------------------------------------------------------------------------------------------------------------------
Communicate with Hubot
* In the left side panel under channels, click general.
* In the general channel try typing each of these:
. help
. ping
. time
------------------------------------------------------------------------------------------------------------------------
SETUP: Create New Channel
NOTE: This SETUP slide is only required if you did NOT import the existing MongoDB data.
Send message:
/create zmachine
Click: zmachine and then send message:
/invite @hubot
------------------------------------------------------------------------------------------------------------------------
Surprise
Try sending these messages:
. z start adventure
more
look
go east
examine keys
get keys
------------------------------------------------------------------------------------------------------------------------
Surprise…
Try sending these messages:
. z save firstgame
. z stop
. z start adventure
. z restore firstgame
look
inventory
------------------------------------------------------------------------------------------------------------------------
Exploring Compose
$ docker compose top
$ docker compose exec mongo bash
# ps auxwww
# exit
$ docker compose stop
$ docker compose start
$ docker compose pause hubot
$ docker compose unpause hubot
Note: The applications will not respond while paused and on Windows, only Hyper-V containers can be paused.
------------------------------------------------------------------------------------------------------------------------
Stopping Compose Projects
$ docker compose stop
------------------------------------------------------------------------------------------------------------------------
Default Values
* Optional Shell Values
* ${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment.
* ${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment.
------------------------------------------------------------------------------------------------------------------------
Compose with Defaults
$ cat docker-compose-defaults.yaml
* Notice that all the variables now reference environment variables.
$ docker compose -f docker-compose-defaults.yaml up -d
$ docker compose -f docker-compose-defaults.yaml stop
------------------------------------------------------------------------------------------------------------------------
Mandatory Values
* Mandatory Shell Values
* ${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment.
* ${VARIABLE?err} exits with an error message containing err if VARIABLE is unset in the environment.
------------------------------------------------------------------------------------------------------------------------
Mandatory Variable Error
$ docker compose -f docker-compose-env.yaml up -d
invalid interpolation format for
services.hubot.environment.ROCKETCHAT_PASSWORD.
You may need to escape any $ with another $.
required variable HUBOT_ROCKETCHAT_PASSWORD is missing a value:
HUBOT_ROCKETCHAT_PASSWORD must be set!
------------------------------------------------------------------------------------------------------------------------
Differences
$ diff -u docker-compose-default.yaml docker-compose-env.yaml
…
- ROCKETCHAT_PASSWORD: ${HUBOT_ROCKETCHAT_PASSWORD:-bot-pw!}
+ ROCKETCHAT_PASSWORD: ${HUBOT_ROCKETCHAT_PASSWORD:?…must be set!}
…
------------------------------------------------------------------------------------------------------------------------
Protecting secrets
unix:
$ cp env.example.txt .env
Windows:
PS C:\> Copy-Item env.example.txt -Destination .env
------------------------------------------------------------------------------------------------------------------------
Don't Commit Secrets
* The.env file is in the .gitignore file at the root of the git repo, so that we do not commit this file into the repo
by accident.
------------------------------------------------------------------------------------------------------------------------
Editing the .env file
Using and .env file makes it easy to protect secrets that should not be checked into source control.
* Open the .env in your text editor and update the following variables to match these values:
HUBOT_ROCKETCHAT_PASSWORD=bot-pw!
------------------------------------------------------------------------------------------------------------------------
Using the .env File
$ docker compose -f docker-compose-env.yaml up -d
------------------------------------------------------------------------------------------------------------------------
Precedence
* Defaults can be defined in the specified docker-compose.yaml file.
* Compose reads from the .env file and overrides those values.
* Values that are defined in the shell, will override everything.
* See: https://docs.docker.com/compose/environment-variables/envvars-precedence/
[https://docs.docker.com/compose/environment-variables/envvars-precedence/]
------------------------------------------------------------------------------------------------------------------------
Multiple Instances of a Project
* Change a few critical environment variables and then launch duplicate service.
$ ROCKETCHAT_ROOT_URL=http://127.0.0.1:33000 \
ROCKETCHAT_PORT=33000 \
ROCKETCHAT_HOST_PORT=33000 \
HUBOT_HOST_PORT=33001 \
docker compose -p rc2 -f docker-compose-env.yaml up -d
------------------------------------------------------------------------------------------------------------------------
Stopping Compose Projects
$ docker compose -p rc2 -f docker-compose-env.yaml stop
$ docker compose -f docker-compose-env.yaml stop
------------------------------------------------------------------------------------------------------------------------
Over-riding a docker-compose.yaml
$ cp .env ./overrides
$ cd ./overrides
$ docker compose config
$ docker compose -f docker-compose.yaml \
-f example-docker-compose.override.yaml config
* Note: The order of -f parameters matters.
------------------------------------------------------------------------------------------------------------------------
Always Over-riding
$ mv example-docker-compose.override.yaml docker-compose.override.yaml
$ docker compose config
* Note: We ignore docker-compose.override.yaml in the repo's .gitignore file.
------------------------------------------------------------------------------------------------------------------------
Scaling containers
$ docker compose \
up -d --scale zmachine=2
$ docker compose stop
------------------------------------------------------------------------------------------------------------------------
Data Volume Backup Example
$ docker compose pause # If the volume is mounted in a running container.
$ docker run --rm -v mongodb-rocketchat:/bitnami/mongodb busybox \
sh -c 'cd /bitnami/mongodb && tar -cOzf - .' > \
../mongodb/data_volume_image/mongodb-rocketchat-student.tgz
$ tar tvfz ../mongodb/data_volume_image/mongodb-rocketchat-student.tgz
$ docker compose unpause # If the volume is mounted in a running container.
------------------------------------------------------------------------------------------------------------------------
Clean Up
$ cd ..
$ docker compose down
# Optional
$ docker volume rm mongodb-rocketchat
------------------------------------------------------------------------------------------------------------------------
What We Have Learned
* Building docker-compose.yaml files
* Launching services with Compose
* Understanding how components interact
* Using docker compose to interact with services
------------------------------------------------------------------------------------------------------------------------
Additional Reading
* The 12-Factor App
* http://12factor.net/ [http://12factor.net/]
* Official Docker Documentation
* https://docs.docker.com/ [https://docs.docker.com/]
* Docker: Up and Running
* https://dockerupandrunning.com/ [https://dockerupandrunning.com/]
!bg right:40% [marp-themes/images/docker-up-and-running-cover-3e-medium.jpg]
------------------------------------------------------------------------------------------------------------------------
Additional Learning Resources
https://learning.oreilly.com/ [https://learning.oreilly.com/]
------------------------------------------------------------------------------------------------------------------------
Student Survey
Please take a moment to fill out the class survey linked to from the bottom of the ON24 audience screen.
O’Reilly and I value your comments about the class.
Thank you!
------------------------------------------------------------------------------------------------------------------------
Any Questions?
Sean P. Kane [https://www.linkedin.com/in/spkane/]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
Hands-on technical training and engineering
https://techlabs.sh/ [https://techlabs.sh/]
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
marp: true theme: techlabs
Docker: Up and Running
Docker for DevOps
Presented by Sean P. Kane
https://techlabs.sh/ [https://techlabs.sh/]
Release %BUILD_RELEASE%
------------------------------------------------------------------------------------------------------------------------
Instructor
Sean P. Kane [https://www.linkedin.com/in/spkane/]
@spkane [https://twitter.com/spkane]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
https://techlabs.sh [https://techlabs.sh]
!bg right [marp-themes/images/skane-2018-side-black-cropped.jpg]
------------------------------------------------------------------------------------------------------------------------
Follow Along Guide [https://gist.github.com/spkane/686ae98467a840d73cd3e181da9b1194]
Textual Slides
------------------------------------------------------------------------------------------------------------------------
O'Reilly Online Sandbox VM [https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469
[https://learning.oreilly.com/interactive-lab/devops-tools-sandbox/9781098126469]
NOTE: You must be logged into your O'Reilly account and the VM sessions will expire after 60 minutes!
------------------------------------------------------------------------------------------------------------------------
Prerequisites (1 of 3)
NOTE: You MUST have open access to Github if you want to participate in the hands-on portion of class from your local
computer system.
------------------------------------------------------------------------------------------------------------------------
Prerequisites (2 of 3)
* A recent computer and OS
* Recent/Stable Linux, macOS, or Windows 10+
* root/admin rights
* Sufficient resources to run one 2 CPU VM
* CPU Virtualization MUST be enabled in your BIOS/EFI
* Reliable and fast internet connectivity
* Docker Community/Desktop Edition (on Linux also install Docker Compose V2)
------------------------------------------------------------------------------------------------------------------------
Prerequisites (3 of 3)
* A graphical web browser
* A text editor
* A software package manager
* git client
* General comfort with the command line will be helpful.
* [optional] tar, wget, curl, jq, ssh client
* curl.exe for Windows - https://curl.se/windows/ [https://curl.se/windows/]
------------------------------------------------------------------------------------------------------------------------
A Note for Powershell Users
Terminal commands reflect the Unix bash shell. PowerShell users will need to adjust the commands.
* Unix Shell Variables
* export MY_VAR='test'
* echo ${MY_VAR}
* PowerShell Variables
* $env:my_var = "test"
* Get-ChildItem Env:my_var
* Remove-Item Env:\my_var
------------------------------------------------------------------------------------------------------------------------
Translation Key
\ - Unix Shell Line Continuation ` - Powershell Line Continuation (sort of)
${MY_VAR} - Is generally a placeholder in the slides.
------------------------------------------------------------------------------------------------------------------------
Linux Container Mode
* On the Windows platform make sure that you are running in Linux Container mode
[https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers].
* WARNING: On Windows, avoid using Git Bash. WSL2 is highly recommended instead.
!bg right [marp-themes/images/linux-container-mode.png]
------------------------------------------------------------------------------------------------------------------------
A Note About Proxies & VPNs
Proxies can interfere with some Docker activities if they are not configured correctly and VPNs can increase audio and
video streaming issues in class.
* Configure HTTP_PROXY, HTTPS_PROXY, and your web browser.
If required, you can configure a proxy in Docker Desktop via the preferences.
* Docker [https://dev.to/mcastellin/use-docker-with-proxy-servers-tutorial-10gg]
* Docker-Compose [https://stackoverflow.com/questions/34990458/docker-compose-build-and-http-proxy]
------------------------------------------------------------------------------------------------------------------------
Instructor Environment
* Operating System: macOS (v14.X.X+)
* Terminal: iTerm2 (Build 3.X.X+) - https://www.iterm2.com/ [https://www.iterm2.com/]
* Shell Prompt Theme: Starship - https://starship.rs/ [https://starship.rs/]
* Shell Prompt Font: Fira Code - https://github.com/tonsky/FiraCode [https://github.com/tonsky/FiraCode]
* Text Editor: Visual Studio Code (v1.X.X+) - https://code.visualstudio.com/ [https://code.visualstudio.com/]
* export BUILDKIT_COLORS=run=green:warning=yellow:error=red:cancel=cyan
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker client
* The docker command used to control most of the Docker workflow and talk to remote Docker servers.
* Docker server
* The dockerd command used to launch the Docker daemon. This turns a Linux system into a Docker server that can have
containers deployed, launched, and torn down via a remote client.
------------------------------------------------------------------------------------------------------------------------
The Docker Server
* Virtual Machine or Bare Metal
* In general, the Docker server can only be run on Linux. If you are running Linux on your system, you may have the
server running natively on bare metal. On Windows and macOS, it is common to utilize a Linux virtual machine to run
the server and Docker Desktop Edition takes care of this for you.
* In class, it makes no difference. If I refer to the "virtual machine" or the "docker server" know that these terms
are interchangeable for class and it does not really matter how you are setup locally.
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Docker/OCI images
* Docker images consist of one or more filesystem layers and some important metadata that represent all the files
required to run a Dockerized application. A single Docker image can be copied to numerous hosts. A container will
typically have both a name and a tag. The tag is generally used to identify a particular release of an image.
* OCI - Open Container Initiative
------------------------------------------------------------------------------------------------------------------------
Docker in Translation
* Linux Containers
* A Linux Container is a single instantiation of a Docker or OCI-standard image. A specific container can only exist
once; however, you can easily create multiple containers from the same image.
------------------------------------------------------------------------------------------------------------------------
Testing the Docker Setup
$ docker image ls
$ docker container run -d --rm --name quantum \
--publish mode=ingress,published=18080,target=8080 \
docker.io/spkane/quantum-game:latest
$ docker container ls
* In a web browser, navigate to port 18080 on your Docker server.
$ docker container stop quantum
$ docker container ls
------------------------------------------------------------------------------------------------------------------------
Private Registry
$ mkdir ${HOME}/class
$ cd ${HOME}/class
$ git clone https://github.com/spkane/docker-registry.git
$ cd docker-registry
$ docker compose up -d
------------------------------------------------------------------------------------------------------------------------
Private Registry
$ docker login 127.0.0.1:5000
$ docker image ls distribution/distribution:2.8.2
$ docker image tag \
$(docker image ls distribution/distribution:2.8.2 --format "{{.ID}}") \
127.0.0.1:5000/distribution:latest
$ docker image tag \
$(docker image ls distribution/distribution:2.8.2 --format "{{.ID}}") \
127.0.0.1:5000/distribution:2.8.2
$ docker image push 127.0.0.1:5000/distribution:latest
$ docker image push 127.0.0.1:5000/distribution:2.8.2
* Username: myuser
* Password: myuser-pw!
------------------------------------------------------------------------------------------------------------------------
Simple Registry UI
* In your browser navigate to:
* http://127.0.0.1:8080 [http://127.0.0.1:8080]
* When you are finished exploring run:
$ docker compose down
------------------------------------------------------------------------------------------------------------------------
Distribution
* Project Home Page: https://github.com/distribution/distribution [https://github.com/distribution/distribution]
------------------------------------------------------------------------------------------------------------------------
Docker BuildKit
* Project Home Page: https://github.com/moby/buildkit [https://github.com/moby/buildkit]
* Turn it on like this:
export DOCKER_BUILDKIT=1
* and, for the record, you can turn it off with this:
export DOCKER_BUILDKIT=0
------------------------------------------------------------------------------------------------------------------------
Caching
$ git clone https://github.com/spkane/open-mastermind.git
$ time docker build --progress=plain --no-cache \
-t open-mastermind:latest .
$ time docker build --progress=plain \
-t open-mastermind:latest .
$ docker container run -ti open-mastermind:latest
* Test the container
------------------------------------------------------------------------------------------------------------------------
Adding log-symbols
* Add log-symbols to the end of requirements.txt
$ time docker build --progress=plain \
-t open-mastermind:latest .
* Docker re-downloads all our requirements. 😢
------------------------------------------------------------------------------------------------------------------------
Directory Caching
1. Optional: Add # syntax=docker/dockerfile:1 to the top of the Dockerfile. This is no longer required, but it worth
mentioning, as there are times that it can be useful.
2. Let's modify the line in the Dockerfile that runs pip to look like this.
RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt
$ time docker build --progress=plain --no-cache \
-t open-mastermind:latest .
------------------------------------------------------------------------------------------------------------------------
Adding py-events
* Add py-events to the end of requirements.txt
$ time docker build --progress=plain \
-t open-mastermind:latest .
* Docker utilizes the directory cache and only adds the files we need! 🎉
------------------------------------------------------------------------------------------------------------------------
BuildKit Mount Features 1/2
* Bind mount
* --mount=type=bind,target=/root/tests,source=/tests
* Cache mount
* --mount=type=cache,target=/root/.cache
* tmpfs mount
* --mount=type=tmpfs,target=/mnt/tmpfs
------------------------------------------------------------------------------------------------------------------------
BuildKit Mount Features 2/2
* Secret mount
* --mount=type=secret,id=aws,target=/root/.aws/credentials
* SSH mount
* --mount=type=ssh,required=true
------------------------------------------------------------------------------------------------------------------------
BuildKit Network Feature
* Network type
* --network=none|host|default
------------------------------------------------------------------------------------------------------------------------
BuildKit Lab Features 1/2
* To enable lab features set: # syntax=docker/dockerfile:1.6-labs
* Security Context
* --security=insecure|sandbox
------------------------------------------------------------------------------------------------------------------------
BuildKit Lab Features 2/2
* Here Documents
RUN <<EOF
#!/usr/bin/env python
print("Here documents in Docker!!")
EOF
COPY <<-"EOF" /start.sh
Your home directory is: ${HOME}
EOF
------------------------------------------------------------------------------------------------------------------------
Docker Frontend for BuildKit
* Documentation [https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md]
* Release Notes [https://docs.docker.com/build/dockerfile/release-notes/]
------------------------------------------------------------------------------------------------------------------------
Building Multi Platform Containers
* Including:
* linux/amd64 (64bit - Intel/AMD)
* linux/arm64 (64bit - Apple Silicon M1, Raspberry Pi, etc.)
* linux/arm/v7 (32bit - Raspberry Pi, etc.)
------------------------------------------------------------------------------------------------------------------------
Create a New Builder
* Optional - This is the process to setup a new builder for buildx to use.
$ docker buildx rm builder
$ docker buildx create --name builder --driver docker-container --use
$ docker buildx inspect --bootstrap
------------------------------------------------------------------------------------------------------------------------
Multi-Platform Build Preparation
$ git clone https://github.com/spkane/wordchain.git
$ cd wordchain
------------------------------------------------------------------------------------------------------------------------
Multi-Platform Build
* Build for multiple platforms.
* We could add --push to this command if we wanted these to all be uploaded to a registry we have access to.
$ docker buildx build \
--platform linux/arm/v7,linux/arm64/v8,linux/amd64 \
--tag docker.io/spkane/wordchain:test .
* Got an error? Try running the command once more to see if it was transitory.
------------------------------------------------------------------------------------------------------------------------
Build Local Image
* Build it for your local system and load the results into Docker.
* Note: To run the resulting image you will need to replace ${ARCH} in the build command with your valid
architecture. Generally this will either be amd64 or arm64.
$ docker buildx build --platform linux/${ARCH} --load \
--tag docker.io/spkane/wordchain:test .
------------------------------------------------------------------------------------------------------------------------
Test Local Image
* Run the image a few times locally to ensure it works as expected.
$ docker container run docker.io/spkane/wordchain:test random
$ docker container run docker.io/spkane/wordchain:test --help
$ docker container run docker.io/spkane/wordchain:test random -l 3 -d .
------------------------------------------------------------------------------------------------------------------------
Examine Image Manifest
$ docker manifest inspect docker.io/spkane/wordchain:latest
$ docker buildx imagetools inspect docker.io/spkane/wordchain:latest
------------------------------------------------------------------------------------------------------------------------
Manipulating Namespaces
* Containers usually have their own set of namespaces, but they can be configured to share a namespace with the
underlying host or even another container.
------------------------------------------------------------------------------------------------------------------------
Process Namespace (1 of 2)
* Sharing with the host
$ docker container run docker.io/spkane/train-os ps auxwwww
$ docker container run --pid=host docker.io/spkane/train-os ps auxwwww
------------------------------------------------------------------------------------------------------------------------
Process Namespace (2 of 2)
* Sharing with another container
$ docker container run -d --rm --name pid1 docker.io/spkane/train-os \
sleep 60
$ docker container run -ti docker.io/spkane/train-os ps auxwwww
$ docker container run -ti --pid=container:pid1 \
docker.io/spkane/train-os ps auxwwww
------------------------------------------------------------------------------------------------------------------------
Network Namespace
$ docker container run docker.io/spkane/train-os ip link
$ docker container run --net=host docker.io/spkane/train-os ip link
------------------------------------------------------------------------------------------------------------------------
UTS Namespace
$ docker container run -ti docker.io/spkane/train-os hostname
$ docker container run -ti --uts=host docker.io/spkane/train-os hostname
------------------------------------------------------------------------------------------------------------------------
IPC Namespace (1 of 2)
* Create a container that can share memory with the underlying host.
$ docker container run -ti --ipc=host docker.io/spkane/train-os ipcs
------------------------------------------------------------------------------------------------------------------------
IPC Namespace (2 of 2)
* Create a container that can share memory with other containers.
$ docker container run -d --rm --name ipc1 \
--ipc=shareable docker.io/spkane/train-os sleep 60
$ docker container run -ti --ipc=container:ipc1 \
docker.io/spkane/train-os ipcs
------------------------------------------------------------------------------------------------------------------------
Mount (and other) namespaces
$ docker container run -ti --pid=host --privileged=true \
docker.io/spkane/train-os nsenter -t 1 -a cat /etc/os-release
------------------------------------------------------------------------------------------------------------------------
Check out Code
$ cd ${HOME}/class
$ git clone https://github.com/spkane/outyet.git
------------------------------------------------------------------------------------------------------------------------
Developing in Containers
* Note: This demo will not work in the O'Reilly VM sandbox IDE and is best done on your local workstation.
* Why?
* Repeatability
* Instant Environment
* No version drift between developers
* and more…
------------------------------------------------------------------------------------------------------------------------
Installing Visual Studio Code
Install Visual Studio Code from:
* https://code.visualstudio.com/download [https://code.visualstudio.com/download]
------------------------------------------------------------------------------------------------------------------------
Install Extensions
* Launch Visual Studio Code
* Install Extensions:
* Dev Containers
------------------------------------------------------------------------------------------------------------------------
Developing Outyet
* Launch Visual Studio Code
* Select File - Open and select the outyet git checkout folder.
* Prompt appears that says Folder contains a Dev Container configuration file. Reopen folder to develop in a container…
* Select X to close prompt for now.
------------------------------------------------------------------------------------------------------------------------
devcontainer
* The configuration file for the VSCODE development environment.
* ./.devcontainer/devcontainer.json
* The Dockerfile for the dev container that will be used as a local environment.
* ./.devcontainer/Dockerfile
------------------------------------------------------------------------------------------------------------------------
Open in a Container
* Open the Command Palette (F1)
* Enter/Select Dev Containers: Reopen in Container
------------------------------------------------------------------------------------------------------------------------
Open a Terminal
* Open a Terminal: [Ctrl-`]
* go build .
* ./outyet
* Click Open in Browser in the popup dialog
------------------------------------------------------------------------------------------------------------------------
Reopen Locally
* Open the Command Palette (F1)
* Enter/Select Dev Containers: Reopen Folder Locally
------------------------------------------------------------------------------------------------------------------------
Create a New Git Repo
$ cd ${HOME}/class
$ mkdir example-repo
$ cd example-repo
$ git init
------------------------------------------------------------------------------------------------------------------------
Creating a New Development Container
* Open the new example-repo git repo
* Open the Command Palette (F1)
* Enter/Select Dev Containers: Add Development Container Configuration Files…
------------------------------------------------------------------------------------------------------------------------
Dev Container Documentation
* Development Containers Specification [https://containers.dev/]
* Dev Containers in VSCode [https://code.visualstudio.com/docs/remote/containers]
* Attach VSCode to Existing Containers [https://code.visualstudio.com/docs/remote/attach-container]
------------------------------------------------------------------------------------------------------------------------
Podman
* Podman is an opensource daemon-less alternative to Docker.
* Podman is compatible with most traditional Docker commands, etc.
* Podman is multi-platform, but still has rough edges on Windows and macOS.
* Installing Podman
* https://podman.io/getting-started/installation [https://podman.io/getting-started/installation]
------------------------------------------------------------------------------------------------------------------------
What We Have Learned
* How to launch a Private Image Registry
* The new features introduced by Docker BuildKit
* How to build images for multiple architectures
* How to manipulate Linux Namespaces
* How to Develop in a Container w/ Visual Studio Code
------------------------------------------------------------------------------------------------------------------------
Additional Reading
* The 12-Factor App
* http://12factor.net/ [http://12factor.net/]
* Official Docker Documentation
* https://docs.docker.com/ [https://docs.docker.com/]
* Docker: Up and Running
* https://dockerupandrunning.com/ [https://dockerupandrunning.com/]
!bg right:40% [marp-themes/images/docker-up-and-running-cover-3e-medium.jpg]
------------------------------------------------------------------------------------------------------------------------
Additional Learning Resources
https://learning.oreilly.com/ [https://learning.oreilly.com/]
------------------------------------------------------------------------------------------------------------------------
Student Survey
Please take a moment to fill out the class survey linked to from the bottom of the ON24 audience screen.
O’Reilly and I value your comments about the class.
Thank you!
------------------------------------------------------------------------------------------------------------------------
Any Questions?
Sean P. Kane [https://www.linkedin.com/in/spkane/]
!height:2.3cm [marp-themes/images/techlabs-dark.png]
Hands-on technical training and engineering
https://techlabs.sh/ [https://techlabs.sh/]
------------------------------------------------------------------------------------------------------------------------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment