cwgem/build_guide.md

## build_guide.md

      
    Raw
  

              build_guide.md
            
          
    Introduction

This guide looks at what it will take to build dockerd and docker-cli from source with Ubuntu. Ubuntu was chosen as the OS as it uses the same apt-get calls the standard Dockerfile method uses making the experience a bit more seamless. While the official method is building docker in a container, here are some reasons why you may want to build this way:

Custom source modifications for testing
System that does not yet have a docker(d) binary release
You want to test things on a non-standard toolchain
You just want to deep dive

As you can probably tell using this method means you're delving away from support. Please don't file issues/PRs unless you can reproduce in the official build environment. In fact unless you really have one of the strong needs above you're better off building off a container since this guide is mostly a lot of copy/paste from the Dockerfile used in the official build system.
Build Systems

The first thing off is to get us up and running with compiler and build system components. As it's the first thing that needs to be done we'll go ahead and update the apt-get package list as well:
# apt-get update
# apt-get install -y build-essential
# apt-get install -y cmake pkg-config

build-essential contains most of the tools you'll need for compilation (more specifically the gcc compiler). cmake and pkg-config are used to build one of the dockerd dependencies.
Dev and runtime deps

While most of the core deps of both dockerd and docker-cli are golang packages you still need a decent portion of OS packages installed to get things working. Thankfully there's a page specifically dedicated to both build and runtime dependencies. Even more helpful is that the Dockerfile used to build dockerd inside a container has pretty much all the deps you'll need laid out. But first a bit of setup:
# apt-get install -y git
# export CRIU_VERSION=3.6

We'll need git a lot from here on out so best install it first. We'll also need to set CRIU_VERSION which will be shortly used to download a tarball. At time of writing it's 3.6 but you'll want to check the Dockerfile linked above to see what the most current value should be. Now to part 1 of the install:
# apt-get install -y \
        sqlite3 \
	sqlite3-dev \
  libltdl-dev \
	libnet-dev \
	libprotobuf-c0-dev \
	libprotobuf-dev \
	libnl-3-dev \
	libcap-dev \
	protobuf-compiler \
	protobuf-c-compiler \
	python-protobuf \
	&& mkdir -p /usr/src/criu \
	&& curl -sSL https://github.com/checkpoint-restore/criu/archive/v${CRIU_VERSION}.tar.gz | tar -C /usr/src/criu/ -xz --strip-components=1 \
	&& cd /usr/src/criu \
	&& make \
	&& make PREFIX=/usr/local install-criu

Just to be safe I added sqlite3 and the appropriate dev package since the PACKAGING markdown file listed it out. There's also criu for checkpoint and restore functionality. Note that I have it installed in /usr/local versus where Dockerfile wants to put it (/build). This trend continues throughout most of the guide. Now for some more runtime-ish dep work:
# apt-get install -y jq ca-certificates --no-install-recommends
# apt-get install -y libseccomp-dev
# apt-get install -y \
	aufs-tools \
	btrfs-tools \
	iptables \
	jq \
	libdevmapper-dev \
	libudev-dev \
	libsystemd-dev \
	net-tools \
	thin-provisioning-tools \
	vim \
	vim-common \
	xfsprogs \
	zip \
	bzip2 \
	xz-utils \
	--no-install-recommends

Note that you can install most of the above in one fatal swoop technically, but I chose to strategically break them out as I found them in the Dockerfile.
User setup

As dockerd sets its sock file and other items as owned by the docker group, we'll need to go ahead and add that. As I'm using an Ubuntu AMI on AWS I went ahead and added the non-privileged ubuntu user to the group for later work.
# groupadd -r docker
# adduser ubuntu docker

To properly apply changes I went ahead and exited out of the SSH session and re-connected.
Golang install

I know I know, "why are you installing from source? There's an OS package and binaries right?". Well yes but I plan to use this setup for debugging work on docker issues, so being able to switch branches/tags for replicating certain setups is nice. It also means I can utilize newer versions that what may be available in the Ubuntu repos.
Now compilation of golang has a bit of a chicken and the egg problem. You need golang to compile golang. With that in mind I'm going to pull a binary of the latest version off Google's site:
$ wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz
$ sha256sum go1.10.3.linux-amd64.tar.gz
fa1b0e45d3b647c252f51f5e1204aba049cde4af177ef9f2181f43004f901035  go1.10.3.linux-amd64.tar.gz

Yes, I did just SHA verify the download. Remember you should always do this. Now I move the resulting directory to something more friendly, and set it up as the golang to use for bootstrapping:
$ mv go go1.10.3
$ vim ~/.profile

<snip>
# set PATH so it includes user's private bin directories
PATH="$HOME/bin:$HOME/.local/bin:$PATH"

GOROOT_BOOTSTRAP="$HOME/go1.10.3"

export PATH
export GOROOT_BOOTSTRAP

$ source ~/.profile

I went ahead and added the GOROOT_BOOTSTRAP and set it to the previous directory. This is how golang's build system knows what to use to bootstrap compilation. Bootstrapping is the process of compiling to a certain output, using that output to re-compile, and so on putting the code generation as close to the native system as possible. Next I go ahead and load the profile into the current shell so the environment variables show up properly. Now it's time to pull down the golang git repo:
$ git clone https://go.googlesource.com/go go_git
Cloning into 'go_git'...
remote: Sending approximately 219.09 MiB ...
remote: Counting objects: 85, done
remote: Finding sources: 100% (63/63)
remote: Total 348828 (delta 265189), reused 348805 (delta 265189)
Receiving objects: 100% (348828/348828), 218.81 MiB | 28.52 MiB/s, done.
Resolving deltas: 100% (265189/265189), done.
Checking connectivity... done.
$ cd go_git

I pull this into an easier to identify folder and hop into there. Now working off the master branch is really not a good idea, so I'll go ahead and switch to the 1.10 release branch instead:
$ git branch -a
<snip>
remotes/origin/dev.typealias
remotes/origin/master
remotes/origin/release-branch.go1
remotes/origin/release-branch.go1.1
remotes/origin/release-branch.go1.10
remotes/origin/release-branch.go1.2
remotes/origin/release-branch.go1.3
remotes/origin/release-branch.go1.4
remotes/origin/release-branch.go1.5
remotes/origin/release-branch.go1.6
remotes/origin/release-branch.go1.7
remotes/origin/release-branch.go1.8
remotes/origin/release-branch.go1.9
remotes/origin/release-branch.r57
remotes/origin/release-branch.r58
remotes/origin/release-branch.r59
remotes/origin/release-branch.r60
$ git checkout release-branch.go1.10

Now for the actual compilation. The following will bootstrap golang, then run a bunch of tests against it to verify you didn't end up with a horribly broken build (or that you're on a faulty system). This takes quite a long time so grab a snack or something:
$ cd src
$ bash all.bash
<snip>
##### ../misc/cgo/errors
PASS

##### ../misc/cgo/testsigfwd

##### ../test/bench/go1
testing: warning: no tests to run
PASS
ok      _/home/ubuntu/go_git/test/bench/go1     5.420s

##### ../test

##### API check
Go version is "go1.10.3", ignoring -next /home/ubuntu/go_git/api/next.txt

ALL TESTS PASSED
---
Installed Go for linux/amd64 in /home/ubuntu/go_git
Installed commands in /home/ubuntu/go_git/bin
*** You need to add /home/ubuntu/go_git/bin to your PATH

As it mentions I need to put the resulting build in path, but before that I need to do a bit of go setup. GOROOT is needed to tell go that it's in a non-standard directory. GOPATH is needed to tell go where to put and search for various packages. I'll set that up to be ~/docker_build and then setup the various paths:
$ mkdir ~/docker_build
$ vim ~/.profile
<snip>
GOROOT_BOOTSTRAP="/home/ubuntu/go1.10.3"
GOROOT=$HOME/go_git
GOPATH=$HOME/docker_build

# set PATH so it includes user's private bin directories
PATH="$HOME/bin:$HOME/.local/bin:$PATH:$GOROOT/bin:$GOPATH/bin"

export PATH
export GOROOT_BOOTSTRAP
export GOROOT
export GOPATH

$ source ~/.profile
$ go version
go version go1.10.3 linux/amd64

So we setup the two variables and also add both of their binary directories to path. $GOROOT/bin adds the go binary to path and $GOPATH/bin makes sure that any go packages that install binaries can have their binaries available right away. Just to test let's go ahead and pull in godoc:
$ go get golang.org/x/tools/cmd/godoc
$ godoc --help
usage: godoc package [name ...]
        godoc -http=:6060
<snip>

After install the godoc binary is available to use right away given our path update. Of more interest is that GOPATH is now populated with some things:
$ ls -lah $GOPATH
total 16K
drwxrwxr-x 4 ubuntu ubuntu 4.0K Jul 11 23:23 .
drwxr-xr-x 7 ubuntu ubuntu 4.0K Jul 11 23:20 ..
drwxrwxr-x 2 ubuntu ubuntu 4.0K Jul 11 23:23 bin
drwxrwxr-x 3 ubuntu ubuntu 4.0K Jul 11 23:23 src

I'll not go into too much detail on what this means as this is not a golang guide. Just note that bin is where any package installed binaries end up, and src is where packages leave their code for compilation/import purposes.
dockerd source pull

Now dockerd is a unique beast. What used to be docker/docker has now turned into the moby project. It's also where docker/docker ends up redirecting to on GitHub. That said since most of the docker packaging uses docker/docker in its imports we're going to go ahead and pull it down as that. Now, since go get handles GitHub repos and puts it in the proper GOPATH location we'll use that instead:
$ go get -d github.com/docker/docker
package github.com/docker/docker: no Go files in /home/ubuntu/docker_build/src/github.com/docker/docker

No real need to concern with the warning shown. -d to go get simply tells it to just download the source and not install it. This is nice since we're about to go build it from source anyways and handle the installation as well.
dockerd requirement build

Now dockerd uses a couple of runtime packages to do certain things. In particular:

containerd - Handles abstracting a lot of kernel away to make containers easier to work with
runc - Handles the actual running/spawning of containers
proxy - Handles proxying of traffic between host and container (part of libnetwork)
tini - A tiny lightweight init system that can be used by containers for being the initial init process (also what we needed cmake and pkg-config for)
toml - Golang parser for TOML, used by containerd for configuration
vndr - Used by Docker to handle vendoring of packages

While you technically could just install all these through git and so forth docker/docker actually has some install scripts that make the whole process easy. Essentially you call install.sh with whatever it is you want to install as the argument. It sets up some variables for you (including the /usr/local/ PREFIX to install to) and then calls the respective .installer script. What's nice about this is that it restricts the installation to specific commit IDs, further ensuring that the build you get is as compatible as possible with the rest of dockerd.
Now since I'm not a huge fan of running things as root, and given that the installer scripts don't separate out build and install phases I'm to do the install a bit differently. I'll go ahead and set a PREFIX before each install and run each of the installs in a bash loop. This lets me do the whole deal as non-root while throwing it in a path I can copy stuff from later (since I really don't want to add a user directory to root's path, which is what dockerd runs as):
Note containerd by default installs as static in the install script. Due to that failing (most likely needs musl to properly compile statically) I instead decided to install it as dynamic, meaning it needed to be one-offed.
$ mkdir ~/docker_utils
$ cd ~/docker_build/src/github.com/docker/docker/hack/dockerfile/install
$ PREFIX="$HOME/docker_utils" ./install.sh containerd dynamic
$ for package in "runc proxy tini tomlv vndr"
> do
>   PREFIX="$HOME/docker_utils" ./install.sh $package
> done

Now that the whole build process is done as non-root we can pull the binaries over to /usr/local/bin as root:
$ ls -lah ~/docker_utils/
total 85M
drwxrwxr-x 2 ubuntu ubuntu 4.0K Jul 12 00:42 .
drwxr-xr-x 8 ubuntu ubuntu 4.0K Jul 12 00:36 ..
-rwxrwxr-x 1 ubuntu ubuntu  37M Jul 12 00:37 docker-containerd
-rwxrwxr-x 1 ubuntu ubuntu  20M Jul 12 00:37 docker-containerd-ctr
-rwxrwxr-x 1 ubuntu ubuntu 4.0M Jul 12 00:37 docker-containerd-shim
-rwxrwxr-x 1 ubuntu ubuntu 851K Jul 12 00:42 docker-init
-rwxrwxr-x 1 ubuntu ubuntu 2.8M Jul 12 00:42 docker-proxy
-rwxrwxr-x 1 ubuntu ubuntu 7.2M Jul 12 00:42 docker-runc
-rwxrwxr-x 1 ubuntu ubuntu 3.2M Jul 12 00:42 tomlv
-rwxrwxr-x 1 ubuntu ubuntu  11M Jul 12 00:42 vndr
# cp /home/ubuntu/docker_utils/* /usr/local/bin

Now root will be able to call these without having to change $PATH at all. Note that the binaries that have docker prefixes were something named differently during the build phase. The docker install scripts simply rename them after the binaries are built. This is important because dockerd expects these prefixes when calling the binaries.
Dockerd build

Since the dep magic is over it's time to actually build dockerd. First we hit the repo root:
$ cd ~/docker_build/src/github.com/docker/docker

There's a file called hack/make.sh we'll be using here. Since static compilation might end up how the previous containerd built went I'll go ahead and just make it a dynamically linked binary. I'll also tell it to add -w to the LDFLAGS by setting DOCKER_DEBUG to a non-empty string which makes debugging info get output should I need it.
$ DOCKER_DEBUG="TRUE" hack/make.sh dynbinary
<snip>
---> Making bundle: dynbinary (in bundles/dynbinary)
Building: bundles/dynbinary-daemon/dockerd-dev
Created binary: bundles/dynbinary-daemon/dockerd-dev
$ bundles/dynbinary-daemon/dockerd-dev -v
Docker version dev, build 705774a

So yeah that works! Now I want to copy this over to /usr/local/bin to make things less annoying, but just to be safe:
$ ldd bundles/dynbinary-daemon/dockerd-dev
        linux-vdso.so.1 =>  (0x00007fff7b185000)
        libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007fa048fa3000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa0448ca000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa0446c6000)
        libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007fa04446d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa0440a3000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fa043e81000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa043c79000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fa043a57000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fa043776000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa048e0b000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fa048f80000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa04346d000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fa0431fd000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fa042fe9000)

So there's nothing in a weird non-system path it's linking to, meaning I can pretty much copy it over without worry:
# cp /home/ubuntu/docker_build/src/github.com/docker/docker/bundles/dynbinary-daemon/dockerd-dev /usr/local/bin/

Now, there are some systemd files to deal with the daemon starting so I'll go ahead and install those:
# cp /home/ubuntu/docker_build/src/github.com/docker/docker/contrib/init/systemd/docker.{service,socket} /etc/systemd/system/
# chmod 664 /etc/systemd/system/docker.s*

Except the binary provided isn't the same as where systemd is expecting it so we'll need to fudge that a bit:
# vim /etc/systemd/system/docker.service
<snip>
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/local/bin/dockerd-dev -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
<snip>

Here I pretty much adjusted ExecStart to point to the /usr/local/bin binary like it's supposed to. Now to see everything crash and burn, I mean the service starting:
# systemctl daemon-reload
# systemctl start docker.service
# systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-07-12 01:40:01 UTC; 6s ago
     Docs: https://docs.docker.com
 Main PID: 11691 (dockerd-dev)
    Tasks: 29
   Memory: 53.0M
      CPU: 329ms
   CGroup: /system.slice/docker.service
           ├─11691 /usr/local/bin/dockerd-dev -H fd://
           └─11702 docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level info

Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.758707432Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.758777785Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42017b500, CONNECTING" module=grpc
Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.759130313Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42017b500, READY" module=grpc
Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.759158870Z" level=info msg="Loading containers: start."
Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.887584540Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be u
Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.927181130Z" level=info msg="Loading containers: done."
Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.951725482Z" level=info msg="Docker daemon" commit=705774a-unsupported graphdriver(s)=overlay2 version=dev
Jul 12 01:40:01 ip-10-0-0-53 dockerd-dev[11691]: time="2018-07-12T01:40:01.951886919Z" level=info msg="Daemon has completed initialization"

You'll notice not only did the service start but it's running the docker- prefixed binary as I mentioned earlier.
Docker Cli

Now it's time to get into docker-cli install since it's the core way to interface with the dockerd. As with docker/docker we'll go ahead and pull it down using go get:
Note Technically dockerd's build system comes with docker CLI as a part of it, but having it as the standard source layout makes things easier for fudging code and build options
$ go get -d github.com/docker/cli

Thankfully this is less involved than the daemon build. So we'll drop into the source:
$ cd ~/docker_build/src/github.com/docker/cli

Similar to hack/make.sh there are some build scripts located in scripts/build:
$ ls -lah scripts/build/
total 32K
drwxrwxr-x 2 ubuntu ubuntu 4.0K Jul 12 11:32 .
drwxrwxr-x 8 ubuntu ubuntu 4.0K Jul 12 11:30 ..
-rwxrwxr-x 1 ubuntu ubuntu  295 Jul 12 11:30 binary
-rwxrwxr-x 1 ubuntu ubuntu  768 Jul 12 11:30 cross
-rwxrwxr-x 1 ubuntu ubuntu  321 Jul 12 11:30 dynbinary
-rwxrwxr-x 1 ubuntu ubuntu  438 Jul 12 11:30 osx
-rwxrwxr-x 1 ubuntu ubuntu  805 Jul 12 11:30 .variables
-rwxrwxr-x 1 ubuntu ubuntu  439 Jul 12 11:30 windows

As with dockerd we're going to be building a dynamically linked binary. The actual script for doing that isn't so complex, mostly about pulling data from the .variables file which looks like this:
$ cat scripts/build/.variables
#!/usr/bin/env bash
set -eu

PLATFORM=${PLATFORM:-}
VERSION=${VERSION:-"unknown-version"}
GITCOMMIT=${GITCOMMIT:-$(git rev-parse --short HEAD 2> /dev/null || true)}
BUILDTIME=${BUILDTIME:-$(date --utc --rfc-3339 ns 2> /dev/null | sed -e 's/ /T/')}

PLATFORM_LDFLAGS=
if test -n "${PLATFORM}"; then
        PLATFORM_LDFLAGS="-X \"github.com/docker/cli/cli.PlatformName=${PLATFORM}\""
fi

export LDFLAGS="\
-w \
${PLATFORM_LDFLAGS} \
-X \"github.com/docker/cli/cli.GitCommit=${GITCOMMIT}\" \
-X \"github.com/docker/cli/cli.BuildTime=${BUILDTIME}\" \
-X \"github.com/docker/cli/cli.Version=${VERSION}\" \
${LDFLAGS:-} \
"

GOOS="${GOOS:-$(go env GOHOSTOS)}"
GOARCH="${GOARCH:-$(go env GOHOSTARCH)}"
export TARGET="build/docker-$GOOS-$GOARCH"
export SOURCE="github.com/docker/cli/cmd/docker"

This mostly sets up some compile variables for the go build system. Thankfully -w is already in LDFLAGS so nothing to change there. Now to go ahead and get this build going. Due to how the relative paths are setup this needs to be run in the toplevel of the repository (which we're currently in):
$ scripts/build/dynbinary
Building dynamically linked build/docker-linux-amd64

Great builds quite nicely. Just to make docker-cli sudo friendly (since you get root's $PATH at that point), we'll go ahead and copy the resulting build to /usr/local/bin. Before that the usual linking check:
$ ldd build/docker-linux-amd64
        linux-vdso.so.1 =>  (0x00007ffe37972000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2282a22000)
        libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f2282818000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f228244e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2282c3f000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f228224a000)

Okay great nothing in any strange directories. As the client will get used a lot we'll also copy it to the more friendly docker name:
# cp /home/ubuntu/docker_build/src/github.com/docker/cli/build/docker-linux-amd64 /usr/local/bin/docker

Now for a simple run check by using docker version:
$ docker version
Client:
 Version:           unknown-version
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        ee8cdb3
 Built:             Thu Jul 12 11:40:58 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          dev
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       705774a
  Built:            Thu Jul 12 02:23:53 2018
  OS/Arch:          linux/amd64
  Experimental:     false

This shows that not only does the client work, but it can also connect to the server (as well as showing how unsupported it is!). Now it's all compiled for use as you please.