estesp

pull an image

root@ubu1804vm:~# ctr image pull docker.io/library/redis:latest
docker.io/library/redis:latest:                                                   resolved       |+++++++++++++++++++++++++++++++docker.io/library/redis:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:ddf831632db1a51716aa9c2e9b6a52f5035fc6fa98a8a6708f6e83033a49508d:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:780f7dacdc133e899fba9ff09c099828b469030acefe6f3bbc16197b55800cfd: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:75797de34ea7abaf8ebd484896a21f5bda63ffbcade7217dad0be0b8b8333bde:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:f0453552d7f26fc38ffc05fa034aa7a7bc6fbb01bc7bc5a9e4b3c0ab87068627:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:68ced04f60ab5c7a5f1d0b0b4e7572c5a4c8cce44866513d30d9df1a15277d6b:    done           |+++++++++++++++++++++

estesp

Codefresh has a Docker v2-API supporting image registry that is "Powered by Google Container Registry (gcr.io)" (per blog post link).

While gcr.io has its own issues with support of manifest list objects defined in the v2.2 image spec, Codefresh registry seems to have slightly different support than the current implementation of GCR, in that cross-repo blob mount/manifest references don't work at all (they do in GCR); but both registries refuse to accept the media type for a manifest list PUT API call.

Attempting cross-repository push to codefresh registry

estesp

$ ./manifest-tool --debug push from-spec alpine-gcp.yaml 
DEBU[0000] endpoints: [{false https://gcr.io v2 false true 0xc420352000} {false http://gcr.io v2 false true 0xc420352000} {false https://gcr.io v1 false true 
0xc420352180} {false http://gcr.io v1 false true 0xc420352180}] 
DEBU[0000] repoName: test-gcloud-1690/alpine            
INFO[0000] Retrieving digests of images...              
DEBU[0000] authConfig for gcr.io: oauth2accesstoken     
DEBU[0000] endpoints: [{false https://gcr.io v2 false true 0xc420352300} {false http://gcr.io v2 false true 0xc420352300}] 
DEBU[0000] Trying to fetch image manifest of gcr.io/test-gcloud-1690/alpine_ppc64le repository from https://gcr.io v2 
INFO[0001] Image "gcr.io/test-gcloud-1690/alpine_ppc64le:latest" is digest sha256:15780ed5342bafabe42cb609d2ca62534c12c1ae7681b10b978d2a8e781e0957; size: 528 DEBU[0001] Adding manifest references of "gcr.io/test-gcloud-1690/alpine_ppc64le:latest" to blob mount requests 
DEBU[0001] Adding manifest "test-gcloud-1690/alpine_ppc64

estesp

package main

import (
	"bytes"
	"context"
	"fmt"
	"os"
	"strconv"
	"strings"
	"sync"

estesp

The Problem

See Docker issue #24834 for the more complete background. The short story is that if lxcfs is installed (which it is by default in Ubuntu 16.04) and you attempt to use the Docker daemon with user namespaces enabled, most likely you will end up with a confusing "container command could not be invoked error" like the one here:

# docker run --rm busybox date
docker: Error response from daemon: Container command 'date' could not be invoked..

estesp

Image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c

• Packages an exploded tarball as one of the layers: skydns_sources.tar.gz
• This tarball seems to have come from a filesystem where high uids were in use:

/skydns_sources.tar.gz # ls -al
total 12
drwxr-xr-x    3 root     root          4096 Oct 13 20:25 .
drwxr-xr-x 25 root root 4096 Apr 6 18:42 ..

estesp

Sampled from a 1/12/2016 run of standard Linux x64 engine janky run.

Total time of these 25 tests adds up to 663.8 seconds

These 25 make up 2.2% of total tests, but ~29% of total test runtime.

DockerDaemonSuite.TestDaemonNoSpaceleftOnDeviceError            107.143s
DockerTrustSuite.TestTrustedPushWithFailingServer               95.405s
DockerSuite.TestRunSeccompDefaultProfile                        81.711s

estesp

overlayfs copy-up without userns

Setup: create proper directories, with a pre-existing dir (called dir) in lower, and then mount as overlay fs:

$ mkdir upper lower work merged upper lower/dir
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged

You can now create files in the merged/dir to force a copy up of lower/dir to upper/dir:

estesp

Experimental user namespace support

Linux kernel user namespace support provides additional security by enabling a process--and therefore a container--to have a unique range of user and group IDs which are outside the traditional user and group range utilized by the host system. Potentially the most important security improvement is that, by default, container processes running as the root user will have expected administrative privilege (with some restrictions) inside the container but will effectively be mapped to an unprivileged uid on the host.

estesp

package main

import (
	"fmt"
	"net"
	"os"
)

func main() {
	addrs, err := net.LookupIP(os.Args[1])