buildkit notes

buildkit-scratch-pad.txt

## Running as container with docker connection helper

docker container run -d --name buildkitd --privileged moby/buildkit:latest

export BUILDKIT_HOST=docker-container://buildkitd

buildctl build --frontend dockerfile.v0 \
    --local context=. \
    --local dockerfile=. \
    --opt platform=linux/amd64 \
    --output type=image,name=docker.io/jossemargt-cto-ai/image,push=false

----

## Default connection addr

default     
 unix:///run/buildkit/buildkitd.sock

---

## Running rootless

docker run \
    --name buildkitd \
    -d \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    --device /dev/fuse \
    moby/buildkit:rootless --oci-worker-no-process-sandbox


buildctl --addr docker-container://buildkitd build \
    --frontend dockerfile.v0 \
    --local context=. \
    --local dockerfile=. \
    --opt platform=linux/amd64 \
    --output type=image,name=docker.io/jossemargt-cto-ai/image,push=false

## Running rootless + buildctl on another container

The unix socket is on /run/user/1000/buildkit/buildkitd.sock but it is owned by UID 1000
so one could drop "--volume /run/user/1000" still the ownership user:group IDs must match
on the client side.

docker run --rm \
    --name buildkitd \
    --volume /run/user/1000 \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    --device /dev/fuse \
    moby/buildkit:rootless \
        --oci-worker-no-process-sandbox 

EDIT: I mistakenly intended to create a volume /run/user/1000/buildkit/, but it will generate
a root owned diretory, instead one could re-use the /run/user/1000 volume which was pre-declared
on moby/buildkit:rootless image

---

This will work, however it will have the same problem as above, where the UID:GID must be identical
in both ends (buildkit and client)

docker run --rm \
    --name buildkitd \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    --device /dev/fuse \
    moby/buildkit:rootless \
        --addr unix:///home/user/.local/share/buildkit/buildkitd.sock \
        --oci-worker-no-process-sandbox 

docker run --rm -ti \
    --entrypoint '' \
    --volumes-from buildkitd \
    -e BUILDKIT_HOST=unix:///home/user/.local/share/buildkit/buildkitd.sock \
    moby/buildkit:rootless \
    /bin/ash -c \
        'cd /tmp; \
        echo "FROM alpine" > Dockerfile; \
        buildctl build \
            --frontend dockerfile.v0 \
            --local context=. \
            --local dockerfile=. \
            --opt platform=linux/amd64 \
            --output type=image,name=docker.io/jossemargt-cto-ai/image,push=false'

----

Same as above with mixed archs

docker run --rm \
    --name buildkitd \
    --platform linux/arm64 \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    --device /dev/fuse \
    moby/buildkit:rootless \
        --addr unix:///home/user/.local/share/buildkit/buildkitd.sock \
        --oci-worker-no-process-sandbox 

docker run --rm -ti \
    --platform linux/amd64 \
    --entrypoint '' \
    --volumes-from buildkitd \
    -e BUILDKIT_HOST=unix:///home/user/.local/share/buildkit/buildkitd.sock \
    moby/buildkit:rootless \
    /bin/ash -c \
        'cd /tmp; \
        echo "FROM alpine" > Dockerfile; \
        buildctl build \
            --frontend dockerfile.v0 \
            --local context=. \
            --local dockerfile=. \
            --opt platform=linux/amd64 \
            --output type=image,name=docker.io/jossemargt-cto-ai/image,push=false'

----

Working with docker networks

$ docker network create test-buildkit

$ docker run --rm \
    --name buildkitd \
    --network test-buildkit \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    --device /dev/fuse \
    moby/buildkit:rootless \
        --oci-worker-no-process-sandbox \
        --addr \
        unix:///run/user/1000/buildkit/buildkitd.sock \
        --addr \
        tcp://0.0.0.0:1234 

$ docker run --rm \
    --network test-buildkit \
    --entrypoint '' \
    moby/buildkit:rootless \
    buildctl --addr tcp://buildkitd:1234 debug workers
ID                              PLATFORMS
1l1reky9p4jplssusj0speo57       linux/amd64,linux/amd64/v2,linux/386

---

Using with --link (default network)

$ docker run --rm \
    --name buildkitd \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    --device /dev/fuse \
    moby/buildkit:rootless \
        --oci-worker-no-process-sandbox \
        --addr \
        unix:///run/user/1000/buildkit/buildkitd.sock \
        --addr \
        tcp://0.0.0.0:1234 

$ docker run --rm \
    --entrypoint '' \
    --link buildkitd \
    moby/buildkit:rootless \
    buildctl --addr tcp://buildkitd:1234 debug workers
ID                              PLATFORMS4e7b823d9437
22lsddkhqj3ovl2kb4689g6sm       linux/amd64,linux/amd64/v2,linux/386


---

Without network nor --link doesn't work

----

doing test build w/--link

$ docker run --rm  -ti \
    --entrypoint '' \
    --link buildkitd \
    moby/buildkit:rootless \
    /bin/ash -c \
        'cd /tmp; \
        echo "FROM alpine" > Dockerfile; \
        buildctl --addr tcp://buildkitd:1234 build \
            --frontend dockerfile.v0 \
            --local context=. \
            --local dockerfile=. \
            --opt platform=linux/amd64 \
            --output type=image,name=docker.io/jossemargt-cto-ai/image,push=false'

---

As expected, it is easier when buildkitd is spun up as privileged container.
However, the access problem continues either the UID:GID must match when using an
UNIX socket or the containers should be able to find each other through the docker
network when using a TCP one.

docker run --rm \
    --name buildkitd \
    --volume /run/buildkit \
    --privileged \
    moby/buildkit:latest

docker run --rm  -ti \
    --entrypoint '' \
    --volumes-from buildkitd \
    moby/buildkit:latest \
    /bin/ash -c \
        'cd /tmp; \
        echo "FROM alpine" > Dockerfile; \
        buildctl build \
            --frontend dockerfile.v0 \
            --local context=. \
            --local dockerfile=. \
            --opt platform=linux/amd64 \
            --output type=image,name=docker.io/jossemargt-cto-ai/image,push=false'