morlay/buildx_create_multi_arch_builder_in_k8s.md

## buildx_create_multi_arch_builder_in_k8s.md

      
    Raw
  

              buildx_create_multi_arch_builder_in_k8s.md
            
          
    Buildx create multi-arch builder in k8s

With QEMU emulation

$ KUBECONFIG=${KUBECONFIG} \
    docker buildx create \
        --name=builder \
        --platform=linux/amd64,linux/arm64 \
        --driver=kubernetes \
        --driver-opt=namespace=buildkit,qemu.install=true
Known Issues

QEMU only work well for executing, but for compiling, it will be very slow and may be throw segmentation fault.
In the mode, if still want to compile binaries in docker.
please use FROM --platform=${BUILDPLATFORM} to disable QEUM for compiling stage.
Example for golang:
FROM --platform=${BUILDPLATFORM} golang:1.6 as builder

ARG TARGETARCH
RUN GOARCH=${TARGETARCH} go build -o /bin/app-linux-${TARGETARCH} ./path/to/cmd/app

FROM scratch

ARG TARGETARCH
COPY --from=builder /bin/app-linux-${TARGETARCH} /bin/app
With native nodes

# create builder `builder` and add native x86_64 node
$ KUBECONFIG=${KUBECONFIG} \
    docker buildx create \
        --name=builder \
        --platform=linux/amd64 \
        --node=builder-amd64 \
        --driver=kubernetes \
        --driver-opt=namespace=buildkit,nodeselector="beta.kubernetes.io/arch=amd64"

# append node to same builder with native aarch64 node
$ KUBECONFIG=${KUBECONFIG} \
    docker buildx create \
        --name=builder --append \
        --platform=linux/arm64 \
        --node=builder-arm64 \
        --driver=kubernetes \
        --driver-opt=namespace=buildkit,nodeselector="beta.kubernetes.io/arch=arm64"

KUBECONFIG could be different.
buildx create executing on a pod of multi-arch cluster, KUBECONFIG could be unset, but make sure the pod serviceAccount could access deplopments,pods,configmaps of assigned namespace

Known Issues

In this mode, docker build for different arch on matched native host.
The build time may be longer.
Even FROM --platform=${BUILDPLATFORM} defined, all stages will build for each arch.
However, it is totally native.
Projects, witch needs to build on native system, will be happy with the mode.
Tips

Once buildx create in k8s, the created deployments will not be removed until buildx rm called.
So we could set RUN --mount=type=cache for sharing common caches for different projects (Note: not sharing across different nodes).
However, For nodejs user.
Don't shared npm or yarn global caches (pnpm will be better), restoring caches may be slower than reinstalling.