Adding an updated local binary to an OCP4 coreos node

local-to-coreos.md

EDIT: Nevermind these instructions don't quite work because

$ findmnt -o PROPAGATION /rootfs
PROPAGATION
private

in the MCD pod. So we need a two-step process for oc cp; or better, teach oc cp localfile node/$devnode to work.

Desire: Get updated ostree/podman/kubelet/whatever binary live into target node

In this example I'm using https://github.com/coreos/rpm-ostree/ but it could be any binary.

First, you will need a dev environment matching the target node OS. Setting that up is a bit out of scope of this, but e.g. if you're targeting RHCOS8 which is RHEL8, you can set up a toolbox container based on rhel8, etc.

Anyways, the rest of this assumes you have successfully built the upstream project from git in a container or VM.

make (or whatever build command)

Now you'll have a new target/debug/rpm-ostree binary (or whatever).

Here's my remote cluster:

oc get nodes -o wide
NAME                                       STATUS   ROLES    AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
ci-ln-0fib782-72292-brw5r-master-0         Ready    master   84m   v1.22.1+bac83a5   10.0.0.4      <none>        Red Hat Enterprise Linux CoreOS 410.84.202111302222-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ci-ln-0fib782-72292-brw5r-master-1         Ready    master   84m   v1.22.1+bac83a5   10.0.0.3      <none>        Red Hat Enterprise Linux CoreOS 410.84.202111302222-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ci-ln-0fib782-72292-brw5r-master-2         Ready    master   84m   v1.22.1+bac83a5   10.0.0.5      <none>        Red Hat Enterprise Linux CoreOS 410.84.202111302222-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ci-ln-0fib782-72292-brw5r-worker-a-bjd8k   Ready    worker   71m   v1.22.1+bac83a5   10.0.128.4    <none>        Red Hat Enterprise Linux CoreOS 410.84.202111302222-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ci-ln-0fib782-72292-brw5r-worker-b-crfn9   Ready    worker   74m   v1.22.1+bac83a5   10.0.128.2    <none>        Red Hat Enterprise Linux CoreOS 410.84.202111302222-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ci-ln-0fib782-72292-brw5r-worker-c-mgbjb   Ready    worker   74m   v1.22.1+bac83a5   10.0.128.3    <none>        Red Hat Enterprise Linux CoreOS 410.84.202111302222-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8

I chose the last worker as my "dev node". export devnode=ci-ln-0fib782-72292-brw5r-worker-c-mgbjb

Now, let's unlock the rootfs: oc debug node/$devnode -- nsenter -m -t 1 -- rpm-ostree usroverlay

And now, oc cp lets us copy a binary to a pod. We make use of the fact that the MCD pod on the node has the host rootfs mounted. Let's find the MCD pod:

$ oc -n openshift-machine-config-operator get pods -o wide | grep -F $devnode
machine-config-daemon-z9pxz                  2/2     Running   0          76m   10.0.128.3    ci-ln-0fib782-72292-brw5r-worker-c-mgbjb   <none>           <none>

Let's save that mcd pod: export mcd=$(oc -n openshift-machine-config-operator get pods -o wide | grep -F $devnode | cut -f 1 -d ' ')

Now:

oc -n openshift-machine-config-operator cp target/debug/rpm-ostree machine-config-daemon-z9pxz:/rootfs/usr/bin/rpm-ostree

Will live-copy my local binary to the target node rootfs.