Journal log for kata where devmapper snapshotter is configured

gistfile1.txt

Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.257589778Z" level=info msg="Agent started in the sandbox" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=
sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.333213792Z" level=debug msg="looking for backing file" major=0 minor=154 name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=pkg/device
/config
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.333282634Z" level=debug msg="no loop device" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d error="open /sys/dev/block/0:154/loop/backing_file: no such file or directory
" mount-source=/run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/shm name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers s
ubsystem=container
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.333341368Z" level=debug msg="looking for backing file" major=252 minor=1 name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=pkg/device
/config
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.333386015Z" level=debug msg="no loop device" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d error="open /sys/dev/block/252:1/loop/backing_file: no such file or directory
" mount-source=/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri/sandboxes/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/resolv.conf name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a
4d source=virtcontainers subsystem=container
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.333426655Z" level=info msg="device details" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d device-major=253 device-minor=3 mount-point= name=containerd-shim-v2 pid=63726
 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=container
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.333476056Z" level=info msg="Block device detected" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d device-path=/dev/dm-3 fs-type=ext4 name=containerd-shim-v2 pid=63726 sa
ndbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=container
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.333571156Z" level=info msg="Attaching virtio-scsi device" VirtPath= device=/dev/dm-3 name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d sourc
e=virtcontainers subsystem=device
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.386207342Z" level=info msg="Attach devices" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d devices="[]" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5
c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=container
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.39019292Z" level=info msg="Using sandbox shm" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d shm-size=67108864 source=virtcontainers subs
ystem=kata_agent
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.390234801Z" level=debug msg="ociSpec HugepageLimit" name=containerd-shim-v2 ociSpec Hugepage Resources="[]" pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source
=virtcontainers subsystem=kata_agent
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.39026202Z" level=debug msg="Replacing OCI mount (/etc/resolv.conf) source /var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri/sandboxes/616b959586e55e2e44b57e5c8988a323cfb29c6b671a84235
5cbfb268f891a4d/resolv.conf with /run/kata-containers/shared/containers/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d-ff0f1bd4e50b9150-resolv.conf" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891
a4d source=virtcontainers subsystem=kata_agent
Nov 14 02:25:02 hchoi-gha-test-01 systemd[1]: run-kata\x2dcontainers-shared-sandboxes-616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d-private-616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d\x2dff0f1bd4e50b9150\x2dresolv.conf.mount: Deac
tivated successfully.
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.479033919Z" level=debug msg="Setting container state from  to ready" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d name=containerd-shim-v2 pid=63726 sandbox=616b959586e
55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=container
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.479901117Z" level=debug msg="updateResources: after calculateSandboxMemory" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d sandboxMemoryB
yte=2147483648 sandboxSwapByte=0 sandboxneedPodSwap=false source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.480021485Z" level=debug msg="Request to hypervisor to update vCPUs" cpus-sandbox=1 name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=
virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.48009116Z" level=debug msg="Request to hypervisor to update oldCPUs/newCPUs: 1/1" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=v
irtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.480191082Z" level=debug msg="Sandbox CPUs: 1" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.480244811Z" level=debug msg="Request to hypervisor to update memory" memory-sandbox-size-mb=2048 name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f
891a4d source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.480303634Z" level=debug msg="Sandbox memory size: 2048 MB" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem
=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.522441196Z" level=info msg="Stopping sandbox in the VM" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=sa
ndbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525579418Z" level=info msg="Stopping VM" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525612797Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers/hypervisor subsystem=q
emu
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525676425Z" level=info msg="Cleanup vm path" dir=/run/vc/vm/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d link=/run/vc/vm/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb26
8f891a4d name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers/hypervisor subsystem=qemu
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525736511Z" level=info msg="Detaching endpoint" endpoint-type=virtual name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainer
s subsystem=network
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525762229Z" level=debug msg="Endpoints removed" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=network
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525804867Z" level=info msg="virtiofsd quits" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers/hypervisor subsystem=vi
rtiofsd
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525874545Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers/hypervisor subsystem=q
emu
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525894155Z" level=info msg="Already stopped" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers/hypervisor subsystem=qe
mu
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525883612Z" level=debug msg="Deleting sandbox &{%!s(*v2.Manager=&{/sys/fs/cgroup /sys/fs/cgroup/kubepods-besteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b5
7e5c8988a323cfb29c6b671a842355cbfb268f891a4d}) /kubepods-besteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d %!s(*specs.LinuxCPU=&{<nil> <nil> <nil> <nil> <nil> <nil>   <nil>}) [{%!s(bo
ol=false)  %!s(*int64=<nil>) %!s(*int64=<nil>) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d6b0) %!s(*int64=0xc00036d6b8) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d6c0) %!s(*int64=0xc00036d6c8) rwm} {%!s(bool=true) c %!s(*int64=0xc000416578) %!s(*int64=0xc000416580) rwm}
 {%!s(bool=true) c %!s(*int64=0xc0004165a8) %!s(*int64=0xc0004165b0) rwm} {%!s(bool=true) c %!s(*int64=0xc0004165d8) %!s(*int64=0xc0004165e0) rwm} {%!s(bool=true) c %!s(*int64=0xc000416608) %!s(*int64=0xc000416610) rwm} {%!s(bool=true) c %!s(*int64=0xc000416638) %!s(*in
t64=0xc000416640) rwm} {%!s(bool=true) c %!s(*int64=0xc000416668) %!s(*int64=0xc000416670) rwm} {%!s(bool=true) c %!s(*int64=0xc000416698) %!s(*int64=0xc0004166a0) rwm} {%!s(bool=true) c %!s(*int64=0xc0004166c8) %!s(*int64=0xc0004166d0) rwm} {%!s(bool=true) c %!s(*int64
=0xc0004167b8) %!s(*int64=0xc0004167c0) rwm} {%!s(bool=true) c %!s(*int64=0xc0004167e8) %!s(*int64=0xc0004167f0) rwm} {%!s(bool=true) c %!s(*int64=0xc000416818) %!s(*int64=0xc000416820) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d7e8) %!s(*int64=0xc00036d7f0) m} {%!s(boo
l=true) b %!s(*int64=0xc00036d7e8) %!s(*int64=0xc00036d7f0) m} {%!s(bool=true) c %!s(*int64=0xc00036d7f8) %!s(*int64=0xc00036d7f0) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d800) %!s(*int64=0xc00036d808) rwm} {%!s(bool=true) b %!s(*int64=0xc00044bcb8) %!s(*int64=0xc0004
4bcc0) rwm}] {%!s(int32=0) %!s(uint32=0)}} resource controler" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525924378Z" level=info msg="scanner return error: <nil>" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers/hypervisor
subsystem=qmp
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526007872Z" level=error msg="failed to cleanup the &{%!s(*v2.Manager=&{/sys/fs/cgroup /sys/fs/cgroup/kubepods-besteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2
e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d}) /kubepods-besteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d %!s(*specs.LinuxCPU=&{<nil> <nil> <nil> <nil> <nil> <nil>   <nil>}) [{%
!s(bool=false)  %!s(*int64=<nil>) %!s(*int64=<nil>) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d6b0) %!s(*int64=0xc00036d6b8) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d6c0) %!s(*int64=0xc00036d6c8) rwm} {%!s(bool=true) c %!s(*int64=0xc000416578) %!s(*int64=0xc000416580)
 rwm} {%!s(bool=true) c %!s(*int64=0xc0004165a8) %!s(*int64=0xc0004165b0) rwm} {%!s(bool=true) c %!s(*int64=0xc0004165d8) %!s(*int64=0xc0004165e0) rwm} {%!s(bool=true) c %!s(*int64=0xc000416608) %!s(*int64=0xc000416610) rwm} {%!s(bool=true) c %!s(*int64=0xc000416638) %!
s(*int64=0xc000416640) rwm} {%!s(bool=true) c %!s(*int64=0xc000416668) %!s(*int64=0xc000416670) rwm} {%!s(bool=true) c %!s(*int64=0xc000416698) %!s(*int64=0xc0004166a0) rwm} {%!s(bool=true) c %!s(*int64=0xc0004166c8) %!s(*int64=0xc0004166d0) rwm} {%!s(bool=true) c %!s(*
int64=0xc0004167b8) %!s(*int64=0xc0004167c0) rwm} {%!s(bool=true) c %!s(*int64=0xc0004167e8) %!s(*int64=0xc0004167f0) rwm} {%!s(bool=true) c %!s(*int64=0xc000416818) %!s(*int64=0xc000416820) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d7e8) %!s(*int64=0xc00036d7f0) m} {%!
s(bool=true) b %!s(*int64=0xc00036d7e8) %!s(*int64=0xc00036d7f0) m} {%!s(bool=true) c %!s(*int64=0xc00036d7f8) %!s(*int64=0xc00036d7f0) rwm} {%!s(bool=true) c %!s(*int64=0xc00036d800) %!s(*int64=0xc00036d808) rwm} {%!s(bool=true) b %!s(*int64=0xc00044bcb8) %!s(*int64=0x
c00044bcc0) rwm}] {%!s(int32=0) %!s(uint32=0)}} resource controllers" error="lstat /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice/cri-containerd-616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355
cbfb268f891a4d.scope: no such file or directory" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526076875Z" level=info msg="QMP event channel closed" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers/hypervisor sub
system=qemu
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.525938454Z" level=info msg="console watcher quits" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526165091Z" level=info msg="Cleanup agent" name=containerd-shim-v2 path=/run/kata-containers/shared/sandboxes/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/shared pid=63726 sandb
ox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=fs_share
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526355319Z" level=error msg="failed to Cleanup vm path /run/kata-containers/shared/sandboxes/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d" error="unlinkat /run/kata-containers/s
hared/sandboxes/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/mounts/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d-ff0f1bd4e50b9150-resolv.conf: device or resource busy" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b5
7e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=fs_share
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526402637Z" level=error msg="failed to cleanup share files" error="unlinkat /run/kata-containers/shared/sandboxes/616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/mounts/616b959586e
55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d-ff0f1bd4e50b9150-resolv.conf: device or resource busy" name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d source=virtcontainers subsystem=sandbox
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526475384Z" level=debug msg="Create() end" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29
c6b671a842355cbfb268f891a4d source=containerd-kata-shim-v2
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526544771Z" level=debug msg="State() end" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29c
6b671a842355cbfb268f891a4d source=containerd-kata-shim-v2
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526895027Z" level=debug msg="Delete() start" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb
29c6b671a842355cbfb268f891a4d source=containerd-kata-shim-v2
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.526944356Z" level=debug msg="Delete() end" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323cfb29
c6b671a842355cbfb268f891a4d source=containerd-kata-shim-v2
Nov 14 02:25:02 hchoi-gha-test-01 kata[63726]: time="2023-11-14T02:25:02.527467232Z" level=debug msg="Shutdown() start" container=616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d name=containerd-shim-v2 pid=63726 sandbox=616b959586e55e2e44b57e5c8988a323c
fb29c6b671a842355cbfb268f891a4d source=containerd-kata-shim-v2
Nov 14 02:25:02 hchoi-gha-test-01 systemd[1]: run-kata\x2dcontainers-shared-sandboxes-616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d-shared-616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d\x2dff0f1bd4e50b9150\x2dresolv.conf.mount: Deact
ivated successfully.
Nov 14 02:25:02 hchoi-gha-test-01 systemd[1]: run-kata\x2dcontainers-shared-sandboxes-616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d-shared.mount: Deactivated successfully.
Nov 14 02:25:02 hchoi-gha-test-01 k3s[61753]: E1114 02:25:02.787896   61753 remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: write /sys/fs/cgroup/kubepo
ds-besteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/cgroup.procs: invalid argument: unknown"
Nov 14 02:25:02 hchoi-gha-test-01 k3s[61753]: E1114 02:25:02.787948   61753 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: write /sys/fs/cgroup/kubepods-be
steffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/cgroup.procs: invalid argument: unknown" pod="kata-containers-k8s-tests/handlers"
Nov 14 02:25:02 hchoi-gha-test-01 k3s[61753]: E1114 02:25:02.787980   61753 kuberuntime_manager.go:1122] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: write /sys/fs/cgroup/kubepods-b
esteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/cgroup.procs: invalid argument: unknown" pod="kata-containers-k8s-tests/handlers"
Nov 14 02:25:02 hchoi-gha-test-01 k3s[61753]: E1114 02:25:02.788051   61753 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"handlers_kata-containers-k8s-tests(3b5a9f6e-5448-40a1-9535-f4b61b28ba72)\" with CreatePodSandboxError
: \"Failed to create sandbox for pod \\\"handlers_kata-containers-k8s-tests(3b5a9f6e-5448-40a1-9535-f4b61b28ba72)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: write /sys/fs/cgroup/kubepods-besteffort-pod3b5a9f6e_544
8_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d/cgroup.procs: invalid argument: unknown\"" pod="kata-containers-k8s-tests/handlers" podUID=3b5a9f6e-5448-40a1-9535-f4b61b28ba72
Nov 14 02:25:03 hchoi-gha-test-01 systemd[1]: run-k3s-containerd-io.containerd.grpc.v1.cri-sandboxes-616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d-shm.mount: Deactivated successfully.
Nov 14 02:25:05 hchoi-gha-test-01 k3s[61753]: W1114 02:25:05.631466   61753 manager.go:1159] Failed to process watch event {EventType:0 Name:/kubepods-besteffort-pod3b5a9f6e_5448_40a1_9535_f4b61b28ba72.slice:cri-containerd:616b959586e55e2e44b57e5c8988a323cfb29c6b671a842
355cbfb268f891a4d WatchSource:0}: container "616b959586e55e2e44b57e5c8988a323cfb29c6b671a842355cbfb268f891a4d" in namespace "k8s.io": not found
Nov 14 02:25:15 hchoi-gha-test-01 systemd[1]: var-lib-rancher-k3s-agent-containerd-tmpmounts-containerd\x2dmount2625705067.mount: Deactivated successfully.