Podman container exit killing Sway WM (running as a systemd service)

README.md

the issue

sway process stops as soon as podman container exits, eg podman run --rm -it bash -c 'exit 0'

see resolution.md

background

For the last 1-2 years after switching XFCE on Arch Linux to Sway (Wayland) on NixOS I had on and off issues with podman container exits resulting in sway window manager exiting.

It might have been caused by moving sway process inside a systemd service, but at the beginning I was not using enough containers to pinpoint the time.

findings

container's exit code doesn't matter

podman run --rm -it bash -c 'exit 1' has the same effect

the issue is not triggered until container exits

it can run indefinitely, but as soon as container exits Sway closes down

executing from SSH session doesn't trigger the issue

podman run --rm -it bash -c 'exit 0' executed from within SSH session does not impact anything (there is no issue)

container exit results in a clean shutdown of the WM

Nov 08 13:12:10 oams sway[5909]: 00:08:59.248 [DEBUG] [sway/ipc-server.c:386] Sending shutdown::exit event

processes end up in the wrong systemd *.slice when running under Sway

see cgroups-sway.md

they end up in the proper slice when running under SSH, see cgroups-ssh.md

something connects to Sway's IPC when container starts

Nov 08 15:27:06 oams sway[22442]: 00:13:03.684 [DEBUG] [sway/ipc-server.c:198] New client: fd 95
Nov 08 15:27:06 oams sway[22442]: 00:13:03.685 [INFO] [sway/ipc-server.c:558] IPC Client 95 disconnected
Nov 08 15:27:07 oams sway[22442]: 00:13:05.525 [DEBUG] [sway/ipc-server.c:328] Sending window::title event

oci-containers issues on NixOS

Remco> I ended up at https://github.com/NixOS/nixpkgs/issues/249332
Remco> And through that on https://github.com/NixOS/nixpkgs/pull/248315
Remco> Which has:
Remco> "This probably usually goes unnoticed because systemd follows up by sending SIGTERM to all processes in the control group and this is respected. But if this signal is ignored, the container isn't stopped until systemd's timeout of 120 seconds expires."
Remco> So if it sends a sigterm not just for the containers but also for conmon/slirp, that means sway would shut down

cgroups-ssh.md

CGroups assigned in SSH session

systemctl --user status excerpt

└─user.slice
 ├─libpod-27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef.scope
 │ └─container
 │   └─20743 bash
 ├─libpod-conmon-27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef.scope
 │ └─20741 /nix/store/7pq7ivk0n64dx77gqyg9rqqcp9gj6vfv-conmon-2.1.8/bin/conmon --api-version 1 -c 27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef -u 27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef -r /nix/store/c18yxnslk3g2ccj1dmq087rm5hsknqny-crun-1.10/bin/crun -b /home/kdn/.local/share/containers/storage/vfs-containers/27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef/userdata -p /run/user/31893/containers/vfs-containers/27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef/userdata/pidfile -n charming_noether --exit-dir /run/user/31893/libpod/tmp/exits --full-attach -s -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/31893/containers/vfs-containers/27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef/userdata/oci-log -t --conmon-pidfile /run/user/31893/containers/vfs-containers/27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef/userdata/conmon.pid --exit-command /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped --exit-command-arg --root --exit-command-arg /home/kdn/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/31893/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/31893/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/kdn/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 27af1ee832b1fa1a3c24da67fbac697a3a7aad7ed7ce7cd47e27497ec3ffc4ef
 ├─podman-20715.scope
 │ ├─20715 /run/current-system/sw/bin/podman run --rm -it bash
 │ └─20738 /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/libexec/podman/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 -e 4 --netns-type=path /run/user/31893/netns/netns-302bdf2a-7dc7-6d58-7acb-93b3704a2248 tap0
 └─podman-pause-21c2ac77.scope
   └─17453 /run/current-system/sw/bin/podman

cgroups-sway.md

CGroups assigned under Sway session

systemctl --user status excerpt

├─session.slice
...
│ ├─kdn-sway.service
│ │ ├─17931 /nix/store/45mjrvjzhp0daa1iypbhyq03schk3bcb-sway-unwrapped-1.8.1/bin/sway --verbose --debug
│ │ ├─18177 /nix/store/7j3di4mic85i6c172x9ghcx2h1yy3mh0-swayr-0.27.1/bin/swayrd
...
│ │ ├─21063 /run/current-system/sw/bin/podman run --rm -it bash
│ │ ├─21081 /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/libexec/podman/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 -e 4 --netns-type=path /run/user/31893/netns/netns-96c1c63d-003a-8fc8-1119-97d8981ddf30 tap0
│ │ └─21087 /nix/store/7pq7ivk0n64dx77gqyg9rqqcp9gj6vfv-conmon-2.1.8/bin/conmon --api-version 1 -c 9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9 -u 9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9 -r /nix/store/c18yxnslk3g2ccj1dmq087rm5hsknqny-crun-1.10/bin/crun -b /home/kdn/.local/share/containers/storage/vfs-containers/9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9/userdata -p /run/user/31893/containers/vfs-containers/9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9/userdata/pidfile -n zen_ishizaka --exit-dir /run/user/31893/libpod/tmp/exits --full-attach -s -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/31893/containers/vfs-containers/9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9/userdata/oci-log --sdnotify-socket=/run/user/31893/systemd/notify -t --conmon-pidfile /run/user/31893/containers/vfs-containers/9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9/userdata/conmon.pid --exit-command /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped --exit-command-arg --root --exit-command-arg /home/kdn/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/31893/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/31893/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/kdn/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9
...
└─user.slice
 ├─libpod-9745405db529477fa5fac1f587e73ee6aae9e01dfefc3f6cfc1de86d659d51a9.scope
 │ └─container
 │   └─21089 bash
 └─podman-pause-21c2ac77.scope
   └─17453 /run/current-system/sw/bin/podman

clean-shutdown.md

Sway shuts down cleanly after container exits

Nov 08 13:12:09 oams podman[10949]: 2023-11-08 13:12:09.41543012 +0100 CET m=+0.032067838 system refresh
Nov 08 13:12:09 oams podman[10949]: 2023-11-08 13:12:09.425024693 +0100 CET m=+0.041662830 image pull 9877ef007e1a526be54451dfd369e9e180f18104a7e9733f5831e90cec1e60cc bash
Nov 08 13:12:09 oams podman[10949]: 2023-11-08 13:12:09.541445546 +0100 CET m=+0.158083264 container create 61588e677c7ca01f8f0f7856d7843427d82e92ca68caac2d3ed904f3f5719795 (image=docker.io/library/bash:latest, name=fervent_raman)
Nov 08 13:12:09 oams podman[10949]: 2023-11-08 13:12:09.605994678 +0100 CET m=+0.222632885 container init 61588e677c7ca01f8f0f7856d7843427d82e92ca68caac2d3ed904f3f5719795 (image=docker.io/library/bash:latest, name=fervent_raman)
Nov 08 13:12:09 oams podman[10949]: 2023-11-08 13:12:09.608186837 +0100 CET m=+0.224824556 container start 61588e677c7ca01f8f0f7856d7843427d82e92ca68caac2d3ed904f3f5719795 (image=docker.io/library/bash:latest, name=fervent_raman)
Nov 08 13:12:09 oams podman[10949]: 2023-11-08 13:12:09.612252752 +0100 CET m=+0.228891378 container attach 61588e677c7ca01f8f0f7856d7843427d82e92ca68caac2d3ed904f3f5719795 (image=docker.io/library/bash:latest, name=fervent_raman)
Nov 08 13:12:10 oams fervent_raman[10974]: [27B blob data]
Nov 08 13:12:10 oams fervent_raman[10974]: exit
Nov 08 13:12:10 oams podman[10949]: 2023-11-08 13:12:10.863486843 +0100 CET m=+1.480124562 container died 61588e677c7ca01f8f0f7856d7843427d82e92ca68caac2d3ed904f3f5719795 (image=docker.io/library/bash:latest, name=fervent_raman)
Nov 08 13:12:10 oams podman[10986]: 2023-11-08 13:12:10.948496014 +0100 CET m=+0.079581923 container remove 61588e677c7ca01f8f0f7856d7843427d82e92ca68caac2d3ed904f3f5719795 (image=docker.io/library/bash:latest, name=fervent_raman)
Nov 08 13:12:10 oams sway[5909]: 00:08:59.248 [DEBUG] [sway/ipc-server.c:386] Sending shutdown::exit event
Nov 08 13:12:10 oams sway[5909]: 00:08:59.248 [INFO] [sway/main.c:418] Shutting down sway
Nov 08 13:12:10 oams sway[5909]: 00:08:59.248 [DEBUG] [sway/desktop/layer_shell.c:347] Layer surface destroyed (waybar)

resolution.md

Resolution

there are a many ways to problem's resolution:

1. kdn-sway.service:
   1. replace NotifyAccess=all with something else (main/exec)
   2. do not export NOTIFY_SOCKET to dbus-update-activation-environment --systemd
      1. alternatively rename or clear it completely
   3. integrate session manager like sway-systemd to automatically distribute cgroups among applications
      • this might not work for short-lived containers
2. operating system/podman configuration:
   1. operating system: change podman defaults/OS configuration to make it run with --sdnotify=ignore
3. podman implementation:
   1. modify detection of running within systemd service less eager to properly discover this case

Root cause

The root cause is combination of:

1. my window manager (Sway) is running as a systemd service kdn-sway.service of Type=notify and NotifyAccess=all
2. podman by default is sending systemd notifications (sdnotify/systemd-notify)
   • sdnotify implementation
   • setting MAINPID=
3. sending MAINPID= notification results in takeover of the service, search the man page for MAINPID=
   • in effect the latest started container becomes kdn-sway.service's primary process
   • which in turn results in cleanly shutting down Window Manager upon that latest container's exit

follow-up actions

1. implement a working solution:
   1. preferably NotifyAccess=exec or NotifyAccess=main
   2. confirm whether removing NOTIFY_SOCKET export works at all
2. create a https://github.com/containers/podman issue asking for comments/possible resolutions
3. ease discovery of the issue for other people:
   1. create issue at https://github.com/swaywm/sway
      1. add documentation pieces at systemd-integration page https://github.com/swaywm/sway/wiki/Systemd-integration
   2. create issue at https://github.com/NixOS/nixpkgs
4. add a warning to NotifyAccess=all documentation at https://github.com/systemd/systemd

with-debug.md

running podman with --log-level=debug

journal when running with --log-level=debug

Nov 08 15:51:43 oams podman[4937]: 2023-11-08 15:51:43.950871691 +0100 CET m=+176.206970936 container died bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf (image=docker.io/library/alpine:latest, name=festive_chatterjee)
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Called cleanup.PersistentPreRunE(/nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped --root /home/kdn/.local/share/containers/storage --runroot /run/user/31893/containers --log-level debug --cgroup-manager systemd --tmpdir /run/user/31893/libpod/tmp --network-config-dir  --network-backend netavark --volumepath /home/kdn/.local/share/containers/storage/volumes --db-backend boltdb --transient-store=false --runtime crun --storage-driver vfs --events-backend journald --syslog container cleanup --rm bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf)"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Setting custom database backend: \"boltdb\""
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using conmon from $PATH: \"/nix/store/7pq7ivk0n64dx77gqyg9rqqcp9gj6vfv-conmon-2.1.8/bin/conmon\""
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Initializing boltdb state at /home/kdn/.local/share/containers/storage/libpod/bolt_state.db"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using graph driver vfs"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using graph root /home/kdn/.local/share/containers/storage"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using run root /run/user/31893/containers"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using static dir /home/kdn/.local/share/containers/storage/libpod"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using tmp dir /run/user/31893/libpod/tmp"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using volume path /home/kdn/.local/share/containers/storage/volumes"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using transient store: false"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="[graphdriver] trying provided driver \"vfs\""
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Initializing event backend journald"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="using runtime \"crun\" from $PATH: \"/nix/store/c18yxnslk3g2ccj1dmq087rm5hsknqny-crun-1.10/bin/crun\""
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="using runtime \"runc\" from $PATH: \"/nix/store/m53qvf8p1x20982hcsfl083qg7njz9r0-runc-1.1.9/bin/runc\""
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Using OCI runtime \"/nix/store/c18yxnslk3g2ccj1dmq087rm5hsknqny-crun-1.10/bin/crun\""
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=info msg="Setting parallel job count to 49"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Checking if container bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf should restart"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Removing container bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Cleaning up container bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Tearing down network namespace at /run/user/31893/netns/netns-6a98ca23-db75-2a66-4847-2967aad5b0a9 for container bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Successfully cleaned up container bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf"
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Unmounted container \"bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf\""
Nov 08 15:51:43 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:43+01:00" level=debug msg="Removing all exec sessions for container bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf"
Nov 08 15:51:44 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:44+01:00" level=debug msg="Container bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf storage is already unmounted, skipping..."
Nov 08 15:51:44 oams podman[5016]: 2023-11-08 15:51:44.037595179 +0100 CET m=+0.079373377 container remove bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf (image=docker.io/library/alpine:latest, name=festive_chatterjee)
Nov 08 15:51:44 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:44+01:00" level=debug msg="Called cleanup.PersistentPostRunE(/nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped --root /home/kdn/.local/share/containers/storage --runroot /run/user/31893/containers --log-level debug --cgroup-manager systemd --tmpdir /run/user/31893/libpod/tmp --network-config-dir  --network-backend netavark --volumepath /home/kdn/.local/share/containers/storage/volumes --db-backend boltdb --transient-store=false --runtime crun --storage-driver vfs --events-backend journald --syslog container cleanup --rm bb6220a2a2b2e0cf5d7376858b87336d77ddc9f519d48c39a9137696200639cf)"
Nov 08 15:51:44 oams /nix/store/bxvv96w2n1ikaxi5ayp44jawnalpsmh8-podman-4.7.0/bin/.podman-wrapped[5016]: time="2023-11-08T15:51:44+01:00" level=debug msg="Shutting down engines"
Nov 08 15:51:44 oams sway[3546]: 00:07:15.638 [DEBUG] [sway/ipc-server.c:386] Sending shutdown::exit event
Nov 08 15:51:44 oams sway[3546]: 00:07:15.638 [INFO] [sway/main.c:418] Shutting down sway
Nov 08 15:51:44 oams sway[3546]: 00:07:15.638 [DEBUG] [sway/ipc-server.c:328] Sending window::close event

running podman within systemd-run --user foot terminal doesn't trigger the issue, so theory that container exit kills the whole cgroup/slice holds

podman run --sdnotify=ignore fixed the issue completely, it must be interacting with how i run my Sway service:

        serviceConfig = {
          Type = "notify";
          NotifyAccess = "all";