inhumantsar/notes.md

## notes.md

      
    Raw
  

              notes.md
            
          
    SELinux == Pain

Always a shitshow.
after migrating over to CoreOS and rootless podman, the servarr stack has been throwing thousands of SELinux errors all the time. caddy generated a bunch of errors and slowed down during startup. plex, sonarr, and radarr shat the bed hard. plex in particular would lock up and take minutes to launch because of it, impacting everything else going on. this evening it took 4 tries to get it started, even after i set selinux to permissive.
no fucking more.

people say "udica". it's apparently a python script that generates SELinux configs for podman containers. let's give it a go.
step one: install udica. pip or native pkg manager...
$ sudo rpm-ostree install udica
$ sudo rpm-ostree apply-live         # yolo

following redhat docs... (sidenote: for all redhat/fedora's flaws, they really do have the most badass set of docs. everything you could ever want to do is in there... somewhere.)
$ podman inspect caddy > caddy.json
$ sudo udica -j caddy.json caddy

Policy caddy created!

Please load these modules using:
# semodule -i caddy.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:caddy.process" parameter

$ cat caddy.cil
(block caddy
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process http_port_t ( tcp_socket (  name_bind )))
    (allow process http_port_t ( tcp_socket (  name_bind )))
    (allow process reserved_port_t ( udp_socket (  name_bind )))
    (allow process unreserved_port_t ( tcp_socket (  name_bind )))
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))

    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))


$ sudo semodule -i caddy.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

adding security label type blah blah to the quadlet...
$ vi caddy.container
$ cat caddy.container
# caddy.container
[Container]
...
SecurityLabelType=caddy.process
...

$ install_podman_units

$ systemctl --user restart caddy
$ podman inspect caddy
...
               "SecurityOpt": [
                    "label=type:caddy.process"
               ],
...

$ ps -efZ | grep caddy.process
system_u:system_r:caddy.process:s0:c875,c876 sam 49911 49909  0 23:32 ?    00:00:00 caddy docker-proxy

check cockpit... and lol no.
still getting port, socket, and permission denied on Caddyfile.
fucking file labels, as if octal wasn't weird enough

this site describes checking and fixing file labels... let's try that.
$ sudo ls -lZ data/
[sudo] password for sam:
total 14
-rw-------. 1 590824 590824 unconfined_u:object_r:unlabeled_t:s0      199 Apr  4 15:48 Caddyfile
drwx------. 7 590824 590824 system_u:object_r:container_file_t:s0       9 Apr 11 23:32 caddy
drwxr-xr-x. 2 590824 590824 unconfined_u:object_r:container_file_t:s0   5 Apr  4 06:54 custom_ssl

$ sudo semanage fcontext -a -t container_file_t data/*
$ sudo ls -lZ data/
total 14
-rw-------. 1 590824 590824 unconfined_u:object_r:unlabeled_t:s0      199 Apr  4 15:48 Caddyfile
drwx------. 7 590824 590824 system_u:object_r:container_file_t:s0       9 Apr 11 23:32 caddy
drwxr-xr-x. 2 590824 590824 unconfined_u:object_r:container_file_t:s0   5 Apr  4 06:54 custom_ssl

$ sudo restorecon -RF data/*

well, it's not reporting that caddyfile issue anymore at least...
let's see if that survives a container restart
$ systemctl --user restart caddy

so far so good. still getting socket errors though:

SELinux is preventing caddy from name_connect access on the tcp_socket port 9095.

type=AVC msg=audit(1712879017.460:1759): avc: denied { name_connect } for pid=49911 comm="caddy" dest=9095 scontext=system_u:system_r:caddy.process:s0:c875,c876 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=1


SELinux is preventing caddy from connectto access on the unix_stream_socket /run/user/1001/podman/podman.sock

type=AVC msg=audit(1712879049.550:1765): avc: denied { connectto } for pid=49911 comm="caddy" path="/run/user/1001/podman/podman.sock" scontext=system_u:system_r:caddy.process:s0:c875,c876 tcontext=unconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=1


SELinux is preventing caddy from name_connect access on the tcp_socket port 9095.

type=AVC msg=audit(1712879108.668:1779): avc: denied { name_connect } for pid=56796 comm="caddy" dest=9091 scontext=system_u:system_r:caddy.process:s0:c629,c681 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=1

let's try udica again...
$ podman inspect caddy > caddy2.json
$ mv caddy.cil caddy.cil.1
$ sudo udica -j caddy2.json caddy
$ diff caddy.cil.1 caddy.cil
...nothing...

k...
let's try fucking with this cil file

$ cat caddy.cil
(block caddy
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process http_port_t ( tcp_socket (  name_bind )))
    (allow process http_port_t ( tcp_socket (  name_bind )))
    (allow process reserved_port_t ( udp_socket (  name_bind )))
    (allow process unreserved_port_t ( tcp_socket (  name_bind )))
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))

    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))

hmm. lots of file stuff.. container_file_t would be for the normal mounted things. var_t i guess would be files that the system thinks are system files. http_port_t makes sense since the container wants to listen on 80 and 443, though why two? and why? it's not binding to those ports. it's rootless so it only gets to bind to 8000 and 4443. 8000 is probably the reserved_port_t. maybe the loopbacks?
for comparison...
$ cat ../../configs/caddy/caddy.container
# caddy.container
[Container]
ContainerName=caddy
Environment=CADDY_DOCKER_NO_SCOPE=true CADDY_DOCKER_CADDYFILE_PATH=/data/Caddyfile
Image=localhost/tinglecaddy:latest
AutoUpdate=local
SecurityLabelType=caddy.process

# we'll bind containers to this loopback using labels and unique ports. default is 10.0.2.2
Network=slirp4netns:allow_host_loopback=true,port_handler=slirp4netns
PublishPort=8000:80
PublishPort=4443:443

...

anyway, let's start with this error:
denied { name_connect } blah blah dest=9091
scontext=system_u:system_r:caddy.process:s0:c629,c681 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket

so it's denying name_connect on 9091.. scontext i guess just defines the source of the selinux request. tcontext would be the target. unreserved_port_t is a name... tcp_socket is the type...  and we want to allow name_connect...
looks like this line in the .cil file is the biggun:
    (allow process unreserved_port_t ( tcp_socket (  name_bind )))

maybe add name_connect after name_bind?
$ vi caddy.cil
$ diff caddy.cil.1 caddy.cil
9c9
<     (allow process unreserved_port_t ( tcp_socket (  name_bind )))
---
>     (allow process unreserved_port_t ( tcp_socket (  name_bind name_connect )))
$ sudo semodule -i caddy.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}
$ systemctl --user restart caddy
$ systemctl --user restart sonarr

that seems to have done the thing for the port XXXX errors
now trying the unix_stream_socket one...
$ cp caddy.cil caddy.cil.2
$ vi caddy.cil
$ diff caddy.cil caddy.cil.2
18d17
<     (allow process container_runtime_t ( unix_stream_socket ( connectto )))


🎉 we might actually have a fix on our hands...

k so now sonarr. seems like most of the containers deal with the same set of errors so hopefully it's simply a matter of the file label thing that the Caddyfile needed...
type=AVC msg=audit(1712879977.948:2017): avc: denied { read } for pid=11578 comm=504D53204C696255706461746572 name=43686170746572203230202D20546865205369656765206F6620746865204E6F7274682028506172742054776F292E786D6C dev="zfs" ino=32975 scontext=system_u:system_r:container_t:s0:c430,c445 tcontext=system_u:object_r:container_file_t:s0:c356,c692 tclass=file permissive=1

$ podman inspect sonarr > sonarr.json
$ sudo udica -j sonarr.json sonarr
Policy sonarr created!

Please load these modules using:
# semodule -i sonarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:sonarr.process" parameter

$ cat sonarr.cil
(block sonarr
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process unreserved_port_t ( tcp_socket (  name_bind )))
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))

$ vi sonarr.container
$ cat sonarr.container
[Container]
...
Volume=/var/rust/nassau/sonarr:/config:Z
Volume=/var/rust/nassau/tv:/tv:Z
Volume=/var/rust/nassau/downloads:/downloads:Z
SecurityLabelType=sonarr.process
...

hmmm... those :Zs...

reminds me of a reddit comment i saw earlier:

Avoid using :Z unless the bounded directory is going to be used for that specific container and nothing else. A lower-case :z is ideal when a directory has a purpose beyond that specific container.

$ vi sonarr.container
$ cat sonarr.container
[Container]
...
Volume=/var/rust/nassau/sonarr:/config:z
Volume=/var/rust/nassau/tv:/tv:z
Volume=/var/rust/nassau/downloads:/downloads:z
SecurityLabelType=sonarr.process
...

k.
let's try it without redoing udica first...
$ sudo semodule -i sonarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

$ install_podman_units
Creating or updating /home/sam/.config/containers/systemd/plex.container...
Creating or updating /home/sam/.config/containers/systemd/prowlarr.container...
Creating or updating /home/sam/.config/containers/systemd/radarr.container...
Creating or updating /home/sam/.config/containers/systemd/requestrr.container...
Creating or updating /home/sam/.config/containers/systemd/sabnzbd.container...
Creating or updating /home/sam/.config/containers/systemd/sonarr.container...
Creating or updating /home/sam/.config/containers/systemd/transmission.container...

$ systemctl --user restart sonarr

no file errors at least. had a lot of those the last time.

SELinux is preventing curl from name_connect access on the tcp_socket port 8989.

8989 is its own port tho...


type=AVC msg=audit(1712881094.748:2081): avc: denied { name_connect } for pid=76268 comm="curl" dest=8989 scontext=system_u:system_r:sonarr.process:s0:c747,c789 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=1


SELinux is preventing .NET ThreadPool from lock access on the file /proc/stat.

this would be its own /proc/stat tho...


type=AVC msg=audit(1712881094.844:2082): avc: denied { lock } for pid=76248 comm=2E4E455420546872656164506F6F6C path="/proc/stat" dev="proc" ino=4026532041 scontext=system_u:system_r:sonarr.process:s0:c747,c789 tcontext=system_u:object_r:proc_t:s0 tclass=file permissive=1

ok so maybe :Z is probably a good idea on its config dir mount.
$ vi sonarr.container
$ cat sonarr.container
...
Volume=/var/rust/nassau/sonarr:/config:Z
...

and the connect issue should be the same fix as before...
$ vi sonarr.cil
$ cat sonarr.cil
...
    (allow process unreserved_port_t ( tcp_socket (  name_bind name_connect )))
...

$ sudo semodule -i sonarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}
$ systemctl --user restart sonarr

k. no more connect errors but...
proc_t: fix it or fuck it?

the /proc/stat (and now /proc/filesystems too) error is still there: SELinux is preventing .NET ThreadPool from lock access on the file /proc/filesystems
type=AVC msg=audit(1712881630.94:2128): avc: denied { lock } for pid=82146 comm=2E4E455420546872656164506F6F6C path="/proc/filesystems" dev="proc" ino=4026532091 scontext=system_u:system_r:sonarr.process:s0:c281,c880 tcontext=system_u:object_r:proc_t:s0 tclass=file permissive=1


/proc/stat offers up info about CPU activity
/proc/filesystems describes what filesystems the system supports

$ ls -lZ /proc/stat
-r--r--r--. 1 root root system_u:object_r:proc_t:s0 0 Apr 11 22:20 /proc/stat
$ $ cat /proc/stat
cpu  207149 43329 121827 20152537 219286 11889 8157 0 43205 0
cpu0 17614 303 9016 824686 10739 741 793 0 11148 0
cpu1 12738 88 3030 841833 6448 378 879 0 9913 0
cpu2 9577 313 4645 838524 11190 568 288 0 4592 0
cpu3 6613 83 2559 847899 7743 265 345 0 4083 0
cpu4 7814 199 4392 841182 10774 579 320 0 3026 0
cpu5 4311 41 2552 851375 6958 270 81 0 1787 0
cpu6 7104 196 4413 841738 10766 594 295 0 2354 0
cpu7 3726 63 2397 852638 6366 293 88 0 1236 0
cpu8 31931 15666 14482 783178 18020 752 641 0 1421 0
cpu9 14281 4475 9325 827087 9613 351 270 0 1113 0
cpu10 32586 14669 15295 782683 17885 792 748 0 1486 0
cpu11 13826 5051 9338 825687 11027 344 146 0 1040 0
cpu12 8208 663 8258 831631 15029 831 427 0 0 0
cpu13 2887 199 2068 851281 8944 254 68 0 0 0
cpu14 6425 517 6024 832863 17549 1098 651 0 0 0
cpu15 2455 107 1758 851718 8808 210 657 0 0 0
cpu16 4062 129 3872 848488 7204 666 226 0 0 0
cpu17 3765 112 3295 850853 5813 626 272 0 0 0
cpu18 3360 94 2941 851337 6330 548 314 0 0 0
cpu19 3094 101 2624 852318 6341 441 124 0 0 0
cpu20 2689 67 2526 855619 3975 338 131 0 0 0
cpu21 2570 65 2340 855829 4118 311 163 0 0 0
cpu22 2528 52 2230 856140 4127 274 102 0 0 0
cpu23 2974 65 2434 855939 3506 353 116 0 0 0
...

$ ls -lZ /proc/filesystems
-r--r--r--. 1 root root system_u:object_r:proc_t:s0 0 Apr 11 22:20 /proc/filesystems
$ cat /proc/filesystems
nodev   sysfs
nodev   tmpfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cgroup2
nodev   cpuset
nodev   devtmpfs
nodev   configfs
nodev   debugfs
nodev   tracefs
nodev   securityfs
nodev   sockfs
nodev   bpf
nodev   pipefs
nodev   ramfs
nodev   hugetlbfs
nodev   devpts
        ext3
        ext2
        ext4
nodev   autofs
nodev   efivarfs
nodev   mqueue
nodev   selinuxfs
nodev   binder
        btrfs
nodev   pstore
        fuseblk
nodev   fuse
nodev   fusectl
        xfs
nodev   zfs
nodev   rpc_pipefs
nodev   overlay
nodev   binfmt_misc


$ podman exec -it sonarr /bin/bash
root@830267ea1de3:/# cat /proc/stat
cpu  206968 43329 121636 20073725 218753 11848 8135 0 43205 0
cpu0 17605 303 9003 821484 10645 738 791 0 11148 0
cpu1 12736 88 3028 838513 6448 377 877 0 9913 0
cpu2 9572 313 4636 835217 11190 566 287 0 4592 0
cpu3 6612 83 2557 844636 7684 264 345 0 4083 0
cpu4 7808 199 4376 837886 10771 576 318 0 3026 0
cpu5 4310 41 2550 848052 6958 270 81 0 1787 0
cpu6 7090 196 4398 838442 10766 590 294 0 2354 0
cpu7 3726 63 2396 849315 6366 293 87 0 1236 0
cpu8 31896 15666 14463 780021 17911 750 641 0 1421 0
cpu9 14274 4475 9317 824038 9356 350 270 0 1113 0
cpu10 32534 14669 15273 779435 17885 789 747 0 1486 0
cpu11 13821 5051 9332 822376 11026 343 146 0 1040 0
cpu12 8201 663 8249 828330 15027 829 425 0 0 0
cpu13 2886 199 2065 847961 8944 253 68 0 0 0
cpu14 6416 517 6007 829572 17547 1095 649 0 0 0
cpu15 2454 107 1756 848396 8808 210 657 0 0 0
cpu16 4059 129 3865 845189 7204 661 224 0 0 0
cpu17 3761 112 3287 847550 5811 624 271 0 0 0
cpu18 3355 94 2934 848032 6329 547 312 0 0 0
cpu19 3092 101 2620 849001 6341 441 124 0 0 0
cpu20 2686 67 2521 852305 3975 336 130 0 0 0
cpu21 2567 65 2335 852516 4117 310 163 0 0 0
cpu22 2525 52 2226 852825 4126 273 102 0 0 0
cpu23 2973 65 2429 852622 3506 352 115 0 0 0
...

root@830267ea1de3:/# cat /proc/filesystems
nodev   sysfs
nodev   tmpfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cgroup2
nodev   cpuset
nodev   devtmpfs
nodev   configfs
nodev   debugfs
nodev   tracefs
nodev   securityfs
nodev   sockfs
nodev   bpf
nodev   pipefs
nodev   ramfs
nodev   hugetlbfs
nodev   devpts
        ext3
        ext2
        ext4
nodev   autofs
nodev   efivarfs
nodev   mqueue
nodev   selinuxfs
nodev   binder
        btrfs
nodev   pstore
        fuseblk
nodev   fuse
nodev   fusectl
        xfs
nodev   zfs
nodev   rpc_pipefs
nodev   overlay
nodev   binfmt_misc

so these files are root-owned and read-only. they don't expose anything crazy.
but.
/proc isn't just used for friendly info, that's where running processes live.
that's a good thing though, because container engines mesh the two together. some things are exposed to the containers anyway and others are strictly for the container's use.
this only has a small number of processes, all owned by the container's root while all the system stuff owned by nobody is the host's.
root@830267ea1de3:/# ls -l /proc
total 0
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 1
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 11
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 13
dr-xr-xr-x   9 abc    abc       0 Apr 11 19:27 141
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 142
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 34
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 35
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 36
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 37
dr-xr-xr-x   9 root   root      0 Apr 11 19:27 45
dr-xr-xr-x   9 root   root      0 Apr 11 19:47 498
dr-xr-xr-x   9 root   root      0 Apr 11 19:48 514
drwxrwxrwt   2 root   root     40 Apr 11 19:27 acpi
dr-xr-xr-x  19 nobody nobody    0 Apr 11 19:27 asound
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 bootconfig
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 buddyinfo
dr-xr-xr-x   4 nobody nobody    0 Apr 11 19:27 bus
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 cgroups
-r--r--r--   1 nobody nobody  407 Apr 11 19:43 cmdline
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 consoles
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 cpuinfo
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 crypto
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 devices
-r--r--r--   1 nobody nobody    0 Apr 11 19:43 diskstats
...

it should be safe to apply but it might also be worth seeing if sonarr eats shit without permission to lock on proc_t resources.
i think i'll leave it for now and move on. still no file errors since the restart. that's a good sign.
seems safe to repeat the process with the others...

$ podman inspect radarr > radarr.json
$ sudo udica -j radarr.json radarr
Policy radarr created!

Please load these modules using:
# semodule -i radarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:radarr.process" parameter

$ cat radarr.cil
(block radarr
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process unreserved_port_t ( tcp_socket (  name_bind )))
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))

this doesn't have name_connect either, so let's pre-emptively add that before applying it...
$ sudo chown sam: radarr.cil
$ sed -e 's/tcp_socket (  name_bind )))/tcp_socket (  name_bind name_connect )))/' radarr.cil
...
    (allow process unreserved_port_t ( tcp_socket (  name_bind name_connect )))
...

lgtm, it'll be good to just use sed rather than vi from here on...
$ sed -i 's/tcp_socket (  name_bind )))/tcp_socket (  name_bind name_connect )))/' radarr.cil
$ sudo semodule -i radarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

k, now this has the :Z's like sonarr did. let's fix those while adding the security label...
$ sed -e 's/s:Z/s:z/' radarr.container
...
Volume=/var/rust/nassau/radarr:/config:Z
Volume=/var/rust/nassau/movies:/movies:z
Volume=/var/rust/nassau/downloads:/downloads:z
...

handy that the dirs we want to change all end in s.
$ sed -e 's/^Label/SecurityLabelType=sonarr.process\n\nLabel/' radarr.container
...
Volume=/var/rust/nassau/downloads:/downloads:Z

SecurityLabelType=sonarr.process

Label=caddy...

good enough. well, apart from sonarr.process
$ sed -i 's/s:Z/s:z/' radarr.container
$ sed -i 's/^Label/SecurityLabelType=radarr.process\n\nLabel/' radarr.container
$ cat radarr.container
...
Volume=/var/rust/nassau/radarr:/config:Z
Volume=/var/rust/nassau/movies:/movies:z
Volume=/var/rust/nassau/downloads:/downloads:z

SecurityLabelType=radarr.process
...

lgtm. let's goooo
$ install_podman_units
Creating or updating /home/sam/.config/containers/systemd/plex.container...
Creating or updating /home/sam/.config/containers/systemd/prowlarr.container...
Creating or updating /home/sam/.config/containers/systemd/radarr.container...
Creating or updating /home/sam/.config/containers/systemd/requestrr.container...
Creating or updating /home/sam/.config/containers/systemd/sabnzbd.container...
Creating or updating /home/sam/.config/containers/systemd/sonarr.container...
Creating or updating /home/sam/.config/containers/systemd/transmission.container...
$ systemctl --user restart radarr 

no file errors, just the same proc_t ones sonarr had. 🎉
time for a wee script

#!/bin/bash
[[ -z $1 ]] && echo "gimme a container name dummy" && exit 1
podman inspect $1 > $1.json
sudo udica -j $1.json $1
sed -i 's/tcp_socket (  name_bind )))/tcp_socket (  name_bind name_connect )))/' $1.cil

echo
cat $1.cil
read -p "good? " -n 1 -r
echo
[[ $REPLY =~ ^[Yy]$ ]] || exit 1
sudo semodule -i $1.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

sed -i 's/s:Z/s:z/' $1.container
sed -i "s/^Label/SecurityLabelType=$1.process\n\nLabel/" $1.container

echo
cat $1.container
read -p "good? " -n 1 -r
echo
[[ $REPLY =~ ^[Yy]$ ]] || exit 1
install_podman_units
systemctl --user restart $1

$ vi fixup.sh
$ chmod +x fixup.sh
$ ./fixup.sh
gimme a container name dummy
$ ./fixup.sh prowlarr
[sudo] password for sam:

Policy prowlarr created!

Please load these modules using:
# semodule -i prowlarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:prowlarr.process" parameter

cat: prowlarr.cli: No such file or directory
good? ^C

derp
$ vi fixup.sh
$ ./fixup.sh prowlarr

Policy prowlarr created!

Please load these modules using:
# semodule -i prowlarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:prowlarr.process" parameter

(block prowlarr
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process neutron_port_t ( tcp_socket (  name_bind name_connect )))
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))
)good?

hmm... neutron_port_t. that's a new one. openstack's networking stack apparently uses 9696, which is what prowlarr is using in my setup. since i'm not using openstack or neutron, it's really just a naming issue here and should be functionally equiv to the others.
onward!
$ ./fixup.sh prowlarr

Policy prowlarr created!

Please load these modules using:
# semodule -i prowlarr.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:prowlarr.process" parameter

(block prowlarr
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process neutron_port_t ( tcp_socket (  name_bind name_connect )))
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))
)good? y

# prowlarr.container
[Container]
ContainerName=prowlarr
Environment=PGID=9999 PUID=9999 TZ=America/Chicago
Image=lscr.io/linuxserver/prowlarr:develop
Volume=/var/rust/nassau/prowlarr:/config:Z
Volume=/var/rust/nassau/downloads:/downloads:z

SecurityLabelType=prowlarr.process

...

good? y
Creating or updating /home/sam/.config/containers/systemd/plex.container...
Creating or updating /home/sam/.config/containers/systemd/prowlarr.container...
Creating or updating /home/sam/.config/containers/systemd/radarr.container...
Creating or updating /home/sam/.config/containers/systemd/requestrr.container...
Creating or updating /home/sam/.config/containers/systemd/sabnzbd.container...
Creating or updating /home/sam/.config/containers/systemd/sonarr.container...
Creating or updating /home/sam/.config/containers/systemd/transmission.container...

no selinux errors! 🎉
$ ./fixup.sh requestrr
...

(block requestrr
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process unreserved_port_t ( tcp_socket (  name_bind name_connect )))
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))
)good? y

...
Volume=/var/rust/nassau/requestrr:/root/config:Z

SecurityLabelType=requestrr.process
...

good? y
...
Creating or updating /home/sam/.config/containers/systemd/requestrr.container...
...

this hasn't thrown any selinux errors to date as far as i'm aware, so no issues here.
onward!
sab's turn:
$ ./fixup.sh sabnzbd
...
    (allow process http_cache_port_t ( tcp_socket (  name_bind name_connect )))
...
)good?

same as before, sab is using 8080 which is falls under http_cache_port_t: tcp 3128, 8080, 8118.
onward!
...
)good? y
...
Volume=/var/rust/nassau/sabnzbd:/config:Z
Volume=/var/rust/nassau/downloads:/downloads:z

SecurityLabelType=sabnzbd.process
...

good? y
...
Creating or updating /home/sam/.config/containers/systemd/sabnzbd.container...
...

a new selinux error appears!

sab's throwing a new proc_t, this time for ioctl perms on /proc/cpuinfo. going to ignore this one like the others and see if it shits the bed when this goes back to enforcing.
there was an actual new one too: SELinux is preventing python3 from name_connect access on the tcp_socket port 563.
type=AVC msg=audit(1712885421.835:2274): avc: denied { name_connect } for pid=116016 comm="python3" dest=563 scontext=system_u:system_r:sabnzbd.process:s0:c249,c836 tcontext=system_u:object_r:hi_reserved_port_t:s0 tclass=tcp_socket permissive=1

563 is an NNTP port. not something we would consider a "privileged" port today, but it's old enough to have been standardized with a sub-1000 number before privileged ports were really a thing. nbd there, but is there a way to scope the selinux policy so that it can't connect on other privileged ports?
my provider does offer the same services on port 443 though.. and i suspect doesn't have the same kind of selinux restriction. trying that first...

logged into sab.
Settings -> Servers
Show Details
Check Advanced Settings
Change 563 to 443
Test and Save
systemctl --user restart sabnzbd.service

now, it just so happened that sab got a new linux iso to download at the last reboot (or was checking up on one recently added). it's currently waiting 3 minutes before trying to download. this will be a good live test for sonarr's new selinux setup. 🍿
iso downloaded and imported without an selinux warning! 🎉
onward!
transmission is a special snowflake


it rawdogs the tcp and udp torrent ports for incoming connections
a quirk in naming and standards mean the :z sed trick won't work

so i manually made the :z change before running fixup.sh, and yeah:
    (allow process unreserved_port_t ( tcp_socket (  name_bind name_connect )))
    (allow process ephemeral_port_t ( tcp_socket (  name_bind name_connect )))
    (allow process ephemeral_port_t ( udp_socket (  name_bind )))

i don't think transmission will need to talk to itself the way the *arrs do.

[!NOTE] Editor's Note
Foreshadowing...

cancelling that and going manual from here.
$ vi transmission.cil
$ cat transmission.cil
...
    (allow process unreserved_port_t ( tcp_socket (  name_bind )))
    (allow process ephemeral_port_t ( tcp_socket (  name_bind )))
    (allow process ephemeral_port_t ( udp_socket (  name_bind )))
...

$ sudo semodule -i transmission.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

$ sed -i 's/s:Z/s:z/' transmission.container
$ sed -i "s/^Label/SecurityLabelType=$1.process\n\nLabel/" transmission.container
$ cat transmission.container
...
Volume=/var/rust/nassau/downloads:/downloads:z
Volume=/var/rust/nassau/downloads/incomplete:/incomplete:z
Volume=/var/rust/nassau/transmission:/config:Z
Volume=/etc/localtime:/etc/localtime:ro

# raw dog the bittorrent ports
PublishPort=51413:51413
PublishPort=51413:51413/udp

# connect up to the reverse proxy.
SecurityLabelType=.process
...

derp
$ sed -i 's/SecurityLabelType=.process/SecurityLabelType=transmission.process/' transmission.container
sam@hogfather:/var/rust/nassau/quadlets$ cat transmission.container
...
SecurityLabelType=transmission.process
...

better. the # connect up to the reverse proxy comment is in the wrong place now too, but whatever, i still have to go back into these another time and fix the system startup relationships (the [Install]\nAfter=... is a straight up syntax error...) so fuck it.
$ install_podman_units
...
$ systemctl --user restart transmission

oh no! selinux errors again

SELinux is preventing blah from name_connect access on the tcp_socket port moo.

type=AVC msg=audit(1712886916.651:2309): avc: denied { name_connect } for pid=131395 comm="nc" dest=9091 scontext=system_u:system_r:transmission.process:s0:c554,c830 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=1

type=AVC msg=audit(1712886918.992:2310): avc: denied { name_connect } for pid=131391 comm="transmission-da" dest=49152 scontext=system_u:system_r:transmission.process:s0:c554,c830 tcontext=system_u:object_r:virt_migration_port_t:s0 tclass=tcp_socket permissive=1

so the first one indicates that it does need to talk to itself. the second one is a little weirder.
virt_migration_port_t is a tricky one too. makes me wonder how of this named port types i'll run into. hopefully most will fall under unreserved or ephemeral...
the tricky part about that is that i do use libvirtd and i do use those ports on the host system. used to be that i'd just trust transmission's devs to not try and fuck with my VMs, but in light of the xz backdooring, that's harder to do these days.
it's certainly possible to set it up so that the policy only permits internet-bound connections on that port, but i don't particularly want to get into dicking around with iptables on this host. not atm.
more and more i'm seeing a lot of good reason to move all this spaghetti into a dedicated VM and pass block devices through to it. afaik though it's not even possible to pass a zfs pool through, never mind a single mount, and all my shit is on a single pool. i've done the nfs share thing but that was slow and a pita.
next time i pick up a new pair of 20TB+ disks, i'll migrate everything over to LVM or maybe finally get an HBM (unlikely). i'd lose some performance to LVM but not as much as i would to NFS.
honestly though, sab covers 99.9% of things, so fuck it. i'll fix the first error and let the other one ride. transmission won't shit the bed over it.
$ vi transmission.cil
$ cat transmission.cil
...
    (allow process unreserved_port_t ( tcp_socket (  name_bind name_connect )))
...
$ sudo semodule -i transmission.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}
$ systemctl --user restart transmission

no more error for port 9090, so it seems transmission is talking to itself just fine.
onward!
the main event

plex.
this mfer has thrown all kinds of errors since day one. mainly file access errors. it's also a special snowflake in that it needs to:

expose a port to the internet
have access to /dev/dri so it can transcode on the intel igpu
use Network=host so it can do things like mdns announcements and stream locally

let's see how this goes.
$ ./fixup.sh plex

Policy plex created!

Please load these modules using:
# semodule -i plex.cil /usr/share/udica/templates/base_container.cil

Restart the container with: "--security-opt label=type:plex.process" parameter

(block plex
    (blockinherit container)
    (allow process process ( capability ( chown dac_override fowner fsetid kill net_bind_service setfcap setgid setpcap setuid sys_chroot )))

    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process container_file_t ( sock_file ( append getattr open read write )))
    (allow process var_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write )))
    (allow process var_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write )))
    (allow process var_t ( fifo_file ( getattr read write append ioctl lock open )))
    (allow process var_t ( sock_file ( append getattr open read write )))
)good? y

# plex.container
[Container]
ContainerName=plex
Environment=PUID=9999 PGID=9999 TZ=Etc/UTC VERSION=latest
Image=lscr.io/linuxserver/plex:latest
Network=host
Tmpfs=/transcode
AddDevice=/dev/dri:/dev/dri
Volume=/var/rust/nassau/plex:/config:Z
Volume=/var/rust/nassau/tv:/tv:Z
Volume=/var/rust/nassau/movies:/movies:z

[Service]
Restart=on-failure

[Install]
After=ready.target

good? ^C

ok so the .cil file looks pretty sparse compared to the others, likely because it's not using the same publish port mechanism as the others. i'm also not seeing anything for /dev/dri in there. i'm not sure i've ever seen selinux errors related to it though, so maybe that's fine?
tv doesn't end in s so the sed trick didn't work either. i'll fix that first.
$ vi plex.container
$ install_podman_units
$ systemctl --user restart plex

🤞
no errors yet... let's try fucking with it. first, force a scan of the libraries.

still no errors. now how about we change some metadata... (andrew scott is the best btw)

still good. what if we create an optimized version...

(lol, i can tell when it has to write stripes to the older drives, 20x drops to 9x and jumps back up)
still no errors! 🎉
apparently it even added some thumbnails in the background too:
Generated new chapter thumbnails for Fallen Angels.
Generated new chapter thumbnails for Pine Barrens.
Generated new chapter thumbnails for University.
Generated new chapter thumbnails for Another Toothpick.
Generated new chapter thumbnails for Guy Walks into a Psychiatrist's Office.

ok well that's it for nassau services.
now for the big test

time to turn enforcing on and restart them all again, one by one, watching to see if any of them shit the bed.
$ systemctl --user restart <service> && podman logs -f <service>


 Caddy ✅ 2024-04-11
 Sonarr ✅ 2024-04-11
 Radarr ✅ 2024-04-11
the usual /proc errors from these two, no bed shitting
 Prowlarr ✅ 2024-04-11
 Sabnzbd ✅ 2024-04-11
 Transmission ✅ 2024-04-11
connect: Permission denied -> name_connect access on the tcp_socket port 49152.
no bed shitting
 Requestrr ✅ 2024-04-11
no selinux errors
some warnings about ephemeral encryption keys and 2 "config changed restarting bot" messages
no bed shitting (i think, todo)
 Plex ✅ 2024-04-11

🎉🎉🎉

now in fairness, this isn't all of my container services. cockpit and jupyter might get angry after a reboot. the windows VM is also not running atm, so it might have something to say later. but for now, seems like great success.
fuck what a pita though.