Skip to content

Instantly share code, notes, and snippets.

@dfr
Last active September 4, 2023 21:52
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save dfr/ac4dc043ee3780b690c5887a61f53494 to your computer and use it in GitHub Desktop.
Save dfr/ac4dc043ee3780b690c5887a61f53494 to your computer and use it in GitHub Desktop.
FreeBSD podman tech demo

This will pull in source code for podman, buildah and related modules, build everything and install to /usr/local. This all happens in a directory named 'build' which can be deleted to clean up or to force a clean build.

mkdir -p build
fetch https://gist.github.com/dfr/ac4dc043ee3780b690c5887a61f53494/raw/1e55da486792ffda61c9d6070d5d834888be9590/buildah-install.sh
chmod +x buildah-install.sh
(cd build && ../buildah-install.sh)

Make a container and run things inside it:

c=$(sudo buildah from docker.io/kwiat/freebsd:13.0-RELEASE)
sudo buildah run $c freebsd-version
sudo buildah run $c ifconfig
sudo buildah rm -a

Download and run images in podman:

sudo podman run --rm docker.io/dougrabson/hello

The containers will use the default 'podman' network which is defined in /usr/local/etc/cni/net.d/87-podman-bridge.conflist. This relies on NAT to allow the container traffic out to the internet and I use pf with the following simple pf.conf:

nat on egress inet from <cni-nat> to any -> (egress)
nat on egress inet6 from <cni-nat> to !ff00::/8 -> (egress)
rdr-anchor "cni-rdr/*"
table <cni-nat>

Note: I'm using the OpenBSD convention to identify the host's main interface by putting it into the 'egress' group using ifconfig, e.g.:

sudo ifconfig vtnet0 group egress

There is a lot of room for improvement in this area - NAT works fairly well for ipv4 but can get confused with ipv6 if the egress interface has non-routable addresses assigned to it. Port mapping is very limited and does not work for connections from localhost. Perhaps someone with better pf skills can help figure out how to get this working (probably needs to NAT from localhost back to the container network).

Stats for running podman containers can be accessed using the 'podman stats' command. This relies on the RACCT accounting framework which is present in GENERIC kernels but must be enabled by adding kern.racct.enable="1" to /boot/loader.conf.

#! /bin/sh
set -x
echo Installing necessary tools and modules
sudo pkg update
sudo pkg install -y bazel go gmake gsed gpgme git bash pkgconf gcc python3
sudo kldload nullfs
sudo kldload fdescfs
sudo kldload pf
sudo sysrc kld_list="nullfs fdescfs pf"
echo Build and install ocijail:
git clone https://github.com/dfr/ocijail.git
(cd ocijail && bazel run //:install -- -s /usr/local/bin)
echo Build and install CNI plugins
git clone -b freebsd https://github.com/dfr/plugins.git
sudo mkdir -p /usr/local/libexec/cni
(cd plugins && ./build_freebsd.sh && sudo cp bin/* /usr/local/libexec/cni)
echo Build and install buildah:
git clone -b v1.28.0 https://github.com/containers/buildah.git
(cd buildah && gmake && sudo install bin/buildah /usr/local/bin)
echo Build and install conmon:
git clone https://github.com/containers/conmon.git
(cd conmon && gmake CC=gcc && sudo install bin/conmon /usr/local/sbin)
echo Build and install podman:
git clone https://github.com/containers/podman.git
(cd podman && gmake && sudo gmake install)
echo Initialise storage:
sudo zfs create -o mountpoint=/var/db/containers zroot/containers
echo Install config files
git clone https://github.com/dfr/containers-etc.git
sudo mkdir -p /usr/local/etc/containers
sudo mkdir -p /usr/local/etc/cni/net.d
sudo cp -r containers-etc/containers/* /usr/local/etc/containers
sudo cp -r containers-etc/cni/net.d/* /usr/local/etc/cni/net.d
@outbackdingo
Copy link

outbackdingo commented Aug 3, 2022

This part of the install process appears broken, there is no freebsd branch in conmon

fatal: Remote branch freebsd not found in upstream origin

echo Build and install conmon:
git clone -b freebsd https://github.com/dfr/conmon.git
(cd conmon && gmake CC=gcc && sudo install bin/conmon /usr/local/sbin)

@dfr
Copy link
Author

dfr commented Aug 3, 2022 via email

@outbackdingo
Copy link

you mean from https://github.com/containers/conmon ?? I tried that and it fails to build, im a marshmallow, directions please :)

root@cbsd:~/build/conmon # gmake
gmake: hack/seccomp-notify.sh: No such file or directory
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/usr/local/include -DVERSION="2.1.3" -DGIT_COMMIT=""4ca293730749b80893216228dd2b07bd5397e23b"" -o src/conmon.o -c src/conmon.c
src/conmon.c:43:22: error: variable 'dev_null_r_cleanup' set but not used [-Werror,-Wunused-but-set-variable]
cleanup_close int dev_null_r_cleanup = -1;
^
src/conmon.c:44:22: error: variable 'dev_null_w_cleanup' set but not used [-Werror,-Wunused-but-set-variable]
cleanup_close int dev_null_w_cleanup = -1;
^
src/conmon.c:45:22: error: variable 'dummyfd' set but not used [-Werror,-Wunused-but-set-variable]
cleanup_close int dummyfd = -1;
^
3 errors generated.
gmake: *** [Makefile:71: src/conmon.o] Error 1

@outbackdingo
Copy link

nevermind it works gmake CC=gcc

@dfr
Copy link
Author

dfr commented Aug 4, 2022

I fixed the 'hack/seccomp-notify.sh: No such file or directory' problem and submitted a PR upstream. I also re-added the freebsd branch (with this PR) so the build script should work again.

@dfr
Copy link
Author

dfr commented Aug 4, 2022

There still seem to be some issues running containers in buildah and podman though when I test this on a fresh VM. I'll work on debugging it

@dfr
Copy link
Author

dfr commented Aug 4, 2022

Running 'kldload fdescfs' helps get buildah going - I updated the build script and freshened up the podman branch so that it contains the support for fdescfs.

@dfr
Copy link
Author

dfr commented Oct 18, 2022

Updated to build buildah, conmon and podman from upstream sources rather than my work-in-progress branches.

@tendstofortytwo
Copy link

Hi, just a note, running images via podman fails now with the error: Error: OCI runtime error: ocijail: malformed ociVersion 1.1.0-rc.2 . This seems to be partially because ocijail doesn't parse the version correctly, and partially because it doesn't support versions > 1.0.x.

@dfr
Copy link
Author

dfr commented May 13, 2023

Hi, just a note, running images via podman fails now with the error: Error: OCI runtime error: ocijail: malformed ociVersion 1.1.0-rc.2 . This seems to be partially because ocijail doesn't parse the version correctly, and partially because it doesn't support versions > 1.0.x.

I'll take some time to update ocijail to work with the latest oci version. In the meantime, it might be best to install from ports or pkg - I'll be updating them too over the next week or so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment