Skip to content

Instantly share code, notes, and snippets.

@jvns

jvns/containers-arent-magic.sh Secret

Last active February 25, 2024 23:55
Show Gist options
  • Star 70 You must be signed in to star a gist
  • Fork 16 You must be signed in to fork a gist
  • Save jvns/ea2e4d572b4e2285148b8e87f70eed73 to your computer and use it in GitHub Desktop.
Save jvns/ea2e4d572b4e2285148b8e87f70eed73 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
containers-arent-magic.sh
# you have to run this as root
# It only runs on Linux (namespaces and cgroups only exist on Linux)
# if you don't have it, cgcreate is in the libcgroup package
set -eux # let's be safe
# Download the container (it's in a github gist published by my github account)
# This is just the frapsoft/fish Fish Docker container flattened into a single tarball
# You can also easily make your own tarball to run instead of this one with `docker export`
wget bit.ly/fish-container -O fish.tar
# extract fish.tar into a directory
mkdir container-root
cd container-root
tar -xf ../fish.tar
# generate a random cgroup id
uuid="cgroup_$(shuf -i 1000-2000 -n 1)"
# create the cgroup
cgcreate -g "cpu,cpuacct,memory:$uuid"
# assign CPU/memory limits to the cgroup
cgset -r cpu.shares=512 "$uuid"
cgset -r memory.limit_in_bytes=1000000000 "$uuid"
# The following line does a lot of work:
# 1. cgexec: use our new cgroup
# 2. unshare: make and use a new PID, network, hostname, and mount namespace
# 3. chroot: change root directory to current directory
# 4. mount: use the right /proc in our new mount namespace
# 5. hostname: change the hostname in the new hostname namespace to something fun
cgexec -g "cpu,cpuacct,memory:$uuid" \
unshare -fmuipn --mount-proc \
chroot "$PWD" \
/bin/sh -c "/bin/mount -t proc proc /proc && hostname container-fun-times && /usr/bin/fish"
# Here are ome fun things to try once you're running your container!
# Run them both in the container and in a normal shell and see the difference
# - ps aux
# - ifconfig
# - hostname
@jvns
Copy link
Author

jvns commented Oct 1, 2020

What commands didn't work exactly? You might need to install some tools, like the above comment mentioned about the libcgroup-tools package

@ndom91
Copy link

ndom91 commented Jul 2, 2022 •
edited

@jvns I've updated it a bit to check for existing files/directories when run multiple times. But also updates the commands for cgroups v2!

There's some new controllers and a single hierarchy design. For example, for memory there are these two pertinent values

  • memory.high: the kernel will attempt to keep memory usage below this configuration.
  • memory.max: if memory reaches this level the OOM killer (a system used to sacrifice one or more processes to free up memory for the system when all else fails) is invoked on the cgroup.

For cpu's, I haven't quite figured it out yet haha. There is no more cpuacct, the new cpu controller superseded cpu and cpuacct. However, I don't understand exactly how cpu.max works yet. It can take a range from 0 to 100000. 20000 seems to work fine for this demo..

# you have to run this as root
# It only runs on Linux (namespaces and cgroups only exist on Linux)
# if you don't have it, cgcreate is in the libcgroup package

set -eux # let's be safe

# Download the container (it's in a github gist published by my github account)
# This is just the frapsoft/fish Fish Docker container flattened into a single tarball
# You can also easily make your own tarball to run instead of this one  with `docker export`
if [[ ! -f "$(pwd)/fish.tar" ]]; then
  wget bit.ly/fish-container -O fish.tar
fi

# prepare cgroup directory
if [[ -d "$(pwd)/container-root" ]]; then
  rm -r container-root
fi

mkdir container-root
cd container-root

# extract fish.tar into a directory
tar -xf ../fish.tar

# generate a random cgroup id
uuid="cgroup_$(shuf -i 1000-2000 -n 1)"

# create the cgroup
cgcreate -g "cpu,memory:$uuid"

# assign CPU/memory limits to the cgroup
cgset -r cpu.max=20000 "$uuid"
cgset -r memory.max=1000000000 "$uuid"

# The following line does a lot of work:
# 1. cgexec: use our new cgroup
# 2. unshare: make and use a new PID, network, hostname, and mount namespace
# 3. chroot: change root directory to current directory
# 4. mount: use the right /proc in our new mount namespace
# 5. hostname: change the hostname in the new hostname namespace to something fun
cgexec -g "cpu,memory:$uuid" \
  unshare -fmuipn --mount-proc \
  chroot "$PWD" \
  /bin/sh -c "/bin/mount -t proc proc /proc && hostname container-fun-times && /usr/bin/fish"

# Here are ome fun things to try once you're running your container!
# Run them both in the container and in a normal shell and see the difference
# - ps aux
# - ifconfig
# - hostname

Runs out of the box on an up-to-date Arch system (after installing libcgroup). Ubuntu 22.04 should also have cgroups v2 by default now.

@vosechu
Copy link

vosechu commented Feb 20, 2024

My friends and I tried to run this today on an M1 mac and of course, got an Exec format error. When we ran file bin/sh it says it's built for x86-64 (which I believe is intel, not ARM). So if people get that error, that's why. We didn't find a way around it before the end of our timebox.

Thank you @ndom91 for the updated instructions! That was really helpful and got us almost to the end!

@vosechu
Copy link

vosechu commented Feb 21, 2024 via email

@jvns
Copy link
Author

jvns commented Feb 21, 2024

I don't remember exactly, but I think I made a Docker container and then used docker export to export it as a tarball

@vosechu
Copy link

vosechu commented Feb 25, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment