Skip to content

Instantly share code, notes, and snippets.

@tprelog
Last active November 23, 2023 01:58
Show Gist options
  • Star 40 You must be signed in to star a gist
  • Fork 7 You must be signed in to fork a gist
  • Save tprelog/7988dc6b196775f33929beb19f0090d7 to your computer and use it in GitHub Desktop.
Save tprelog/7988dc6b196775f33929beb19f0090d7 to your computer and use it in GitHub Desktop.
Use docker-compose on TrueNAS SCALE 22.12 (Bluefin) without Kubernetes
#!/usr/bin/env bash
#
# Enable docker and docker-compose on TrueNAS SCALE (no Kubernetes)
#
# This script is a hack! Use it at your own risk!!
# Using this script to enable Docker is NOT SUPPORTED by ix-systems!
# You CANNOT use SCALE Apps while using this script!
#
# 1 Create a dedicated Docker dataset in one of your zpools
# 2 Save this script somewhere else on your zpool, not in the Docker dataset
# 3 Edit line 20 of the script, set a path to the Docker dataset you created
# 4 You can now start Docker by running the script from the SCALE console
#
# For these changes to persist after SCALE reboots and upgrades, run the script at start-up
# Schedule this script to run via System Settings -> Advanced -> Init/Shutdown Scripts
# Click Add -> Type: Script and choose this script -> When: choose to run as Post Init
## Set a path to your docker dataset
docker_dataset='/mnt/tank/docker'
## HEREDOC: docker/daemon.json
read -r -d '' JSON << END_JSON
{
"data-root": "${docker_dataset}",
"storage-driver": "overlay2",
"exec-opts": [
"native.cgroupdriver=cgroupfs"
]
}
END_JSON
## path to docker daemon file
docker_daemon='/etc/docker/daemon.json'
if [ ${EUID} -ne 0 ]; then
echo "Please run this script as root or using sudo"
elif [ "$(systemctl is-enabled k3s)" == "enabled" ]; then
echo "You can not use this script while k3s is enabled"
elif [ "$(systemctl is-active k3s)" == "active" ]; then
echo "You can not use this script while k3s is active"
elif ! which docker &> /dev/null; then
echo "Docker executable not found"
elif ! chmod +x /usr/bin/docker-compose &> /dev/null; then
echo "Failed to make docker-compose executable"
elif ! install -d -m 755 -- /etc/docker &> /dev/null; then
echo "Failed to install directory: /etc/docker"
elif ! zfs list "${docker_dataset}" &> /dev/null; then
echo "Dataset not found: ${docker_dataset}"
else
echo "Checking file: ${docker_daemon}"
if test "${JSON}" != "$(cat ${docker_daemon} 2> /dev/null)"; then
echo "Updating file: ${docker_daemon}"
jq -n "${JSON}" > ${docker_daemon}
if [ "$(systemctl is-active docker)" == "active" ]; then
echo "Restarting Docker"
systemctl restart docker
elif [ "$(systemctl is-enabled docker)" != "enabled" ]; then
echo "Enable and starting Docker"
systemctl enable --now docker
fi
fi
fi
@AlmeidaXIX
Copy link

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

You're a genius! Thank you!
Do you mind explaining why this works? Noob here.

Sure.

Take a you look in /etc/systemd/system/docker.service.d/override.conf and you'll see this EnvironmentFile=/etc/docker.env

Docker service will fail to start if the file is missing with the following error docker.service: Failed to load environment files: No such file or directory

You can see the log with command journalctl -u docker.service

Thank you.

When I checked /etc/systemd/system/docker.service.d/override.conf the only thing inside was LimitCORE=1. The journalclt info I confess that it's hard for me to understand much but I did see docker.service: Failed to load environment files: No such file or directory just didn't have a clue how to solve that. I guess I learned something new today.

I have a similar though more simple script to make this all work and I'm also thinking about adding the touch /etc/docker.env bit to it.

@mforce
Copy link

mforce commented Feb 25, 2023

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

You're a genius! Thank you!
Do you mind explaining why this works? Noob here.

Sure.
Take a you look in /etc/systemd/system/docker.service.d/override.conf and you'll see this EnvironmentFile=/etc/docker.env
Docker service will fail to start if the file is missing with the following error docker.service: Failed to load environment files: No such file or directory
You can see the log with command journalctl -u docker.service

Thank you.

When I checked /etc/systemd/system/docker.service.d/override.conf the only thing inside was LimitCORE=1. The journalclt info I confess that it's hard for me to understand much but I did see docker.service: Failed to load environment files: No such file or directory just didn't have a clue how to solve that. I guess I learned something new today.

I have a similar though more simple script to make this all work and I'm also thinking about adding the touch /etc/docker.env bit to it.

Did this work for you?

@scotrod
Copy link

scotrod commented Apr 4, 2023

I'm getting an error when trying to run a container that uses a local volume. I've set-up the script with /mnt/spins3/appdata/docker for docker_dataset and I can see that all the docker files are there.

`root@truenas# docker volume create portainer_data portainer_data

root@truenas[~]# docker volume ls DRIVER VOLUME NAME local portainer_data

root@truenas[~]# docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest docker: Error response from daemon: invalid volume specification: 'portainer_data:/data': portainer_data path not allowed to be mounted. See 'docker run --help'.`

Anyone else getting this issue?

Did you managed to get around this? A year later I cannot install Portainer agent on the TrueNAS Scale host.

@Jip-Hop
Copy link

Jip-Hop commented Apr 4, 2023

I used to use portainer on TrueNAS SCALE before I switched to jailmaker. Worked for me when I used bind mounts instead of docker volumes. Just create a directory where you want to keep your portainer data and mount it inside the portainer container instead of using the volume. No need to create a volume in this case either. But beware this method of running docker will stop working in October.

@scotrod
Copy link

scotrod commented Apr 4, 2023

I used to use portainer on TrueNAS SCALE before I switched to jailmaker. Worked for me when I used bind mounts instead of docker volumes. Just create a directory where you want to keep your portainer data and mount it inside the portainer container instead of using the volume. No need to create a volume in this case either. But beware this method of running docker will stop working in October.

Thanks for replying. Having read all kinds of discussions about docker implementation on TrueNAS Scale, I'm thinking of moving out my Docker containers entirely from it and using it for what it was intended to do in the first place - storage.

@scepterus
Copy link

@scotrod
Copy link

scotrod commented Apr 11, 2023

@scotrod Did you try my guide on the subject? https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882

Thank you! Definetely saving this for future references. That said, I really don't like TrueNAS' integrated Docker solution. I will look up if I can hook up everything via NFS shares, so the containers access the data via the network. It's going to hit the performance for sure, but for my purposes I think it should be fine.

@Ixian
Copy link

Ixian commented Apr 11, 2023

@scotrod Did you try my guide on the subject? https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882

Thank you! Definetely saving this for future references. That said, I really don't like TrueNAS' integrated Docker solution. I will look up if I can hook up everything via NFS shares, so the containers access the data via the network. It's going to hit the performance for sure, but for my purposes I think it should be fine.

We've all been down this same path for the same reasons and you should really look at Jip-Hops Jailmaker script.

@scepterus
Copy link

Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.

@scotrod
Copy link

scotrod commented Apr 11, 2023

Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.

They are removing Docker now!?

@scepterus
Copy link

Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.

They are removing Docker now!?

Yeah, it was discussed here, in 23.10 they will remove Docker from TrueNAS, but my script should bring it back on the next reboot right after the update. (Still not tested, of course…) They'd give us more of a headache if they removed apt, but that would still be solvable. This will always be a Linux server with a network connection, so almost everything is possible.

@Jip-Hop
Copy link

Jip-Hop commented Sep 3, 2023

IMO the best way to run docker, if you don't want to run it inside a 'jail' with jailmaker, is not to enable the package manager and install docker. Instead, download and run the binaries. You can set it up in a portable way so your docker install will be preserved on upgrades. And you minimize the interaction with the TrueNAS host (not enabling and installing from the package manager with possibly unintended side effects). I think this way it should even be possible to get it to run alongside TrueNAS Apps. If someone would create a script for this I recommend to take a look at what I'm doing with ubernerd. It uses nerdctl (which is a mostly docker compatible tool to run containers) which has the additional benefit of being able to run 'jails'. This docker install script could be inspiring too. It can install the latest docker binaries, but it's missing the part to make the installation contained to a single directory (to keep it preserved after upgrade and limit the amount of interaction with the host rootfs).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment