Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@tprelog
Last active November 23, 2023 01:58
Show Gist options
  • Star 40 You must be signed in to star a gist
  • Fork 7 You must be signed in to fork a gist
  • Save tprelog/7988dc6b196775f33929beb19f0090d7 to your computer and use it in GitHub Desktop.
Save tprelog/7988dc6b196775f33929beb19f0090d7 to your computer and use it in GitHub Desktop.
Use docker-compose on TrueNAS SCALE 22.12 (Bluefin) without Kubernetes
#!/usr/bin/env bash
#
# Enable docker and docker-compose on TrueNAS SCALE (no Kubernetes)
#
# This script is a hack! Use it at your own risk!!
# Using this script to enable Docker is NOT SUPPORTED by ix-systems!
# You CANNOT use SCALE Apps while using this script!
#
# 1 Create a dedicated Docker dataset in one of your zpools
# 2 Save this script somewhere else on your zpool, not in the Docker dataset
# 3 Edit line 20 of the script, set a path to the Docker dataset you created
# 4 You can now start Docker by running the script from the SCALE console
#
# For these changes to persist after SCALE reboots and upgrades, run the script at start-up
# Schedule this script to run via System Settings -> Advanced -> Init/Shutdown Scripts
# Click Add -> Type: Script and choose this script -> When: choose to run as Post Init
## Set a path to your docker dataset
docker_dataset='/mnt/tank/docker'
## HEREDOC: docker/daemon.json
read -r -d '' JSON << END_JSON
{
"data-root": "${docker_dataset}",
"storage-driver": "overlay2",
"exec-opts": [
"native.cgroupdriver=cgroupfs"
]
}
END_JSON
## path to docker daemon file
docker_daemon='/etc/docker/daemon.json'
if [ ${EUID} -ne 0 ]; then
echo "Please run this script as root or using sudo"
elif [ "$(systemctl is-enabled k3s)" == "enabled" ]; then
echo "You can not use this script while k3s is enabled"
elif [ "$(systemctl is-active k3s)" == "active" ]; then
echo "You can not use this script while k3s is active"
elif ! which docker &> /dev/null; then
echo "Docker executable not found"
elif ! chmod +x /usr/bin/docker-compose &> /dev/null; then
echo "Failed to make docker-compose executable"
elif ! install -d -m 755 -- /etc/docker &> /dev/null; then
echo "Failed to install directory: /etc/docker"
elif ! zfs list "${docker_dataset}" &> /dev/null; then
echo "Dataset not found: ${docker_dataset}"
else
echo "Checking file: ${docker_daemon}"
if test "${JSON}" != "$(cat ${docker_daemon} 2> /dev/null)"; then
echo "Updating file: ${docker_daemon}"
jq -n "${JSON}" > ${docker_daemon}
if [ "$(systemctl is-active docker)" == "active" ]; then
echo "Restarting Docker"
systemctl restart docker
elif [ "$(systemctl is-enabled docker)" != "enabled" ]; then
echo "Enable and starting Docker"
systemctl enable --now docker
fi
fi
fi
@Ornias1993
Copy link

Ornias1993 commented Jan 16, 2023

First off:
This is not "running natively" it runs in something that is in between a container and chroot.
Don't blindly believe things that are not actually proven/tested ;-)

Second:
You never can update the kernel (initramfs) in a container, not in this one and not in a docker container either.
No, iX is never going to enable you to run kernelmodifications lol.

@injeolmibingsoo
Copy link

All that said have you already tried it and seen it fail? The Bluefin kernel (i.e. the latest production release of SCALE) added support for newer Intel processors as well as newer power efficiency modes. You may not need to do this step in the first place.

Thanks for responding! Yeah, it didn't work out-of-the-box for me. Oh well, despite what people are saying about the prowess Intel QuickSync, I'm finding that the P2000 is transcodes at a much higher frame rate than my i3-9100, so I'm switching to using nvenc instead and have stopped playing around with IQSV.

First off: This is not "running natively" it runs in something that is in between a container and chroot. Don't blindly believe things that are not actually proven/tested ;-)

Second: You never can update the kernel (initramfs) in a container, not in this one and not in a docker container either. No, iX is never going to enable you to run kernelmodifications lol.

Thanks for responding! I see! Very useful to know. Looks like we'll be at the mercy of TrueNAS for a few things then!

@gghub77
Copy link

gghub77 commented Feb 22, 2023

Just updated to Bluefin. I deleted my docker dataset and set up the enable-docker script (modified to add nvidia runtime etc), then performed the update. Upon reboot I manually launched portainer, then I brought up each one of my stacks one by one. Nice byproduct is it forced me to grab all the latest images. Feels great to no longer have all of those ghost snapshots with the overlay2 driver. Things feel a bit more solid/snappier over all. I've also upgraded from Jellyfin to Emby and it picked up my 1050ti right away. Emby is running so fast and smooth it's unbelievable, feels better than Netflix Disney+ etc. Seafile, NGINX manager, ARRs, guacamole, emby, joplin, transmission_ovpn are running rock solid and fast also (off an old SATA SSD). Oh and 2 VMs for VS code-server (with hack to access the M$ extension store). All on a 10 year old CPU with 20gb of mismatched ram. I've looked at OMV, Unraid, Synology, Xpenology etc etc and this setup is 1000% the best NAS + Docker + Media Server solution period. They can completely butcher and cripple Cobia and try to block docker (they can't) blah blah, and I can die happy with this Bluefin setup.

@Ornias1993
Copy link

To be clear: The performance things you see is not likely related to the docker setup you did.
Also: Cobia is not going to be "butchered" or "crippled", they just removed something they are not using anymore for the kubernetes stack.

Just because that might or might not affect you for hacks like these, doesn't mean it was intended for fuck with people using docker.
That being said: With docker removed, it should be safer now to run docker by installing it via apt-get :)

Why? Because iX isn't using it anymore, so less chance of potential conflicts!

@mforce
Copy link

mforce commented Feb 23, 2023

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

@ash0x1b
Copy link

ash0x1b commented Feb 23, 2023

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

@gghub77
Copy link

gghub77 commented Feb 23, 2023

People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.

@Ornias1993
Copy link

People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.

There is nothing new being butchered. Bluefin hostPath validation existed since bluefin launch.

@gghub77
Copy link

gghub77 commented Feb 24, 2023

People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.

There is nothing new being butchered.
Bluefin hostPath validation existed since bluefin launch.

Ok let me be more clear: a hostpath volume under a shared dataset would pass validation under 22.12.0, this “bug” was “fixed” in 22.12.1. This breaks 99% of people’s plex/emby/jellyfin configurations.

@Ornias1993
Copy link

People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.

There is nothing new being butchered.
Bluefin hostPath validation existed since bluefin launch.

Ok let me be more clear: a hostpath volume under a shared dataset would pass validation under 22.12.0, this “bug” was “fixed” in 22.12.1. This breaks 99% of people’s plex/emby/jellyfin configurations.

This was warned against by me on SO MANY places, it was in fact a bug and never intended to work. It was fixed a week or so after release of Bluefin even.

@gghub77
Copy link

gghub77 commented Feb 24, 2023

Ok i’ll keep my buggy 22.12.0 and warn others to do the same, thanks.

@Ornias1993
Copy link

Ok i’ll keep my buggy 22.12.0 and warn others to do the same, thanks.

It's one of the reasons I build NFS support into truecharts. Works flawlessly for media libraries for most of our users.

@Jip-Hop
Copy link

Jip-Hop commented Feb 24, 2023

Docker inside a jail made with jailmaker survived the update to 22.12.1 without any issues. I've also never ran into host path validation. I have nextcloud use the same local path I use for my SMB shares.

@Ixian
Copy link

Ixian commented Feb 24, 2023

People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.

There is nothing new being butchered.
Bluefin hostPath validation existed since bluefin launch.

Ok let me be more clear: a hostpath volume under a shared dataset would pass validation under 22.12.0, this “bug” was “fixed” in 22.12.1. This breaks 99% of people’s plex/emby/jellyfin configurations.

Hostpath validation shouldn’t matter for folks using this or other docker workarounds. 22.12.1 works fine with Jip-Hops Jailmaker. I personally use Plex, Emby, and several arrs with this setup.

@AlmeidaXIX
Copy link

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

You're a genius! Thank you!

Do you mind explaining why this works? Noob here.

@ash0x1b
Copy link

ash0x1b commented Feb 24, 2023

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

You're a genius! Thank you!

Do you mind explaining why this works? Noob here.

Sure.

Take a you look in /etc/systemd/system/docker.service.d/override.conf and you'll see this
EnvironmentFile=/etc/docker.env

Docker service will fail to start if the file is missing with the following error
docker.service: Failed to load environment files: No such file or directory

You can see the log with command journalctl -u docker.service

@andyschmid
Copy link

andyschmid commented Feb 24, 2023

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

Do the contents of /etc get reset on the next upgrade?

This should probably be added to the enable-docker.sh script:

touch /etc/docker.env

@ash0x1b
Copy link

ash0x1b commented Feb 24, 2023

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

Do the contents of /etc get reset on the next upgrade?

This should probably be added to the enable-docker.sh script:

touch /etc/docker.env

I believe this is the case.

@AlmeidaXIX
Copy link

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

You're a genius! Thank you!
Do you mind explaining why this works? Noob here.

Sure.

Take a you look in /etc/systemd/system/docker.service.d/override.conf and you'll see this EnvironmentFile=/etc/docker.env

Docker service will fail to start if the file is missing with the following error docker.service: Failed to load environment files: No such file or directory

You can see the log with command journalctl -u docker.service

Thank you.

When I checked /etc/systemd/system/docker.service.d/override.conf the only thing inside was LimitCORE=1. The journalclt info I confess that it's hard for me to understand much but I did see docker.service: Failed to load environment files: No such file or directory just didn't have a clue how to solve that. I guess I learned something new today.

I have a similar though more simple script to make this all work and I'm also thinking about adding the touch /etc/docker.env bit to it.

@mforce
Copy link

mforce commented Feb 25, 2023

It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?

Create empty /etc/docker.env file.

You're a genius! Thank you!
Do you mind explaining why this works? Noob here.

Sure.
Take a you look in /etc/systemd/system/docker.service.d/override.conf and you'll see this EnvironmentFile=/etc/docker.env
Docker service will fail to start if the file is missing with the following error docker.service: Failed to load environment files: No such file or directory
You can see the log with command journalctl -u docker.service

Thank you.

When I checked /etc/systemd/system/docker.service.d/override.conf the only thing inside was LimitCORE=1. The journalclt info I confess that it's hard for me to understand much but I did see docker.service: Failed to load environment files: No such file or directory just didn't have a clue how to solve that. I guess I learned something new today.

I have a similar though more simple script to make this all work and I'm also thinking about adding the touch /etc/docker.env bit to it.

Did this work for you?

@scotrod
Copy link

scotrod commented Apr 4, 2023

I'm getting an error when trying to run a container that uses a local volume. I've set-up the script with /mnt/spins3/appdata/docker for docker_dataset and I can see that all the docker files are there.

`root@truenas# docker volume create portainer_data portainer_data

root@truenas[~]# docker volume ls DRIVER VOLUME NAME local portainer_data

root@truenas[~]# docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest docker: Error response from daemon: invalid volume specification: 'portainer_data:/data': portainer_data path not allowed to be mounted. See 'docker run --help'.`

Anyone else getting this issue?

Did you managed to get around this? A year later I cannot install Portainer agent on the TrueNAS Scale host.

@Jip-Hop
Copy link

Jip-Hop commented Apr 4, 2023

I used to use portainer on TrueNAS SCALE before I switched to jailmaker. Worked for me when I used bind mounts instead of docker volumes. Just create a directory where you want to keep your portainer data and mount it inside the portainer container instead of using the volume. No need to create a volume in this case either. But beware this method of running docker will stop working in October.

@scotrod
Copy link

scotrod commented Apr 4, 2023

I used to use portainer on TrueNAS SCALE before I switched to jailmaker. Worked for me when I used bind mounts instead of docker volumes. Just create a directory where you want to keep your portainer data and mount it inside the portainer container instead of using the volume. No need to create a volume in this case either. But beware this method of running docker will stop working in October.

Thanks for replying. Having read all kinds of discussions about docker implementation on TrueNAS Scale, I'm thinking of moving out my Docker containers entirely from it and using it for what it was intended to do in the first place - storage.

@scepterus
Copy link

@scotrod
Copy link

scotrod commented Apr 11, 2023

@scotrod Did you try my guide on the subject? https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882

Thank you! Definetely saving this for future references. That said, I really don't like TrueNAS' integrated Docker solution. I will look up if I can hook up everything via NFS shares, so the containers access the data via the network. It's going to hit the performance for sure, but for my purposes I think it should be fine.

@Ixian
Copy link

Ixian commented Apr 11, 2023

@scotrod Did you try my guide on the subject? https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882

Thank you! Definetely saving this for future references. That said, I really don't like TrueNAS' integrated Docker solution. I will look up if I can hook up everything via NFS shares, so the containers access the data via the network. It's going to hit the performance for sure, but for my purposes I think it should be fine.

We've all been down this same path for the same reasons and you should really look at Jip-Hops Jailmaker script.

@scepterus
Copy link

Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.

@scotrod
Copy link

scotrod commented Apr 11, 2023

Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.

They are removing Docker now!?

@scepterus
Copy link

Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.

They are removing Docker now!?

Yeah, it was discussed here, in 23.10 they will remove Docker from TrueNAS, but my script should bring it back on the next reboot right after the update. (Still not tested, of course…) They'd give us more of a headache if they removed apt, but that would still be solvable. This will always be a Linux server with a network connection, so almost everything is possible.

@Jip-Hop
Copy link

Jip-Hop commented Sep 3, 2023

IMO the best way to run docker, if you don't want to run it inside a 'jail' with jailmaker, is not to enable the package manager and install docker. Instead, download and run the binaries. You can set it up in a portable way so your docker install will be preserved on upgrades. And you minimize the interaction with the TrueNAS host (not enabling and installing from the package manager with possibly unintended side effects). I think this way it should even be possible to get it to run alongside TrueNAS Apps. If someone would create a script for this I recommend to take a look at what I'm doing with ubernerd. It uses nerdctl (which is a mostly docker compatible tool to run containers) which has the additional benefit of being able to run 'jails'. This docker install script could be inspiring too. It can install the latest docker binaries, but it's missing the part to make the installation contained to a single directory (to keep it preserved after upgrade and limit the amount of interaction with the host rootfs).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment