Skip to content

Instantly share code, notes, and snippets.

@Jip-Hop
Last active December 30, 2023 21:02
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d to your computer and use it in GitHub Desktop.
Save Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d to your computer and use it in GitHub Desktop.
Persistent Debian 'jail' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts. Without modifying the host OS at all thanks to systemd-nspawn!
@Talung
Copy link

Talung commented Feb 28, 2023

Jailmaker should not interfere with the IX kubernetes Apps :)

Excellent news. Best of both worlds. Hopefully they will leave Jailmaker alone :) I will try transition to this soon to add to your testing data. Thanks :)

@qudiqudi
Copy link

@Talung you can even use both in conjunction. I have every service inside jails reverse proxied through the Truecharts Traerfik app via the "external service" app. Though, I plan to ditch k3s altogether, but it works very well.

@Talung
Copy link

Talung commented Feb 28, 2023

Hi there,
Noob question with binding mounts. I have been reading through the previous comments to find out about binding some folders to the new machine. I have docker setup and its working with the "hello World" thing.

My jail is called "dockerjail" otherwise pretty much followed all the instructions including docker install recommendations.

With systemd-nspawn which I am assuming you run in a truenas shell, I am unsure of the -D option. I keep getting doesn't look like an OS root directory (os-release file is missing). Refusing. Should this be pointing to the actual jail rootfs folder. ie /mnt/pond/jailmaker/jails/dockerjail/rootfs/

Also, once this is mounted is it something that needs to be done on every subsequent reboot?

systemd-nspawn --capability=all -b -D "????/mnt/pond/jailmaker/jails/dockerjail/rootfs????" --system-call-filter='add_key keyctl bpf' --bind=/mnt/pond/appdata:/mnt/lake/media:/mnt/lake/cloud

Thanks

@Jip-Hop
Copy link
Author

Jip-Hop commented Feb 28, 2023

Hi. You should not need to run systemd-nspawn yourself. The jlmkr.py script takes care of that. You should create a jail with jlmkr.py create. Then also specify the bind mounts when the wizard asks you to. Then run jlmkr.py start dockerjail on each reboot. Hope that makes sense.

@Talung
Copy link

Talung commented Feb 28, 2023

ah ok. Must have missed that part of the setup. Or maybe I skipped it just to get it working for testing. Is there a way to update it now, or should I just make a new one. Thing is, I would probably run into situations where I would want to add more mounts in at a later stage.

I have everything else setup with no issues. So script and documentation was fine in that respect. thanks

EDIT. went through install again and saw what was happening. It is way easier than I thought, just need to modify the config file and add: systemd_nspawn_user_args=--bind='/mnt/pond/appdata/' for the various folders or --bind-ro=

Added this part for anybody else that runs into the same issue. :)

@Jip-Hop
Copy link
Author

Jip-Hop commented Feb 28, 2023

Exactly! Glad you got it solved :)

@Talung
Copy link

Talung commented Mar 1, 2023

Quick update for you. Everything is working great. Been able to move all the native docker stuff into the jail without really needing to change much. Even the "internal networking" I setup went through just fine ie. in portainer "networks". Haven't need to use Traefek or anything like that, works as it did before. I haven't tried any nvidia passthrough yet and have noticed that you have a recent update on that, so will look at that at a later stage.

I did notice that there is no daemon.json file in the container and am assuming I just need to add in my copy from "enable-docker.sh" in there. I haven't done that yet.

Also, I have noticed all the images are stored with the jail, which means if I recreate the jail I would need to re-download all the images again. All my data is outside of it, so not really an issue tbh, except during development stages.

Just wanted to add my thanks to you for doing this. 👍

@Jip-Hop
Copy link
Author

Jip-Hop commented Mar 1, 2023

Good to hear! And convenient that you have a nvidia GPU :) Indeed still in the process of getting that working properly.

I don't know what you have in the daemon.json. I don't think you need to do anything with it when running docker inside the jail. I know I haven't touched that file myself since switching.

I think you can also bind mount the directory where docker stores the images to a directory on the host. That way when you recreate a jail all the images are still downloaded. But since you also need to reinstall docker in this fresh jail perhaps that won't work the way I think/hope it does.

@Talung
Copy link

Talung commented Mar 1, 2023

Essentialy my last daemon.json was

{
  "data-root": "/mnt/pond/dockerset",
  "storage-driver": "overlay2",
  "exec-opts": [
    "native.cgroupdriver=cgroupfs"
  ],
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
      }
    }
}

But, as you mention with the bind mounts, if I add that dockerset, then it would make the jail extremely thin. Meaning the only thing I would need to do when creating it is a small script to install docker and setup the various files. That should be easy enough. Will be trying that soon with the nvidia changes and seeing if I can get that working. Got an old GTX 1070 in there.

Btw, I take it the quick/correct way to remove a jail is just to remove the folder after it is stopped?

EDIT: Interestingly enough this is what TrueNAS has changed the daemon.json file to:

{
	"data-root": "/mnt/pond/ix-applications/docker", 
        "exec-opts": ["native.cgroupdriver=cgroupfs"], 
	"iptables": false, 
	"bridge": "none", 
	"storage-driver": "overlay2"
}

No mention of nvidia in there.

@Jip-Hop
Copy link
Author

Jip-Hop commented Mar 1, 2023

Yes, you can just stop the jail and then remove the corresponding directory from the jails directory. I'd add a sleep of 1 second in between those 2 commands. Noticed the rm would fail because some files were not unmounted yet immediately after stopping the jail with machinectl stop.

@ZestyChicken
Copy link

2 questions for you guys - can you pull your TrueNAS Scale users into the jail and work with consistent users between the jail and Scale host?

For example, I have an apps user/group in Scale which has ownership of my media files (set up for TrueCharts). It looks like there is a flag -u that may do that but the language is confusing.

Also, if I wanted to expose port 80 & 443 for traefik, I know I can create an alias for my primary ip address but would this IP need to be attached to the jail via host or bridge networking? I set up a bridge but was unable to deploy traefik presumedly because it was complicting with the TrueNAS GUI.

@qudiqudi
Copy link

2 questions for you guys - can you pull your TrueNAS Scale users into the jail and work with consistent users between the jail and Scale host?

You have to translate your TN users into unique Users/Groups inside the jail. You can map them to the same UID/GID but I would advise against it to prevent permission issues.

Also, if I wanted to expose port 80 & 443 for traefik, I know I can create an alias for my primary ip address but would this IP need to be attached to the jail via host or bridge networking?

There is a networking section of the readme.md, read it carefully, as it gives concise yet direct instructions on how to do that. If you want to reverse proxy via traefik, your jail needs an IP address from your DHCP server. This is possible via a network bridge on the TN host.

@LimonChoms
Copy link

can amdgpu(like 5700xt,5600g...) run on jail?

@Jip-Hop
Copy link
Author

Jip-Hop commented Apr 28, 2023

I don't think so. As far as I know TrueNAS SCALE doesn't include drivers for AMD GPUs. But I could be wrong!

@LimonChoms
Copy link

I don't think so. As far as I know TrueNAS SCALE doesn't include drivers for AMD GPUs. But I could be wrong!

No,it has amdgpu drivers,it works on my jellyfin docker for hw-encoder very well.It's just that I don't know how to make it work in jail.

@Jip-Hop
Copy link
Author

Jip-Hop commented Apr 28, 2023

In that case I think it should be possible to use the GPU in the jail as well! But I don't know how and I don't have the time and hardware (AMD GPU) to figure this out. Perhaps you could draw inspiration form the nvidia GPU passhtrough issue?

@nguypaul
Copy link

nguypaul commented May 7, 2023

I just wanted to say thanks for this script. I finally migrated my 15 apps from Truecharts over to a jail. I'm using portainer and docker compose to manage everything and it is working nicely. Nvidia GPU passthrough took some effort but I finally got that working with Jellyfin and Nextcloud. Traefik is also working well as my reverse proxy.

I disabled k3s and now my CPU usage doesn't hover around 15% anymore. Very Happy so far.

@Jip-Hop
Copy link
Author

Jip-Hop commented May 7, 2023

Glad to hear that @nguypaul 😄👍

@worksmarter
Copy link

Really excited about this but I ran into a snag on my system. When prompted by the script I added the --network-bridge=br1 --resolv-conf=bind-host and the jail fails to boot - comes up with an error... My need is to setup portainer in the jail. Docker-compose installed without a problem, but I can't figure out how to make the network bridge work so when I install Portainer I can reach it.

Right now I am at the stage where the jail is installed and running; so is docker-compose, but I don't know exactly how I add the "--network-bridge=br1 --resolv-conf=bind-host" to the config file to bring up the bridge. Sorry, I am sure this is trivial, but I am not having any luck adding it in the "jlmkr.py" file. Is it possible you can assist me? I looked at the config file on my first attempt and failed trying to add that tidbit of code to it. It never brought up a bridge for portainer.

@nguypaul
Copy link

nguypaul commented May 8, 2023

When prompted by the script I added the --network-bridge=br1 --resolv-conf=bind-host and the jail fails to boot - comes up with an error...

I'm not sure if my scenario is the same as yours, but when creating the jail the only customization I add is the --bind='/mnt/pathtodata' to mount my shares.

The jail automatically has full access to the host network so I can reach portainer just by using the ip address of my host and the port configured for portainer. I can share my portainer compose file if you think that will be helpful.

@worksmarter
Copy link

Okay, hey thanks. Yes, I too set the flags to bind my mount points. (and the flags for a network bridge). I just can't get networking going properly and now I basically have no network connectivity. Following the instructions and setting up a bridge as per instructions is not working for me. Right now I am trying to get the networking running again, for I can't even ping google! Frustrating - I know this can't be so hard...

@Jip-Hop
Copy link
Author

Jip-Hop commented May 8, 2023

I assume you have been patient for the jail to be assigned an IP addresses via DHCP? For me (and others reported too) that it can take a while, up to 30 seconds I guess. If you leave the jail with bridge up for a bit and network still doesn't work then it's a different problem. But as @nguypaul mentions, a bridge is not required to use portainer in the jail. So unless you have a specific reason to go 'the hard way' I'd just use it with the default networking.

@worksmarter
Copy link

Thank you Sir and thank you for the wonderful script. I finally got everything talking to one another (Internet, apps). I am obviously doing something wrong with portainer. I never had an issue using the docker-compose truechart and pointing it at a docker-compose.yml file. I can't see how I would use that approach here. I am at a complete loss, which saddens me as I am sure this system seems to work swimmingly for everyone else. And sadly how to install and access portainer is not within the scope of this gist.

@worksmarter
Copy link

worksmarter commented May 8, 2023

Should there be an address in this? (Output from ./jlmkr.py list)
MACHINE CLASS SERVICE OS VERSION ADDRESSES
docker-compose container systemd-nspawn debian 11 -

@nguypaul
Copy link

nguypaul commented May 8, 2023

@worksmarter share your portainer compose file.

@worksmarter
Copy link

There is where it appears to be:
Run this with docker compose up -d and it comes right up as everyone said it would. I feel so stupid but it had to be the ports.

`version: '3'

services:
portainer:
image: portainer/portainer-ce:latest
container_name: portainer
restart: unless-stopped
security_opt:
- no-new-privileges:true
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./portainer-data:/data
ports:
- 9000:9000`

Wish I had come to that conclusion before posting. AND hasltleing around with bridging the networks, what a hassle that was. Thanks for your patience everyone. Great script JipHop!

@tprelog
Copy link

tprelog commented Jun 17, 2023

I just heard read the news 😞

Update: 15-6-2023. Looks like the systemd-container package has been removed since 22.12.3...

I haven't used your jailmaker script, but this still sucks.

@Talung
Copy link

Talung commented Jun 17, 2023

I just heard read the news 😞

Update: 15-6-2023. Looks like the systemd-container package has been removed since 22.12.3...

I haven't used your jailmaker script, but this still sucks.

wow... Good thing I am busy converting back to Proxmox for vm/container needs and will be running TrueNAS as a VM in that. For the little bit of VM's i was doing it just became too much hassle fighting Truenas imho. Now it is being relegated to ONLY a file server.

@aardvarkl
Copy link

Now that would be annoying as jailmakr works perfectly, doesn't have the crappy IX routing that IX has inflicted on K3S and is simpler.

I might not uprgade for a while

@Ixian
Copy link

Ixian commented Jun 18, 2023

I mean, by now we should all figure that Ix Systems is dead-set on SCALE being used their way or the highway, and non-commercial users are simply a pool of free beta testers for them, so if you hate how they handle apps - and it is dumb, no two ways about it - then best case use it as a simple, dumb fileserver, or leave it. Time to move on from the AIO idea.

@nader-eloshaiker
Copy link

You want to know what really doesn’t make sense about this, they are using K3S as the platform to scale your app server requirements . I don’t see many enterprises using that for their app servers. Also note that it is called TrueNAS and not TrueApp, they are shoe horning application enterprise server requirements into an enterprise file server.

@Ixian
Copy link

Ixian commented Jun 22, 2023

It only doesn't make sense because we're missing context. Likely it is as simple as: Ix Systems, being a for-profit business in a tough market (there's no money in storage hardware sales except at scale, so you have to differentiate with software these days), is trying to better define the space they are in so they can expand, and based on (presumably) their own internal market research concluded this is the way to go. Will it work or not? Does it make sense outside of Ix Systems? Who knows? It doesn't matter.

Folks like us are their field beta testers, more or less. Heck they aren't even subtle about that, their Enterprise marketing highlights how many Truenas (and SCALE) users there are, etc. We get a decent fileserver and whatever else they want to throw on top of it but let's not be under any illusion that they really care how any of us want to use the system. If we aren't paying for it then we are the product simple as that.

@qudiqudi
Copy link

I'd like to give a more positive notion into this discussion. Yes, iX, at the moment, is presumably trying to contain their user base inside their newly build ecosystem, which happen to be a catalogue of k8s apps. Nobody says that this is not gonna change in the future. It is just what they decided to do and it's their product, so that's the way it is. BUT, they have a over a decade long track record of changing their product according to what the community wants and needs. See i. e. the change from FreeBSD warden jails to iocage. Every time a "breaking" change was introduced, it lead to some kind of "backlash" in the Forums, later also on reddit. But over time everybody agreed that it was for the better. And they still have a huge list of stuff users want and I'm convinced they are looking into it. Though, at the end of the day we have the great pleasure of using their product for free and they still need to pay staff. So I'm pledging for this: we vote the according JIRA-Tickets with the features we want to see and we further voice our constructive criticism here, in the forums, on Discord and on (reddit maybe not anymore).

@Talung
Copy link

Talung commented Jun 23, 2023

For me, honestly it just became way too much effort to keep on "Fixing" and finding work-arounds on their updates. I want to use Docker, I want to use VM's, I want to use ZFS and I want to use shares.

TrueNAS can no longer fulfill these needs for me without constant fighting, so I gave up. I have reinstalled Proxmox on my box, installed Docker Natively, and setup my ZFS storage. I just need to add in Shares via SMB and NFS with config files. Then a cronjob to do snapshots and replication. Is it a pain that it is not gui, sure, but end of the day is that it works and I have no worries that a single point update will kill everything.

I am done with TrueNAS... If I want it, I will go to TrueNAS Core because Scale is a wast of time for Home User.

@Codelica
Copy link

Codelica commented Jun 23, 2023

FWIW, there is another option if you're just looking for a clean Docker/Compose workspace on Scale and have a decent grasp on things. Basically I use Scale's "Launch Docker Image" to create a pod running Docker (technically a Docker-in-Docker image ala https://hub.docker.com/_/docker or similar -- I'm running one based on DIND some nVidia additions).

At that point there is a "safe" and isolated install of Docker running which can be used -- as Docker on the Scale host could be removed at any point. Using that Docker pod I just start up a container that gives me access and all the tools I would want on a "Docker host" (sshd, compose, code-server, cli tools, etc) which I then use to get access to my "Docker host".

Anyway, that's been working well since the first release of Scale with roughly 30 containers running. "Launch Docker Image" allows the base Docker install to have host networking and even nVidia passthrough (in addition to using it with official IX apps), so it's pretty flexible overall. Obviously in the App UI in Scale I just see my "Docker" app which is running the show. ;)

Not exactly the same as just an isolated host workspace, but quite flexible in the end. Just tossing it out there for those who might find it as an interesting option.

@aardvarkl
Copy link

@Codelica I understand that docker is going away soon - so your solution (if I am correct) will no longer work

@Codelica
Copy link

@aardvarkl that would be the Docker installation that is on the TrueNAS Scale host itself, which has always been in jeopardy. My solution was to run my own Docker install separately as a pod on Scale and use that instance for services. (It's similar to how TrueCharts does their docker-compose chart) So removal of Docker on the Scale host won't affect it. The only real danger would be if IX decides that people can't launch any of their own custom pods and must choose from a catalog. Could happen, but seems doubtful to me.

@aardvarkl
Copy link

@Codelica would you try something for me. In one of your containers that has traceroute, just run a traceroute to a device on your LAN that isn't your router / gateway and post the result please?

@Codelica
Copy link

@aardvarkl sure, although that will depend what networking driver the container uses (host, bridge, macvlan,etc).

But like this is in a host mode container pinging a LAN host:

SeaShell Docker 2023-06-26 08-49-30

and this is in a bridge mode container pinging a LAN host:

email-tr

@aardvarkl
Copy link

aardvarkl commented Jun 26, 2023 via email

@Codelica
Copy link

@aardvarkl it's not. I use a class B at home (10.10.x.x). 10.10.100.1 is a desktop in my bedroom. My router is 10.10.0.1.

@aardvarkl
Copy link

The reason I ask is a few months ago I tested exactly that scenario - due to a wierd issue I was experiencing.
I found that all traffic leaving the container was going to the default gateway and from there being diverted back to the LAN.

I was running Policy Based Routing on my router which was diverting the traffic out of a VPN even when it was destined to the LAN which was initially puzzling, and then annoying even if easy enough to get around.

At the moment I cannot find a suitable container that has traceroute in (the one I used to use was Heimdall - but that no longer has traceroute inbuilt)

@Codelica
Copy link

@aardvarkl that does sound odd. in the end you do have some control over the network side of things though, especially if you have an extra interface on your Scale box. I attach 2 interfaces to my main Docker/DIND container, one of which is directly LAN connected with static IP. That will route LAN traffic directly using that interface. If I want to route Internet traffic out that interface also (vs Scale's default interface) then it would take something like a ip route change default via 10.10.0.1 dev net1 to move it from using eth0 (Scale's) by default.

But what you're describing sounds like it doesn't have a direct route for the LAN. Would probably take an ip route list in the main Docker/DIND container to see what it's thinking.

@aardvarkl
Copy link

It is / was an IX bug. All traffic leaving a K3S container (from Truecharts or IX) that I tested was going to the default gateway, even if on-net and relying on the GW to redirect back to the LAN. IX declined to accept this as a bug saying that this was working correctly. We had something of an argument during which it was agreed that one of us didn't know what the other was talking about when it came to routing behaviour
A Traceroute went as follows:

image-20220921-084323

As far as I am aware thats still going on - but I cannot test for it atm as I cannot find a container that has traceroute any longer (not that I have tried them all and in fact don't use many any longer) and I don't feel like collecting traffic at the firewall.

Jailmaker routes properly, allows me access to the entire hardware of the host and does so simply with less overhead as far as I can tell. Long may it continue (@Jip-Hop )

@Codelica
Copy link

That does sound crazy. I don't want to litter this gist, and can't say I fully understand your configuration, but that's definitely not what I'm seeing. Just to confirm I pulled up a Web shell session in Scale for my "IX Official" Plex container. Then did an apt-get update && apt-get install traceroute net-tools and checked traceroute and routing:

scale-plex-shell

So either that was resolved (I'm on the latest Scale) or there is some other routing issue at play there.

@aardvarkl
Copy link

aardvarkl commented Jun 27, 2023

are there PM's in github? IF you would like to take this offline / elsewhere
cos I am seeing - when I duiplicate what you have done - the same odd behaviour. I would love this to be a configuration issue

@Codelica
Copy link

@aardvarkl Github doesn't but I'm Codelica on Discord and on the TrueNAS community there also.

@aardvarkl
Copy link

and I am Confused on TN on Discord - I think I have sent you a request

@PackElend
Copy link

At that point there is a "safe" and isolated install of Docker running which can be used

can you define safe? If you add an external interface is any network traffic to/from the Docker going through that interface or is there still shared network resources of the host?

@Codelica
Copy link

can you define safe? If you add an external interface is any network traffic to/from the Docker going through that interface or is there still shared network resources of the host?

By safe I mean if IX decides to remove Docker from their base Scale host install (like they going to do I believe) and just keep k3s/containerd, my install of Docker (running under their system) shouldn't be affected. As it's really no different than any other custom container running. So unless they do away with all custom containers, leaving only installs from app catalogs, it should be fine. That would be extreme IMO, and even then I guess it could be done by creating a catalog and config, etc -- but would be a pain.

As far as networking goes, adding other interfaces just gives flexibility. Basically I wanted my Docker apps with their own interface (leaving NAS & Plex stuff alone on my main 10G interface). So I gave it one interface on the LAN one one that's a private network for internal service backends for other machines (dbs, message bus, etc) as I do dev work. Both show (net1, net2) in the docker container with local routes to their subnets. So local traffic from Docker apps to the LAN uses net1 for example. But by default the default internet route would be the eth0 interface that the custom container provides via k3s. But that can be changed by just changing the default route within the Docker container to point to my gateway off net1 for example.

Anyway, I have messages from you guys on Discord so we can continue there. : )

@Jip-Hop
Copy link
Author

Jip-Hop commented Aug 13, 2023

I think it makes more sense now to continue the discussion over here:
https://github.com/Jip-Hop/jailmaker/discussions

😄

@Jip-Hop
Copy link
Author

Jip-Hop commented Aug 14, 2023

Does any of you use jailmaker alongside Apps? Please let me know about your experience in this poll in order to support this pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment