Skip to content

Instantly share code, notes, and snippets.

@Jip-Hop

Jip-Hop/README.md

Last active April 9, 2024 07:20
Show Gist options
  • Star 9 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d to your computer and use it in GitHub Desktop.
Save Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d to your computer and use it in GitHub Desktop.
Download ZIP
Persistent Debian 'jail' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts. Without modifying the host OS at all thanks to systemd-nspawn!
@Talung
Copy link

Talung commented Jun 23, 2023 •
edited

For me, honestly it just became way too much effort to keep on "Fixing" and finding work-arounds on their updates. I want to use Docker, I want to use VM's, I want to use ZFS and I want to use shares.

TrueNAS can no longer fulfill these needs for me without constant fighting, so I gave up. I have reinstalled Proxmox on my box, installed Docker Natively, and setup my ZFS storage. I just need to add in Shares via SMB and NFS with config files. Then a cronjob to do snapshots and replication. Is it a pain that it is not gui, sure, but end of the day is that it works and I have no worries that a single point update will kill everything.

I am done with TrueNAS... If I want it, I will go to TrueNAS Core because Scale is a wast of time for Home User.

@Codelica
Copy link

Codelica commented Jun 23, 2023 •
edited

FWIW, there is another option if you're just looking for a clean Docker/Compose workspace on Scale and have a decent grasp on things. Basically I use Scale's "Launch Docker Image" to create a pod running Docker (technically a Docker-in-Docker image ala https://hub.docker.com/_/docker or similar -- I'm running one based on DIND some nVidia additions).

At that point there is a "safe" and isolated install of Docker running which can be used -- as Docker on the Scale host could be removed at any point. Using that Docker pod I just start up a container that gives me access and all the tools I would want on a "Docker host" (sshd, compose, code-server, cli tools, etc) which I then use to get access to my "Docker host".

Anyway, that's been working well since the first release of Scale with roughly 30 containers running. "Launch Docker Image" allows the base Docker install to have host networking and even nVidia passthrough (in addition to using it with official IX apps), so it's pretty flexible overall. Obviously in the App UI in Scale I just see my "Docker" app which is running the show. ;)

Not exactly the same as just an isolated host workspace, but quite flexible in the end. Just tossing it out there for those who might find it as an interesting option.

@aardvarkl
Copy link

@Codelica I understand that docker is going away soon - so your solution (if I am correct) will no longer work

@Codelica
Copy link

@aardvarkl that would be the Docker installation that is on the TrueNAS Scale host itself, which has always been in jeopardy. My solution was to run my own Docker install separately as a pod on Scale and use that instance for services. (It's similar to how TrueCharts does their docker-compose chart) So removal of Docker on the Scale host won't affect it. The only real danger would be if IX decides that people can't launch any of their own custom pods and must choose from a catalog. Could happen, but seems doubtful to me.

@aardvarkl
Copy link

@Codelica would you try something for me. In one of your containers that has traceroute, just run a traceroute to a device on your LAN that isn't your router / gateway and post the result please?

@Codelica
Copy link

@aardvarkl sure, although that will depend what networking driver the container uses (host, bridge, macvlan,etc).

But like this is in a host mode container pinging a LAN host:

SeaShell Docker 2023-06-26 08-49-30

and this is in a bridge mode container pinging a LAN host:

email-tr

@aardvarkl
Copy link

aardvarkl commented Jun 26, 2023 via email

@Codelica
Copy link

@aardvarkl it's not. I use a class B at home (10.10.x.x). 10.10.100.1 is a desktop in my bedroom. My router is 10.10.0.1.

@aardvarkl
Copy link

The reason I ask is a few months ago I tested exactly that scenario - due to a wierd issue I was experiencing.
I found that all traffic leaving the container was going to the default gateway and from there being diverted back to the LAN.

I was running Policy Based Routing on my router which was diverting the traffic out of a VPN even when it was destined to the LAN which was initially puzzling, and then annoying even if easy enough to get around.

At the moment I cannot find a suitable container that has traceroute in (the one I used to use was Heimdall - but that no longer has traceroute inbuilt)

@Codelica
Copy link

@aardvarkl that does sound odd. in the end you do have some control over the network side of things though, especially if you have an extra interface on your Scale box. I attach 2 interfaces to my main Docker/DIND container, one of which is directly LAN connected with static IP. That will route LAN traffic directly using that interface. If I want to route Internet traffic out that interface also (vs Scale's default interface) then it would take something like a ip route change default via 10.10.0.1 dev net1 to move it from using eth0 (Scale's) by default.

But what you're describing sounds like it doesn't have a direct route for the LAN. Would probably take an ip route list in the main Docker/DIND container to see what it's thinking.

@aardvarkl
Copy link

It is / was an IX bug. All traffic leaving a K3S container (from Truecharts or IX) that I tested was going to the default gateway, even if on-net and relying on the GW to redirect back to the LAN. IX declined to accept this as a bug saying that this was working correctly. We had something of an argument during which it was agreed that one of us didn't know what the other was talking about when it came to routing behaviour
A Traceroute went as follows:

image-20220921-084323

As far as I am aware thats still going on - but I cannot test for it atm as I cannot find a container that has traceroute any longer (not that I have tried them all and in fact don't use many any longer) and I don't feel like collecting traffic at the firewall.

Jailmaker routes properly, allows me access to the entire hardware of the host and does so simply with less overhead as far as I can tell. Long may it continue (@Jip-Hop )

@Codelica
Copy link

That does sound crazy. I don't want to litter this gist, and can't say I fully understand your configuration, but that's definitely not what I'm seeing. Just to confirm I pulled up a Web shell session in Scale for my "IX Official" Plex container. Then did an apt-get update && apt-get install traceroute net-tools and checked traceroute and routing:

scale-plex-shell

So either that was resolved (I'm on the latest Scale) or there is some other routing issue at play there.

@aardvarkl
Copy link

aardvarkl commented Jun 27, 2023 •
edited

are there PM's in github? IF you would like to take this offline / elsewhere
cos I am seeing - when I duiplicate what you have done - the same odd behaviour. I would love this to be a configuration issue

@Codelica
Copy link

@aardvarkl Github doesn't but I'm Codelica on Discord and on the TrueNAS community there also.

@aardvarkl
Copy link

and I am Confused on TN on Discord - I think I have sent you a request

@PackElend
Copy link

At that point there is a "safe" and isolated install of Docker running which can be used

can you define safe? If you add an external interface is any network traffic to/from the Docker going through that interface or is there still shared network resources of the host?

@Codelica
Copy link

can you define safe? If you add an external interface is any network traffic to/from the Docker going through that interface or is there still shared network resources of the host?

By safe I mean if IX decides to remove Docker from their base Scale host install (like they going to do I believe) and just keep k3s/containerd, my install of Docker (running under their system) shouldn't be affected. As it's really no different than any other custom container running. So unless they do away with all custom containers, leaving only installs from app catalogs, it should be fine. That would be extreme IMO, and even then I guess it could be done by creating a catalog and config, etc -- but would be a pain.

As far as networking goes, adding other interfaces just gives flexibility. Basically I wanted my Docker apps with their own interface (leaving NAS & Plex stuff alone on my main 10G interface). So I gave it one interface on the LAN one one that's a private network for internal service backends for other machines (dbs, message bus, etc) as I do dev work. Both show (net1, net2) in the docker container with local routes to their subnets. So local traffic from Docker apps to the LAN uses net1 for example. But by default the default internet route would be the eth0 interface that the custom container provides via k3s. But that can be changed by just changing the default route within the Docker container to point to my gateway off net1 for example.

Anyway, I have messages from you guys on Discord so we can continue there. : )

@Jip-Hop
Copy link
Author

Jip-Hop commented Aug 13, 2023

I think it makes more sense now to continue the discussion over here:
https://github.com/Jip-Hop/jailmaker/discussions

😄

@Jip-Hop
Copy link
Author

Jip-Hop commented Aug 14, 2023

Does any of you use jailmaker alongside Apps? Please let me know about your experience in this poll in order to support this pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment