Skip to content

Instantly share code, notes, and snippets.

@zenxedo

zenxedo/Proxmox-RancherOS-Portainer Ultimate Setup.md

Last active January 18, 2024 05:20
Show Gist options
  • Star 85 You must be signed in to star a gist
  • Fork 22 You must be signed in to fork a gist
  • Save zenxedo/a57e893fc640ccf3720e99fc6b3b879a to your computer and use it in GitHub Desktop.
Save zenxedo/a57e893fc640ccf3720e99fc6b3b879a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Proxmox-RancherOS-Portainer Ultimate Setup.md

Install Proxmox and a RancherOS VM with Portainer to manage docker containers

WARNING MAY BE INCORRECT AND INCOMPLETE, USE AT YOUR OWN RISK

I have tried this setup and although it works and may be good for ceratin circumstances I would advise using ubuntu as the base with docker, docker-compose, and portainer. https://gist.github.com/mow4cash/626275e095f7f90898944a85d66b3be6

Link to my docker run file https://gist.github.com/mow4cash/6a25343cdeb0cd115f263dea0a3b623d

Setup Proxmox

  1. Install Proxmox 6.X iso
  2. Console/SSH into Proxmox
  3. nano /etc/apt/sources.list
  4. edit the file to look like this
deb http://ftp.debian.org/debian buster main contrib
deb http://ftp.debian.org/debian buster-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

# security updates
deb http://security.debian.org buster/updates main contrib
  1. apt update && apt dist-upgrade -y
  2. reboot system

Install RancherOS

  1. Upload the RancherOS iso to (local)pve
  2. Setup a VM with RancherOS ISO as CD. Give it at least 3gb ram to start. Rancher Server failed with low ram
  3. Boot
  4. From Console change password -sudo bash -passwd rancher
  5. SSH to rancher@
  6. prepare your ssh keys with putty gen
    -vi cloud-config.yml
  7. paste the cloud config edited with your settings, make sure the pasted data is pated correctly, add your key in a single line and make sure the file has #cloud-config in the beginning
  8. press exit exit :wq to save
#cloud-config

rancher: rancheros
  network:
    interfaces:
      eth0:
        address: 10.68.69.92/24
        gateway: 10.68.69.1
        mtu: 1500
        dhcp: false
    dns:
      nameservers:
      - 1.1.1.1
      - 8.8.4.4

ssh_authorized_keys:
  - ssh-rsa <YOUR KEY>
  1. sudo ros config validate -i cloud-config.yml
  2. sudo ros install -c cloud-config.yml -d /dev/sda
  3. Remove CD Image from VM, and then reboot.
  4. SSH back into RancherOS (rancher@) using your new ssh private key

Create NFS Shares on FreeNAS

  1. create a unix dataset called appsNFS with root and wheel as the user, set a quota for 50gb
  2. create a nfs share to the dataset you created, select all dirs, mapall user:group to root:wheel
  3. enable nfs sharing and select nfsv4, allow non-root, nfsv3 ownership for nfsv4
  4. reboot freenas

Add NFS mnt to RancherOS

sudo ros config set mounts '[["10.68.69.2:/mnt/myVol/appsNFS", "/mnt/appsNFS", "nfs4",""]]'

Install Portainer with NFS share

sudo docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock --restart always --name portainer -v /mnt/appsNFS/portainer:/data portainer/portainer
  1. Navigate to http://hostIP:9000 and select local
  2. When adding volumes to a container select bind and use the path /mnt/appsNFS/whateveryouwanthere

Add macvlan so containers are given an IP and mac from your LAN

https://www.portainer.io/2018/09/using-macvlan-portainer-io/

  1. click add network
  2. select macvlan
  3. enter in your lan network
  4. select enable manual connection
  5. when creating a container select the network you just added and give it an availble static IP

Rancher OS commands and resources

sudo vi /var/lib/rancher/conf/cloud-config.yml  ##edit config file

https://medium.com/the-code-review/clean-out-your-docker-images-containers-and-volumes-with-single-commands-b8e38253c271 https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes

Updates

Proxmox

Your PVE GUI and slect the upgrade button

Rancher OS

sudo ros os upgrade

Update Portainer

docker stop portainer
docker rm portainer
docker pull portainer/portainer:latest
docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock --restart always --name portainer -v /mnt/appsNFS/portainer:/data portainer/portainer

Install Rancher (optional)

  • sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher

  • log in to ranhcer thorugh the web browser

  • Add Cluster.

  • Choose Custom.

  • Enter a Cluster Name. Click Next.

  • From Node Role, select all the roles: etcd, Control, and Worker.

  • Copy the command displayed on screen to your clipboard.

  • Log in to your Rancher host with PuTTy. Run the command copied to your clipboard.

  • When you finish running the command on your Linux host, click Done.

  • Wait for your cluster to finish provisioning

  • Reboot to make sure everything is working right

Creating your first container

  • In your cluster drop down tab select default then deploy
  • give it a name and add the ports and env needed
@8ear
Copy link

8ear commented Jun 15, 2019

Very Nice!
I have done it in another setup.
Proxmox -> KVM VM -> RancherOS
But RancherOS is started via iPXE boot.
So I have a VM with an NFS Server and a TFTP Server and the DHCP distribute the next server and bootloader file. in the iPXE config I directly integrate the cloud-config.yml file. So I can create a new KVM VM add this VM to an fixed DHCP lease and start. After 5minute or so the VM is ready. :-)

Kind regards

@IntrovertedFL
Copy link

IntrovertedFL commented Jul 31, 2019 •
edited

Can I install nginx on my "PVE", and if so would that allow me to create a 4 node Rancher HA Cluster with 4 separate vm's? Would I also be able to issue an LE Cert via Rancher/Ingress for the "PVE Web UI? This article (https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy) made me think it might be possible but I am pretty new to proxmox and rancher. I currently have Proxmox 5.4 - 11 installed. My main goal is to use Rancher to replace Plesk for managing my web apps/projects. I also have a friend with a server with same specs, I bought the one I have in hopes to combine our resources as we develop a lot together. Thanks in advance for any help/suggestions.

@8ear
Copy link

8ear commented Aug 1, 2019

You can do it. But I do not recommend it to you.
You have different possiblities.

  1. Do it on the root system without any virtualization
  2. Install an HA Proxy in a VM (or PFsense like I'm) and route via IPtables all traffic from the server to this VM.
    EXCEPT: your SSH port :-)

I have in one VM a PFsense installed an all traffic will be redirected to this VM. This VM is the gateway between my VMs and LXCs and the Internet. :-)
Advantages:

  • One place for ACME certs
  • One place for new websites
  • One Place for firewall rules
  • ...

@IntrovertedFL
Copy link

IntrovertedFL commented Aug 1, 2019 •
edited

So basically using PFsense as a load balancer? Any articles on this you can share? I am googling myself but if you know of any good ones dealing with proxmox also I would appreciate it.

@supermarsx
Copy link

YAML correction

#cloud-config

rancher:
  network:
    interfaces:
      eth0:
        address: 10.68.69.92/24
        gateway: 10.68.69.1
        mtu: 1500
        dhcp: false
    dns:
      nameservers:
      - 1.1.1.1
      - 8.8.4.4

ssh_authorized_keys:
  - ssh-rsa <YOUR KEY>

@outlander4000
Copy link

Thank you very for bringing me up to speed with this great tutorial!!

@urza
Copy link

urza commented May 15, 2022

Links in your first paragraph lead to 404

@supermarsx
Copy link

Links in your first paragraph lead to 404

Those links are gone unless he saved them, besides now there are better (simple) tools to get the job done such as docker + portainer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment