I've got this idea for an ingress controller VM using OpenWRT. I'll eventually get this implemented on my proxmox machine, but for now I'm developing my implementation in virtualbox.
Using this guide to get the generic x86-64 UEFI image to run on vb: https://openwrt.org/docs/guide-user/virtualization/virtualbox-vm
The first challenge is to design a working networking. One part of this goal will be to get LuCi to not conflict with Nginx. I could have LuCi listen to localhost (127.0.0.1), but for development purposes, I will provide it with an additional host-only testing adapter, so that LuCi can be reached directly for testing.
This VM will have 3 adapters overall:
- Host-only adapter attached to br-lan so the host can act as a client (192.168.x.1)
- Bridged WAN adapter (DHCP)
- Host-only adapter to attach LuCi to (172.16.x.1)
Tip
For easier setup, make sure to attach these adapters before running the vm for the first time (vb will only attach a single NAT adapter by default). Also recommend that you do them in the order listed:
- LAN will automatically be assigned to the first interface
- WAN will be assigned to the second
- 3rd will have to be configured manually
This way the only thing you need to change to get it up and running via CLI/SSH is the lan address with:
uci set network.lan.ipaddr=192.168.x.1
uci commit
service network restart
We will revisit more network settings later. For now, I do recommend logging into LuCi and temporarily disabling gateway settings for the LAN adapter as to not slow down your network. Otherwise, we move on to drive resizing.
UEFI partitions require an additional step for resizing partitions than typical for OpenWRT BIOS installs. This is because UEFI keeps track of partitions used for boot via partition identifiers. To communicate changes in partitions to UEFI, we have to update /boot/grub/grub.cfg
opkg update && opkg install lsblk fdisk losetup resize2fs
fdisk -l
lsblk -o PATH,SIZE,PARTUUID
# fdisk -l > /root/fdisk_old.txt # Save to txt
# lsblk -o PATH,SIZE,PARTUUID > /root/lsblk_old.txt # Save to txtfdisk /dev/sda
# ...
# Command (m for help): <p> # list partitions
# ...
# Command (m for help): <d> # delete partitions
# Partition number (1,2,128, default 128): <2>
# ...
# Command (m for help): <n> # create partitions
# Partition number (2-127, default 2): <2>
# First sector (33280-20971486, default 34816): # use the same start from the p command
# Last sector, +/-sectors or +/-size{K,M,G,T,P} (34816-20971486, default 20969471): # use default
# Command (m for help): <w> # write changes
# ... will error out, ignore ...lsblk -o PATH,SIZE,PARTUUID # to get the new PARTUUID
# 188711c8-63eb-4874-bdc1-2fcd6ed85193
vi /boot/grub/grub.cfg # cBOOT="$(sed -n -e "\|\s/boot\s.*$|{s///p;q}" /etc/mtab)"
PART="${BOOT##*[^0-9]}"
DISK="${BOOT%${PART}}"
ROOT="${DISK}$((PART+1))"
LOOP="$(losetup -f)"
losetup ${LOOP} ${ROOT}
resize2fs -f ${LOOP}
rebootOnce initial networking and drive resizing is complete, login to LuCi (http://192.168.x.1) to setup the rest of the router settings (recommend that you setup a password).
Go to Network > Interfaces > Devices > Add Device. Then add a new bridge (br-luci) with the eth2 port for the 3rd adapter.
Go to Network > Interfaces > Interfaces > Add Interface. Then add an interface for the new bridge with a static IP address (172.16.x.1/24--255.255.255.0 netmask) and add it to the LAN firewall zone.
For convenience, you can install luci-app-uhttpd (a gui counter part to uci > uhttpd)
Go to Services > uHTTPd then remove all http listening addresses and add the ip:port for the 3rd interface (172.16.x.1:80). Leave https alone for the minute. Save and test. If the web ui comes up on the new http address, delete all https entries (we won't need them, since we will be using nginx TLS termination)
This will allow us to run nginx on one interface and uhttpd on the other.
Nginx supports uci out of the box, however for this setup we will be disabling the uci nginx integration (see Future Plans) to support nginx ui.
opkg update && opkg install nginx-ssl
uci set nginx.global.uci_enable=false
uci commit
service nginx reloadI do eventually want to revisit this project and make a few different improvements.
Firstly, uci supports configuring nginx out of the box. Since uci is the main configuration command line for OpenWRT, I'd like to make more use of it. Also, the fact that nginx supports uci could mean that it would be extremely easy to write a LuCi plugin for nginx.
Another approach would be to integrate traefik with uci and LuCi. While I am not aware of any current integrations with uci and traefik, traefik's configuration architecture is pretty simplistic and serializable. Since OpenWRT comes with Lua and Lua supports serializing yaml/toml as well as uci config files, integrating the 2 should not be difficult. By creating a lua-based service manager for traefik, uci config files could be translated to yaml or toml and vice versa. If this is possible, then the following command line could be used to easily setup and configure traefik:
uci set traefik.blah.blah.blah... = ...
uci commit
service traefik restartAnother reverse proxy worth being explored is OAuth2-proxy, but currently my plan is to implement OAuth2-proxy in tandem with traefik or nginx.
As for UI, I may consider using nginx proxy manager instead, but for this project I was interested in playtesting nginx-ui