Skip to content

Instantly share code, notes, and snippets.

@minatoaquaMK2

minatoaquaMK2/rke2-with-cilium-on-wsl2.md

Last active July 28, 2023 16:45
Show Gist options
  • Save minatoaquaMK2/3ce3c8dfa302e2294a0810719fa2ac7b to your computer and use it in GitHub Desktop.
Save minatoaquaMK2/3ce3c8dfa302e2294a0810719fa2ac7b to your computer and use it in GitHub Desktop.
Download ZIP
Install RKE2 with Cilium on WSL2
Raw
rke2-with-cilium-on-wsl2.md

Install RKE2 with Cilium on WSL2

Compiling your own WSL2 Kernel for Cilium

  1. Run a Docker container to build the image
    2. docker run --name wsl-kernel-builder --rm -it ubuntu@20.04 bash

    from inside the container (now bash):

    WSL_COMMIT_REF=linux-msft-wsl-5.15.74.2
apt update && apt install -y git build-essential flex bison libssl-dev libelf-dev bc dwarves

mkdir src
cd src
git init
git remote add origin https://github.com/microsoft/WSL2-Linux-Kernel.git
git config --local gc.auto 0
git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +${WSL_COMMIT_REF}:refs/remotes/origin/build/linux-msft-wsl-5.15.y
git checkout --progress --force -B build/linux-msft-wsl-5.15.y refs/remotes/origin/build/linux-msft-wsl-5.15.y

# adds support for clientIP-based session affinity
sed -i 's/# CONFIG_NETFILTER_XT_MATCH_RECENT is not set/CONFIG_NETFILTER_XT_MATCH_RECENT=y/' Microsoft/config-wsl

# required modules for Cilium
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_TPROXY is not set/CONFIG_NETFILTER_XT_TARGET_TPROXY=y/' Microsoft/config-wsl
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_CT is not set/CONFIG_NETFILTER_XT_TARGET_CT=y/' Microsoft/config-wsl
sed -i 's/# CONFIG_NETFILTER_XT_MATCH_MARK is not set/CONFIG_NETFILTER_XT_MATCH_MARK=y/' Microsoft/config-wsl
sed -i 's/# CONFIG_NETFILTER_XT_MATCH_SOCKET is not set/CONFIG_NETFILTER_XT_MATCH_SOCKET=y/' Microsoft/config-wsl

# build the kernel
make -j2 KCONFIG_CONFIG=Microsoft/config-wsl
  2. Copy the image out of the container
    3. Open up a new terminal: 
    docker cp wsl-kernel-builder:/src/arch/x86/boot/bzImage .
  3. Tell Windows Subsystem for Linux to use your kernel
    4. Edit C:\Users\something\.wslconfig 
    [wsl2]
kernel=C:\\Users\\<your_user>\\bzImage

    Keep the double slashes. Shut down WSL (in PowerShell: wsl --shutdown)

Assigning a Static IP Address to a WSL2 Distribution

Hyper-V creates a hidden virtual switch for WSL2. In Windows, the virtual NIC vEthernet (WSL) is connected to the switch. In WSL2 (Ubuntu), the virtual NIC eth0 is connected to the switch. Communication between the two network endpoints happens over the switch. The virtual NICs (and possibly the switch) are ephemeral and disappear at host system restart time. The NICs are recreated on demand when WSL2 first runs.

To use a custom static IP address we can assign each of the NICs to IP addresses on a shared subnet. It's a good idea to pick a subnet in the Private Address range.

The following steps help set up a fixed IP address for a WSL2 distribution from the host and a fixed IP address for the host from WSL2, with this configuration in mind:

  • Subnet: 192.168.2.0/24

  1. Assign a new IP address to the virtual NIC in Windows

    Assign the virtual NIC connected to WSL2 an additional IP address 192.168.2.1 (Requires "Run as Administrator"):

    netsh interface ip add address "vEthernet (WSL)" 192.168.2.1 255.255.255.0

    To remove in the future:

    netsh interface ip delete address "vEthernet (WSL)" 192.168.2.1

  2. Assign a new IP address to the virtual NIC in WSL2

    Assign the virtual ethernet NIC an additional IP address 192.168.2.2:

    sudo ip addr add 192.168.2.2/24 broadcast 192.168.2.255 dev eth0 label eth0:1

    To remove in the future:

    sudo ip addr del 192.168.2.201/24 dev eth0:1

  3. Set up Windows firewall allow rule (once only)

    The vEthernet (WSL) network device uses the Public Windows network profile, where all traffic is blocked by default. We need to allow traffic from the new 192.168.2.0/24 subnet to access the host Windows machine from WSL2.

    1. Open Windows Defender Firewall with Advanced Security
    2. In Inbound rules, add a new Inbound Rule
      1. Select "Custom Rule"
      2. Select "All programs"
      3. Select "Any" Protocol Type
      4. Scope to remote IP addresses 192.168.2.0/24
      5. Select "Allow the connection"
      6. Select only "Public" for the rule to apply
      7. Name WSL2 or similar
    3. In Inbound rules, remove any existing block rules for applications that WSL2 needs to access, as these will take precedence over the allow rule. These are usually created by Windows when you first run an application (the UAC modal warning asking you about firewall rules sets these up).

Note: As the NICs are ephemeral these changes must be applied following every host system restart

PowerShell command to set up static IP addresses after the WSL2 started (Requires "Run as Administrator"):

netsh interface ip add address "vEthernet (WSL)" 192.168.2.1 255.255.255.0

You can verify the changes by using the following command on the windows

$ netsh interface ip show
---SNIP---
接口 "vEthernet (WSL)" 的配置
    DHCP 已启用:                          否
    IP 地址:                           172.17.144.1
    子网前缀:                        172.17.144.0/20 (掩码 255.255.240.0)
    # The following indicate the a new IP address is assigned to the virtual NIC in Windows
    IP 地址:                           192.168.2.1
    子网前缀:                        192.168.2.0/24 (掩码 255.255.255.0)
    InterfaceMetric:                      5000

Install RKE2 with Cilium

Note: The RKE2 installation process must be run as the root user

  1. Run the installer

    curl -sfL https://get.rke2.io | sh -

  2. Enable the rke2-server service

    systemctl enable rke2-server.service

  3. Add a systemd service to share mount /sys/fs/bpf and /run/cilium/cgroupv2

    cat << EOF > /usr/local/lib/systemd/system/wsl2-rke2-cilium-automount.service
[Unit]
Description=Service that share mount /sys/fs/bpf and /run/cilium/cgroupv2
Wants=network-online.target rke2-server.service
After=rke2-server.service

[Service]
ExecStart=/bin/sh -c 'mount --make-shared /sys/fs/bpf; exit 0'
ExecStart=/bin/sh -c 'mount  --make-shared /run/cilium/cgroupv2; exit 0'

KillMode=none
Type=oneshot
RemainAfterExit=true

[Install]
WantedBy=multi-user.target
EOF

systemctl enable wsl2-rke2-cilium-automount.service
systemctl start wsl2-rke2-cilium-automount.service

  4. Add a systemd service to assign static ip for wsl2 automatically

    cat << EOF > /usr/local/lib/systemd/system/wsl2-static-ip.service
[Unit]
Description=Service that assign a static ip address for wsl2
Wants=network-online.target
After=network-online.target
Before=rke2-server.service

[Service]
ExecStart=/bin/sh -c 'ip addr add 192.168.2.2/24 broadcast 192.168.2.255 dev eth0 label eth0:1; exit 0'

KillMode=none
Type=oneshot
RemainAfterExit=true

[Install]
WantedBy=multi-user.target
EOF

systemctl enable wsl2-static-ip.service
systemctl start wsl2-static-ip.service

  5. Configuring RKE2 Server Nodes

    mkdir -p /etc/rancher/rke2
cat << EOF > /etc/rancher/rke2/config.yaml
cni: cilium
node-ip: 192.168.2.2
EOF

    For more server configuration (https://docs.rke2.io/reference/server_config)

  6. Start the rke2-server service

    systemctl start rke2-server

  7. Add path to $PATH

    cat << EOF > ~/.bashrc
export PATH="/var/lib/rancher/rke2/bin:$PATH"
EOF
source ~/.bashrc

  8. Copy kubeconfig to the default path

    mkdir -p ~/.kube
ln -s /etc/rancher/rke2/rke2.yaml ~/.kube/config

  9. Get node info

    $ kubectl get nodes
NAME              STATUS   ROLES                       AGE    VERSION
<HOSTNAME>   Ready    control-plane,etcd,master   3h4m   v1.24.8+rke2r1

Reference

Compiling your own WSL2 Kernel for Cilium by Hart Hoover(https://harthoover.com/compiling-your-own-wsl2-kernel)

Assigning a Static IP Address to a WSL2 Distribution by wllmsash(https://gist.github.com/wllmsash/1636b86eed45e4024fb9b7ecd25378ce)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment