Skip to content

Instantly share code, notes, and snippets.

@BorysVrublevskyi

BorysVrublevskyi/Kubernetes-Hybrid-Cluster-on-Hyper-V.md

Last active April 8, 2024 12:44
Show Gist options
  • Save BorysVrublevskyi/3c13f63b2d1891623b71098a572eaf55 to your computer and use it in GitHub Desktop.
Save BorysVrublevskyi/3c13f63b2d1891623b71098a572eaf55 to your computer and use it in GitHub Desktop.
Download ZIP
Get your own home lab k8s cluster!
Raw
Kubernetes-Hybrid-Cluster-on-Hyper-V.md

Kubernetes hybrid cluster (Linux+Windows) on Hyper-V with local subnet and internet access

Prepare Host

Install CLI utils on your host machine

Kubectl

Helm

Kustomize (optional)

Prepare host network

Hyper-V NAT manual

New-VMSwitch -SwitchName "K8S-Internal" -SwitchType Internal
Get-NetAdapter # Mention InterfaceIndex of the K8S-Internal swith
$MyInterfaceIndex = 
New-NetIPAddress -IPAddress 30.30.30.1 -PrefixLength 24 -InterfaceIndex $MyInterfaceIndex
New-NetNat -Name "K8S-Internal-vNAT" -InternalIPInterfaceAddressPrefix 30.30.30.0/24
Get-DnsClientServerAddress # -InterfaceAlias "Ethernet"
Set-DnsClientServerAddress -InterfaceIndex $MyInterfaceIndex -ServerAddresses ("1.1.1.1","1.0.0.1")
Set-NetIPInterface -InterfaceIndex $MyInterfaceIndex -InterfaceMetric 30
# Get-NetNat | Remove-NetNat
# Remove-NetIPAddress -IPAddress 30.30.30.1 -InterfaceIndex $MyInterfaceIndex
# Remove-VMSwitch -SwitchName "K8S-Internal"

# Add-NetNatStaticMapping -ExternalIPAddress "0.0.0.0/24" -ExternalPort 64430 -Protocol TCP -InternalIPAddress "30.30.30.20" -InternalPort 6443 -NatName K8S-Internal-vNAT

# New-NetFirewallRule -DisplayName "K8S-Internal" -Direction Inbound  -InterfaceAlias "vEthernet (K8S-Internal)"  -Action Allow

Prepare VMs (choose your favorite Linux distro)

Prepare Debian VM

Download iso

Delete SWAP during installation, enable trim and noarime for "/" partition, set static IP.

nano /etc/resolv.conf
nameserver 1.1.1.1
nameserver 1.0.0.1
ifdown eth0 && ifup eth0
# or ??
/etc/network/interfaces:
dns-nameservers 1.1.1.1 1.0.0.1

systemctl restart networking.service

apt install openssh-server

Kubectl install

apt-get install -y apt-transport-https ca-certificates curl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubectl

Docker install

apt-get update
apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io

Prepare Ubuntu VM

Set static IP and DNS

nano /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    eth0:
      addresses:
      - 30.30.30.20/24
      gateway4: 30.30.30.1
      nameservers:
        addresses:
        - 1.1.1.3
        - 1.0.0.3
  version: 2
sudo netplan apply

Disable SWAP

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a
#nano /etc/fstab # comment swap

Install Docker

https://docs.docker.com/engine/install/ubuntu/

Installing kubeadm

Container runtimes

Configuring a cgroup driver

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get install docker-ce docker-ce-cli containerd.io
sudo usermod -aG docker $USER


sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF


sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

Install Kubernetes

Wiki

sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
sudo rm -f /etc/apt/sources.list.d/kubernetes.list

# Add new k8s keyring and repo. See Wiki

sudo apt-get update

# Install latest version, or...
sudo apt-get install -y kubelet kubeadm kubectl # install latest version

# Install specified version
sudo apt-cache madison kubelet # show avaliable versions
kubever=1.19.12-00
sudo apt-get install -qy --allow-downgrades kubelet=$kubever kubectl=$kubever kubeadm=$kubever

sudo apt-mark hold kubelet kubeadm kubectl

Prepare Centos VM

Install Docker on Centos 7-8

Setup Container Runtime

Install Kubernetes

Disable SWAP

sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a
# nano /etc/fstab # comment swap

Generate new UUID for eth0

NEWUUID=UUID=\"$(uuidgen eth0)\"
grep UUID /etc/sysconfig/network-scripts/ifcfg-eth0
sed -i "s/^UUID.*/$NEWUUID/g" /etc/sysconfig/network-scripts/ifcfg-eth0
grep UUID /etc/sysconfig/network-scripts/ifcfg-eth0

Initialize Master node

# sudo rm -f /etc/cni/net.d/*
# sudo kubeadm init --kubernetes-version $(kubeadm version -o short) --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=k8s-cluster.loc

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=k8s-cluster.loc

sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $USER. $HOME/.kube/config

Join Node

Run on master node to create new token and get join command or use command that was showen after master node was initialized

# Show command to add worker node
kubeadm token create --print-join-command

Add --control-plane to the 'kubeadm join' command to add additional master node

Run the output command on the node you want to join.

Install Container Network Interface (CNI)

Flannel - Linux and hybrid clusters

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
nano kube-flannel.yml

# net-conf.json: |
#     {
#       "Network": "10.244.0.0/16",
#       "Backend": {
#         "Type": "vxlan",
#         "VNI": 4096,
#         "Port": 4789
#       }
#     }

kubectl apply -f kube-flannel.yml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/flannel/flannel-host-gw.yml

Upgrade Kubernetes

sudo kubeadm upgrade plan
mv ~/.kube/config{,.bak}
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo apt-mark unhold kubeadm
sudo apt upgrade
sudo apt update && sudo apt upgrade
sudo apt-mark hold kubelet kubeadm kubectl
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $USER. $HOME/.kube/config

Prepare Windows VM

Set PowerShell as default CLI

powershell
Set-ItemProperty -Path 'HKLM:\Software\Microsoft\Windows NT\CurrentVersion\WinLogon' -Name Shell -Value 'powershell -NoExit -Command "Set-Location %userprofile%"'
Restart-Computer -Force

Set static IP and DNS

netsh interface ipv4 show config
#netsh interface ip set address "Ethernet" static 30.30.30.22 255.255.255.0 30.30.30.1
#netsh interface ip add dns "Ethernet" 1.1.1.3
#netsh interface ip add dns "Ethernet" 1.0.0.3 index=2
Get-NetAdapter # Mention your interface index
$MyInterfaceIndex = # paste index here
# Get-NetIPConfiguration
# Get-NetIpAddress
# Get-NetRoute
New-NetIPAddress –IPAddress 30.30.30.33 -DefaultGateway 30.30.30.1 -PrefixLength 24 -InterfaceIndex $MyInterfaceIndex
Set-DNSClientServerAddress –InterfaceIndex $MyInterfaceIndex –ServerAddresses 1.1.1.1,9.9.9.9

notepad C:\Windows\System32\drivers\etc\hosts

Rename node

Rename-Computer -NewName "K8SW-01"

Install Docker

Wiki page

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name docker -ProviderName DockerMsftProvider
Restart-Computer -Force

Add some GUI

Install Explorer++

curl.exe -LO https://explorerplusplus.com/software/explorer++_1.3.5_x64.zip
Expand-Archive -Path '.\explorer++_1.3.5_x64.zip' -DestinationPath '.\explorer++_1.3.5_x64'
Move-Item -Path .\explorer++_1.3.5_x64\Explorer++.exe -Destination C:\Windows\System32\Explorer++.exe
Remove-Item -Path .\explorer++_1.3.5_x64\ -Recurse -Force

Install CProcess (optional)

curl.exe -LO https://www.nirsoft.net/utils/cprocess.zip
Expand-Archive -Path '.\cprocess.zip' -DestinationPath '.\CProcess'
Move-Item -Path .\CProcess\CProcess.exe -Destination C:\Windows\System32\CProcess.exe
Remove-Item -Path .\CProcess\ -Recurse -Force

# https://www.nirsoft.net/utils/serviwin.zip
# https://www.nirsoft.net/utils/serviwin-x64.zip

Nested virtualization (optional)

The following script will configure nested virtualization for the container host. This script is run on the parent Hyper-V machine. Ensure that the container host virtual machine is turned off when running this script.

# Replace with the virtual machine name
$vm = "<virtual-machine>"

# Configure virtual processor
Set-VMProcessor -VMName $vm -ExposeVirtualizationExtensions $true -Count 4

# Disable dynamic memory
Set-VMMemory -VMName $vm -DynamicMemoryEnabled $false

# Enable mac spoofing
Get-VMNetworkAdapter -VMName $vm | Set-VMNetworkAdapter -MacAddressSpoofing On

Install kubelet, kubeadm, wins.

# curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/PrepareNode.ps1
curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/scripts/PrepareNode.ps1
# Check your Kubeadm version on master node before run:
.\PrepareNode.ps1 -KubernetesVersion v1.21.3

Adding Windows nodes

Wiki page

Delete K8S from Windows

Get-Service -Name "Rancher Wins"
Stop-Service -Name "Rancher Wins"
# Invoke-WebRequest -Uri https://nssm.cc/release/nssm-2.24.zip -OutFile nssm-2.24.zip # must be already in system
# Expand-Archive .\nssm-2.24.zip
# move nssm.exe to current dir
nssm remove rancher-wins confirm
Stop-Service -Name "Kubelet"
nssm remove kubelet confirm
rm C:\etc
rm C:\k
rm C:\var\
# shutdown /r /t 1

Windows debug

get-wmiobject win32_networkadapter | select netconnectionid, name, InterfaceIndex, netconnectionstatus
Get-NetIPAddress -AddressFamily IPv4

# Fix Network
Get-NetIPAddress -InterfaceIndex 12
Get-NetRoute -InterfaceIndex 12
Remove-NetIPAddress -InterfaceIndex 12
New-NetIPAddress -InterfaceIndex 12 -IPAddress 192.168.100.2 -PrefixLength 24
Remote-NetRoute -InterfaceIndex 12
New-NetRoute -InterfaceIndex 12 -NextHop 192.168.100.1 -DestinationPrefix 0.0.0.0/0
ping 192.168.1.1
Get-DnsClientServerAddress -Interface 12
Set-DnsClientServerAddress -Interface 12 -ServerAddresses @("8.8.8.8","8.8.4.4")
ping google.com

Linux Debug

journalctl -xeu kubelet
sudo grep -rl "30.30.30.20" /etc/kubernetes | sort -u

MetalLB Loadbalancer

Kubernetes Bare-metal considerations

MetalLB. Configuration

helm repo add metallb https://metallb.github.io/metallb
helm repo update
helm install metallb metallb/metallb -f metallb-values.yaml
# metallb-values.yaml
configInline:
  address-pools:
   - name: default
     protocol: layer2
     addresses:
     - 30.30.30.130-30.30.30.150

NGINX Ingress Controller

Installation Guide

Basic Usage

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx

Sources:

Kubectl Cheatsheet

Creating a cluster with kubeadm

Adding Windows nodes

Creating Highly Available clusters with kubeadm

Kubernetes Dashboard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment