Deploy AKS cluster with Windows support in custom vnet

aks-windows-custom-vnet.md

FYI

• subnet used for your aks cluster should not overlap with any virtual networks in the environment. This creates a new subnet to be used only by AKS.
• service prinicipal must have at least network contributor role to vnet rg
• service-cidr:
  • Must not be within the virtual network IP address range of your cluster
  • Must not overlap with any other virtual networks with which the cluster virtual network peers
  • Must not overlap with any on-premises IPs
  • Must not be within the ranges 169.254.0.0/16, 172.30.0.0/16, or 172.31.0.0/16
• dns-service-ip must be within service cidr AND must not use the first address, which is used for the kubernetes.default.svc.cluster.local address
• docker-bridge-address must not be within vnet address space AND must not overlap with any other address ranges in use.
• network-plugin must be azure
• enable-vmss is required for Windows

# 1. Create rg:
az group create -l location -n vnet

# 2. Create vnet: 
az network vnet create -g vnet -n vnet --address-prefix 10.0.0.0/16 --subnet-name vm-subnet --subnet-prefix 10.0.0.0/24

# 3. Create vm in vm-subnet: 
az vm create -n vm -g vnet --image UbuntuLTS --ssh-key-value 'C:\path\tp\.ssh\pub' --vnet-name vnet --subnet vm-subnet

# 4. Create a new subnet for aks:  
az network vnet subnet create -n aks-subnet -g vnet --vnet-name vnet --address-prefix 10.0.10.0/24

# 5. Create service principal, give it correct permissions. Contributor access over aks rg (and vnet rg for safe measure), and  ensure Network Contributor at least scoped to vnet rg.

export SP_ID=$(az ad sp create-for-rbac --password $SP_PASSWORD --skip-assignment --query [appId] -o tsv)
az role assignment create --assignee $SP_ID --scope $VNET_ID --role Contributor

# 6. Create aks cluster with VMSS enabled, win creds and network plugin
# other params: 1 node, no wait.
az aks create \
    -n aks-vnet \
    -g aks-vnet \
    -c 1 \
    --enable-vmss \
    --service-principal $SP_ID \
    --client-secret $SP_PASSWORD \
    --windows-admin-password replaceP@ssword \
    --windows-admin-username azureuser \
    --ssh-key-value 'C:\path\to\.ssh\.pub' \
    --vnet-subnet-id /subscriptions/xxxx/resourceGroups/vnet/providers/Microsoft.Network/virtualNetworks/vnet/subnets/aks-subnet \
    --network-plugin azure \
    --docker-bridge-address 172.17.0.1/16 \
    --service-cidr 172.38.0.0/16 \
    --dns-service-ip 172.38.0.10 \
    --no-wait

# 7. Add windows nodepool
az aks nodepool add -g aks-vnet --cluster-name aks-vnet -n winvms --os-type Windows -c 1 --node-vm-size Standard_D2_v2 \
--vnet-subnet-id  /subscriptions/xxxxx/resourceGroups/vnet/providers/Microsoft.Network/virtualNetworks/vnet/subnets/aks-subnet

# 8. Deploy pod and verify connection to vm in other subnet
kubectl create -f iis.yaml
kubectl exec iispodname -- ping internal-ip-of-vm

# 9. Remote into windows node:
kubectl exec -it podname -- ssh nodeIP