wsong/windows_on_kube.md Secret

## windows_on_kube.md

      
    Raw
  

              windows_on_kube.md
            
          
    Windows on Kube

The below instructions describe how to set up a cluster with one Linux manager and one Windows worker such that you can schedule Kubernetes pods with Windows images on the Windows worker. I did this on Azure, but there's nothing Azure-specific in these instructions.
These instructions are for a Windows node running 1709. I have not tested this with Windows 1803, but it will probably work; however, you need to change all the images that use :1709 to :1803 or similar.
On Azure, I recommend VMs of size D4s v3 or larger (4 CPUs, 16GB memory)
Linux manager setup

Make sure that your Linux node has Docker version 17.06.2-ee-8 or above.
export NODE_IP=<Linux node private IP address>
export NODE_PUBLIC_IP=<Linux node public IP address>
# This version of UCP is a special build with Kube 1.11.0-beta.0 and a tweak to
# the node inventory logic so that it can handle Windows Kube nodes
export UCP_VERSION=3.1.0-5a4bc69

docker login
docker run --rm dockereng/ucp:$UCP_VERSION images --list --image-version dev: | xargs -n 1 docker pull
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock \
    dockereng/ucp:$UCP_VERSION \
    install --swarm-port 3376 --force-insecure-tcp --admin-password orcaorcaorca --image-version dev: --host-address $NODE_IP --san $NODE_PUBLIC_IP
Setting up Azure cloud provider (optional)

If you're on Azure, the above instructions will allow you to run Windows pods, but if you want networking to work, you need to make sure UCP gets deployed on the Linux manager with Calico set up in Azure mode.
Write a file called /etc/kubernetes/azure.json on your Linux manager with the following contents:
{
    "tenantId": "<tenant ID>",
    "subscriptionId": "<subscription ID>",
    "aadClientId": "<client ID>",
    "aadClientSecret": "<client secret>",
    "resourceGroup": "<VM resource group>",
    "location": "<VM location>",
    "subnetName": "default",
    "securityGroupName": "<VM security group>",
    "vnetName": "<VM vnet>",
    "useManagedIdentityExtension": false,
    "useInstanceMetadata": true
}

<tenant ID>, <subscription ID>, <client ID>, and <client secret> correspond to an Azure service principal. If you don't have an Azure service principal already, ask Deep Debroy or Flavio Crisciani for these values (as of May 31, 2018).
Fill in all the <VM *> values with the appropriate values for your VM from the Azure portal.
When you perform the UCP install command above, add the --cloud-provider and --pod-cidr flags like this:
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock \
    dockereng/ucp:$UCP_VERSION \
    install --swarm-port 3376 --force-insecure-tcp --admin-password orcaorcaorca --image-version dev: --host-address $NODE_IP --san $NODE_PUBLIC_IP --cloud-provider azure --pod-cidr <VM subnet CIDR>

<VM subnet CIDR> should be the CIDR for your VM's subnet in Azure. It should be of the form 172.16.3.0/24. You can find it by clicking on your VM's "Virtual network/subnet" section in the Azure dashboard and grabbing the "Address space" value.
After installing UCP, create a docker4x/azure-nic-ips service according to the instructions here:
https://docs.docker.com/ee/ucp/admin/install/install-on-azure/#configure-ip-pools-for-azure-stand-alone-vms
Create the azure_ucp_admin.toml secret with the same values that are in the azure.json file.
Windows setup

Make sure that your Linux node has Docker version 17.06.2-ee-8 or above.
These instructions are partially based on the ones in https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows#cluster-subnet-def, although note that that link has a lot of extraneous instructions that won't be necessary for this demo.
Run the below commands in Powershell as an administrator:
$wc = New-Object net.webclient
$wc.Downloadfile("https://dl.k8s.io/v1.11.0-alpha.1/kubernetes-node-windows-amd64.tar.gz", "C:\kubelet.tar.gz")

Install-Module -Force -Name 7Zip4Powershell

function Expand-Tar($tarFile, $dest) {

    $pathToModule = ".\7Zip4Powershell\1.8.0\7Zip4PowerShell.psd1"

    if (-not (Get-Command Expand-7Zip -ErrorAction Ignore)) {
        Import-Module $pathToModule
    }

    Expand-7Zip $tarFile $dest
}

Expand-Tar "C:\kubelet.tar.gz" "C:\kubelet"
Expand-Tar "C:\kubelet\kubelet.tar" "C:\kubelet\kubelet"

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$wc2 = New-Object System.net.webclient
$wc2.Downloadfile("https://github.com/Microsoft/SDN/archive/master.zip", "C:\master.zip")
cd C:\
Expand-Archive master.zip -DestinationPath master
mkdir C:/k/
mv master/SDN-master/Kubernetes/windows/* C:/k/
rm -recurse -force master,master.zip

Set-MpPreference -DisableRealtimeMonitoring $true
docker pull microsoft/nanoserver:1709
docker tag microsoft/nanoserver:1709 microsoft/nanoserver:latest
docker pull microsoft/windowsservercore:1709
docker tag microsoft/windowsservercore:1709 microsoft/windowsservercore:latest
cd C:/k/
docker build -t kubeletwin/pause .
Now you need to manually copy over the UCP certs to the Windows node.
On the Linux manager, run:
docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/ca.pem
Write the output to C:\kubecerts\ca.pem on the Windows node. Do the same for cert.pem and key.pem, writing them to C:\kubecerts\cert.pem and C:\kubecerts\key.pem, respectively.
docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/cert.pem
docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/key.pem
Write the below file to C:\kubecerts\kubelet.conf on the Windows node. Note that you need to replace <Linux public IP address> with the public IP address of the Linux manager:
apiVersion: v1
kind: Config
clusters:
- name: ucp
  cluster:
    certificate-authority: C:\kubecerts\ca.pem
    server: https://<Linux IP address>:6444
users:
- name: kubelet
  user:
    client-certificate: C:\kubecerts\cert.pem
    client-key: C:\kubecerts\key.pem
contexts:
- context:
    cluster: ucp
    user: kubelet
  name: kubelet-context
current-context: kubelet-context

Now start up the kubelet on the Windows node. Note that you need to replace <Windows private IP address> with the private IP of the Windows node:
cd C:\kubelet\kubelet\kubernetes\node\bin
$Env:DOCKER_API_VERSION = "1.30"
$Env:NODE_IP = "<Windows private IP address>"
.\kubelet.exe `
    --allow-privileged=true `
    --anonymous-auth=false `
    --authentication-token-webhook `
    --authorization-mode=Webhook `
    --cadvisor-port=0 `
    --client-ca-file=C:\kubecerts\ca.pem `
    --cluster-dns=10.96.0.10 `
    --cluster-domain=cluster.local `
    --network-plugin=cni `
    --fail-swap-on=false `
    --healthz-port=0 `
    --iptables-drop-bit=31 `
    --iptables-masquerade-bit=30 `
    --kubeconfig=C:\kubecerts\kubelet.conf `
    --node-ip=$Env:NODE_IP `
    --read-only-port=0 `
    --register-node `
    --tls-cert-file=C:\kubecerts\cert.pem `
    --tls-private-key-file=C:\kubecerts\key.pem `
    --network-plugin=cni `
    --cni-bin-dir="c:\k\cni" `
    --cni-conf-dir="c:\k\cni\config" `
    --pod-infra-container-image=kubeletwin/pause `
    --hairpin-mode=promiscuous-bridge `
    --image-pull-progress-deadline=20m `
    --cgroups-per-qos=false `
    --enforce-node-allocatable="" `
    --resolv-conf=""
Now go to the UCP web UI by visiting https://<Linux public IP>:443 and log into the UCP UI with username admin and password orcaorcaorca (or whatever you entered in the install command when you installed UCP on the Linux node). Download a UCP client bundle to your local machine, unzip it, and run source env.sh.
Now deploy this pod by writing the below file to winpod.yml on your local machine and running kubectl apply -f winpod.yml:
apiVersion: v1
kind: Pod
metadata:
  name: testpod
spec:
  nodeSelector:
    beta.kubernetes.io/os: windows
  containers:
  - name: testpod
    image: microsoft/windowsservercore:1709
    command: ["powershell", "-c", "start-sleep 600"]

Now run kubectl get pods and observe that the Windows pod is running on the Windows node.