The below instructions describe how to set up a cluster with one Linux manager and one Windows worker such that you can schedule Kubernetes pods with Windows images on the Windows worker. I did this on Azure, but there's nothing Azure-specific in these instructions.
These instructions are for a Windows node running 1709. I have not tested this with Windows 1803, but it will probably work; however, you need to change all the images that use :1709
to :1803
or similar.
On Azure, I recommend VMs of size D4s v3
or larger (4 CPUs, 16GB memory)
Make sure that your Linux node has Docker version 17.06.2-ee-8 or above.
export NODE_IP=<Linux node private IP address>
export NODE_PUBLIC_IP=<Linux node public IP address>
# This version of UCP is a special build with Kube 1.11.0-beta.0 and a tweak to
# the node inventory logic so that it can handle Windows Kube nodes
export UCP_VERSION=3.1.0-5a4bc69
docker login
docker run --rm dockereng/ucp:$UCP_VERSION images --list --image-version dev: | xargs -n 1 docker pull
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock \
dockereng/ucp:$UCP_VERSION \
install --swarm-port 3376 --force-insecure-tcp --admin-password orcaorcaorca --image-version dev: --host-address $NODE_IP --san $NODE_PUBLIC_IP
If you're on Azure, the above instructions will allow you to run Windows pods, but if you want networking to work, you need to make sure UCP gets deployed on the Linux manager with Calico set up in Azure mode.
Write a file called /etc/kubernetes/azure.json
on your Linux manager with the following contents:
{
"tenantId": "<tenant ID>",
"subscriptionId": "<subscription ID>",
"aadClientId": "<client ID>",
"aadClientSecret": "<client secret>",
"resourceGroup": "<VM resource group>",
"location": "<VM location>",
"subnetName": "default",
"securityGroupName": "<VM security group>",
"vnetName": "<VM vnet>",
"useManagedIdentityExtension": false,
"useInstanceMetadata": true
}
<tenant ID>
, <subscription ID>
, <client ID>
, and <client secret>
correspond to an Azure service principal. If you don't have an Azure service principal already, ask Deep Debroy or Flavio Crisciani for these values (as of May 31, 2018).
Fill in all the <VM *>
values with the appropriate values for your VM from the Azure portal.
When you perform the UCP install command above, add the --cloud-provider
and --pod-cidr
flags like this:
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock \
dockereng/ucp:$UCP_VERSION \
install --swarm-port 3376 --force-insecure-tcp --admin-password orcaorcaorca --image-version dev: --host-address $NODE_IP --san $NODE_PUBLIC_IP --cloud-provider azure --pod-cidr <VM subnet CIDR>
<VM subnet CIDR>
should be the CIDR for your VM's subnet in Azure. It should be of the form 172.16.3.0/24
. You can find it by clicking on your VM's "Virtual network/subnet" section in the Azure dashboard and grabbing the "Address space" value.
After installing UCP, create a docker4x/azure-nic-ips
service according to the instructions here:
Create the azure_ucp_admin.toml
secret with the same values that are in the azure.json
file.
Make sure that your Linux node has Docker version 17.06.2-ee-8 or above.
These instructions are partially based on the ones in https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows#cluster-subnet-def, although note that that link has a lot of extraneous instructions that won't be necessary for this demo.
Run the below commands in Powershell as an administrator:
$wc = New-Object net.webclient
$wc.Downloadfile("https://dl.k8s.io/v1.11.0-alpha.1/kubernetes-node-windows-amd64.tar.gz", "C:\kubelet.tar.gz")
Install-Module -Force -Name 7Zip4Powershell
function Expand-Tar($tarFile, $dest) {
$pathToModule = ".\7Zip4Powershell\1.8.0\7Zip4PowerShell.psd1"
if (-not (Get-Command Expand-7Zip -ErrorAction Ignore)) {
Import-Module $pathToModule
}
Expand-7Zip $tarFile $dest
}
Expand-Tar "C:\kubelet.tar.gz" "C:\kubelet"
Expand-Tar "C:\kubelet\kubelet.tar" "C:\kubelet\kubelet"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$wc2 = New-Object System.net.webclient
$wc2.Downloadfile("https://github.com/Microsoft/SDN/archive/master.zip", "C:\master.zip")
cd C:\
Expand-Archive master.zip -DestinationPath master
mkdir C:/k/
mv master/SDN-master/Kubernetes/windows/* C:/k/
rm -recurse -force master,master.zip
Set-MpPreference -DisableRealtimeMonitoring $true
docker pull microsoft/nanoserver:1709
docker tag microsoft/nanoserver:1709 microsoft/nanoserver:latest
docker pull microsoft/windowsservercore:1709
docker tag microsoft/windowsservercore:1709 microsoft/windowsservercore:latest
cd C:/k/
docker build -t kubeletwin/pause .
Now you need to manually copy over the UCP certs to the Windows node.
On the Linux manager, run:
docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/ca.pem
Write the output to C:\kubecerts\ca.pem
on the Windows node. Do the same for cert.pem
and key.pem
, writing them to C:\kubecerts\cert.pem
and C:\kubecerts\key.pem
, respectively.
docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/cert.pem
docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/key.pem
Write the below file to C:\kubecerts\kubelet.conf
on the Windows node. Note that you need to replace <Linux public IP address>
with the public IP address of the Linux manager:
apiVersion: v1
kind: Config
clusters:
- name: ucp
cluster:
certificate-authority: C:\kubecerts\ca.pem
server: https://<Linux IP address>:6444
users:
- name: kubelet
user:
client-certificate: C:\kubecerts\cert.pem
client-key: C:\kubecerts\key.pem
contexts:
- context:
cluster: ucp
user: kubelet
name: kubelet-context
current-context: kubelet-context
Now start up the kubelet on the Windows node. Note that you need to replace <Windows private IP address>
with the private IP of the Windows node:
cd C:\kubelet\kubelet\kubernetes\node\bin
$Env:DOCKER_API_VERSION = "1.30"
$Env:NODE_IP = "<Windows private IP address>"
.\kubelet.exe `
--allow-privileged=true `
--anonymous-auth=false `
--authentication-token-webhook `
--authorization-mode=Webhook `
--cadvisor-port=0 `
--client-ca-file=C:\kubecerts\ca.pem `
--cluster-dns=10.96.0.10 `
--cluster-domain=cluster.local `
--network-plugin=cni `
--fail-swap-on=false `
--healthz-port=0 `
--iptables-drop-bit=31 `
--iptables-masquerade-bit=30 `
--kubeconfig=C:\kubecerts\kubelet.conf `
--node-ip=$Env:NODE_IP `
--read-only-port=0 `
--register-node `
--tls-cert-file=C:\kubecerts\cert.pem `
--tls-private-key-file=C:\kubecerts\key.pem `
--network-plugin=cni `
--cni-bin-dir="c:\k\cni" `
--cni-conf-dir="c:\k\cni\config" `
--pod-infra-container-image=kubeletwin/pause `
--hairpin-mode=promiscuous-bridge `
--image-pull-progress-deadline=20m `
--cgroups-per-qos=false `
--enforce-node-allocatable="" `
--resolv-conf=""
Now go to the UCP web UI by visiting https://<Linux public IP>:443
and log into the UCP UI with username admin
and password orcaorcaorca
(or whatever you entered in the install
command when you installed UCP on the Linux node). Download a UCP client bundle to your local machine, unzip it, and run source env.sh
.
Now deploy this pod by writing the below file to winpod.yml
on your local machine and running kubectl apply -f winpod.yml
:
apiVersion: v1
kind: Pod
metadata:
name: testpod
spec:
nodeSelector:
beta.kubernetes.io/os: windows
containers:
- name: testpod
image: microsoft/windowsservercore:1709
command: ["powershell", "-c", "start-sleep 600"]
Now run kubectl get pods
and observe that the Windows pod is running on the Windows node.
Using https://github.com/docker/testkit/pull/948