Steps for setting up a Kubernetes 1.9 cluster on Azure with acs-engine

.gitignore

# vim giblets
*.swp
*.swo
*~
# if you run the sample step by step, these will contain secrets
*secret.json
# output from acs-engine
_output/
translations/

README.md

Setting up a manual cluster with ACS-engine

Starting point: https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/windows.md

I did all the steps here using Ubuntu from the Windows Store.

Get latest build - see https://github.com/Azure/acs-engine/releases/

curl -L https://github.com/Azure/acs-engine/releases/download/v0.15.2/acs-engine-v0.15.2-linux-amd64.tar.gz | tar xvfz -
sudo mv acs-engine-v0.15.2-linux-amd64/acs-engine /usr/local/bin
rm -rf acs-engine-v0.15.2-linux-amd64/

Get SSH public key

ls ~/.ssh/id_rsa.pub

Generate Service Principal

export SUBSCRIPTION_ID=`az account show | jq .id | sed "s/\"//"g`
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/${SUBSCRIPTION_ID}" > secret.json

Get template

curl -o kubernetes.json https://raw.githubusercontent.com/Azure/acs-engine/master/examples/windows/kubernetes.json

Now, let's fill in the hard stuff automatically. This will add some data to the template and create kubernetes-secret.json. Don't share that file!

• Your SSH public key from .ssh/id_rsa.pub
• The service account credentials stored in secret.json

export APPID=`jq '.appId' secret.json`
export APPPASS=`jq '.password' secret.json`
cat kubernetes.json | jq ".properties.linuxProfile.ssh.publicKeys[0].keyData = \"`cat ~/.ssh/id_rsa.pub`\"" | jq ".properties.servicePrincipalProfile.clientId = $APPID" | jq ".properties.servicePrincipalProfile.secret = $APPPASS" > kubernetes-secret.json

Now, update it

• Change username & passwords
• Add a dnsprefix
• Choose VM size (hint: az vm list-sizes --location westus2 -otable)
• Add orchestratorVersion for the version you want

    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.1"
    },

Generate the Azure Resource Manager template

$ acs-engine generate ./kubernetes-secret.json

INFO[0000] Generating assets into _output/plang0308...

This will generate a directory named with the dnsPrefix in the template file - plang0308 in this case

Create a resource group

export rgname=plang0308rg
export name=plang0308
az group create --name $rgname --location westus2

Deploy it

az group deployment create --name $name --resource-group $rgname --template-file ./_output/$name/azuredeploy.json  --parameters ./_output/$name/azuredeploy.parameters.json

After several minutes, it will return a whole bunch of JSON. Look for masterFQDN.

      "masterFQDN": {
        "type": "String",
        "value": "plang0308.westus2.cloudapp.azure.com"
      },

SSH to that FQDN with the username you put in the linuxProfile section of kubernetes-secret.json :

ssh patrick@plang0308.westus2.cloudapp.azure.com

Now, you can verify the cluster is up with kubectl get node

If you want to manage this from your local machine, go ahead and disconnect that SSH session. Then copy the kube config to your local machine in a new directory, and set KUBECONFIG

mkdir ~/.$name
export KUBECONFIG=~/.$name/config
ssh patrick@plang0308.westus2.cloudapp.azure.com 'cat ~/.kube/config' >> $KUBECONFIG
kubectl get node

Enabling Hyper=V

With Hyper-V isolation, you can run containers using the same, or older Windows Server versions. For example, you can run a container built with the Windows Server 2016 base image such as microsoft/iis:windowsservercore-ltsc2016 on a Windows Server version 1709 node. It could also run alongside another pod with containers built against Windows Server version 1709 such as microsoft/iis:windowsservercore-1709.

Steps needed:

• Prerequisite: Ensure the Hyper-V role is enabled, and the processor (and virtualization solution) has virtualization support enabled. April 2018 update (KB4093112 / version 10.10.16299.371)
• Add flag to kubectl.exe startup options
• Add annotation to deployment config

Opening feature-gates

Here's an example of the feature gate that's needed. The other options should be kept as-is in the kubelet startup script.

c:\k\kubelet.exe --pod-infra-container-image=kubletwin/pause ... --feature-gates HyperVContainer=true

Note: Check this acs-engine issue - once resolved the next steps will be much easier.

This is easy to do before you create a cluster using acs-engine by:

1. Running acs-engine generate ... as above

• Be sure to use Dv3/Ev3 series VMs

2. Don't run az group deploy yet, wait
3. Modify output/<dnsprefix>/azuredeploy.json to add --feature-gates ...

Deploying a pod with Hyper-V isolation

In the pod deployment, be sure to add the annotation experimental.windows.kubernetes.io/isolation-type: hyperv, and use a tag to ensure you're picking the intended Windows version if there are multiple available.

(See whoami-2016.yaml and whoami-1709.yaml in this same repo)

TODOs

Cleanup steps - remove deployments like this to avoid failures deleting the group

kubernetes-1803.json

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.1"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "",
      "vmSize": "Standard_D2_v3"
    },
    "agentPoolProfiles": [
      {
        "name": "windowspool2",
        "count": 2,
        "vmSize": "Standard_D2_v3",
        "availabilityProfile": "AvailabilitySet",
        "osType": "Windows",
	"osDiskSizeGB": 127
      }
    ],
    "windowsProfile": {
      "adminUsername": "azureuser",
      "adminPassword": "replacepassword1234$",
      "agentWindowsPublisher": "MicrosoftWindowsServer",
      "agentWindowsOffer": "WindowsServerSemiAnnual",
      "agentWindowsSku": "Datacenter-Core-1803-with-Containers-smalldisk"
    },
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": ""
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    }
  }
}

kubernetes.json

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.1"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "",
      "vmSize": "Standard_D2_v3"
    },
    "agentPoolProfiles": [
      {
        "name": "windowspool2",
        "count": 2,
        "vmSize": "Standard_D2_v2",
        "availabilityProfile": "AvailabilitySet",
        "osType": "Windows"
      }
    ],
    "windowsProfile": {
      "adminUsername": "azureuser",
      "adminPassword": "replacepassword1234$"
    },
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": ""
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    }
  }
}

whoami-1709.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-1709
  labels:
    app: whoami-1709
spec:
  replicas: 1
  template:
    metadata:
      name: whoami-1709
      labels:
        app: whoami
      annotations:
        experimental.windows.kubernetes.io/isolation-type: hyperv
    spec:
      containers:
      - name: whoami-1709
        image: stefanscherer/whoami:windows-amd64-1.5.4-1709
        ports:
        - containerPort: 8080
      nodeSelector:
        beta.kubernetes.io/os: windows
  selector:
    matchLabels:
      app: whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoami-1709-svc
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: whoami-1709

whoami-1803.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-1803
  labels:
    app: whoami-1803
spec:
  replicas: 1
  template:
    metadata:
      name: whoami-1803
      labels:
        app: whoami
      annotations:
        experimental.windows.kubernetes.io/isolation-type: hyperv
    spec:
      containers:
      - name: whoami-1803
        image: stefanscherer/whoami:windows-amd64-1.7.0-1803
        ports:
        - containerPort: 8080
      nodeSelector:
        beta.kubernetes.io/os: windows
  selector:
    matchLabels:
      app: whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoami-1803-svc
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 8080
  selector:
    app: whoami-1803

whoami-2016.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-2016
  labels:
    app: whoami-2016
spec:
  replicas: 1
  template:
    metadata:
      name: whoami-2016
      labels:
        app: whoami
      annotations:
        experimental.windows.kubernetes.io/isolation-type: hyperv
    spec:
      containers:
      - name: whoami-2016
        image: stefanscherer/whoami:windows-amd64-1.5.4
        ports:
        - containerPort: 8080
      nodeSelector:
        beta.kubernetes.io/os: windows
  selector:
    matchLabels:
      app: whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoami-2016-svc
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: whoami-2016