k8s-on-acs.md

prereqs

• need to be a global admin to create service principal accounts when actually spinning up a cluster
• create an SSH key and add to your OS keychain (e.g. ssh-add ...)

Issues you may run into!

• "Insufficient privileges to complete the operation" during cluster creation

Your Azure account needs to be a Global Administrator so you can create service principal access.

• "Private key file is encrypted" when importing kubeconfig

Add your SSH private key to your OS keychain before running az acs kubernetes get-credentials

install Azure CLI and login

$ curl -L https://aka.ms/InstallAzureCli | bash

Login to your Azure account:

az login

set ENV vars

• LOCATION: the Azure data center location
• CLUSTER_NAME: the public DNS name of your cluster, e.g. <DNS_PREFIX>.eastus.cloudapp.azure.com
• SSH_KEYFILE: full path to SSH public key

[seanknox:~/src]$ export LOCATION=eastus CLUSTER_NAME=horse-battery-staple SSH_KEYFILE=~/.ssh/id_rsa.pub

create a RESOURCE_GROUP

[seanknox:~/src]$ az group create --name=$CLUSTER_NAME --location=$LOCATION
{
  "id": "/subscriptions/.../resourceGroups/sean",
  "location": "eastus",
  "managedBy": null,
  "name": "sean",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null
}

create k8s cluster in your RESOURCE_GROUP

[seanknox:~/src]$ az acs create --orchestrator-type=kubernetes --resource-group $CLUSTER_NAME --name=$CLUSTER_NAME --dns-prefix=$CLUSTER_NAME --ssh-key-value=$SSH_KEYFILE
waiting for AAD role to propagate.done
{
  "id": "/subscriptions/.../resourceGroups/sean/providers/Microsoft.Resources/deployments/azurecli1487880091.6632926",
  "name": "azurecli1487880091.6632926",
  "properties": {
    "correlationId": "8f50bdee-801d-4515-a875-23960fd449fa",
    "debugSetting": null,
    "dependencies": [],
    "mode": "Incremental",
    "outputs": null,
    "parameters": {
      "clientSecret": {
        "type": "SecureString"
      }
    },
    "parametersLink": null,
    "providers": [
      {
        "id": null,
        "namespace": "Microsoft.ContainerService",
        "registrationState": null,
        "resourceTypes": [
          {
            "aliases": null,
            "apiVersions": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "containerServices"
          }
        ]
      }
    ],
    "provisioningState": "Succeeded",
    "template": null,
    "templateLink": null,
    "timestamp": "2017-02-23T20:19:31.806842+00:00"
  },
  "resourceGroup": "sean"
}

download the master Kubernetes cluster configuration to the ~/.kube/config file:

[seanknox:~/src]$ az acs kubernetes get-credentials -g $RESOURCE_GROUP -n $CLUSTER_NAME

Your cluster will now be accessible via kubectl:

[seanknox:~/src]$ kubectl cluster-info
Kubernetes master is running at https://horse-battery-staple.eastus.cloudapp.azure.com
Heapster is running at https://horse-battery-staple.eastus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://horse-battery-staple.eastus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://horse-battery-staple.eastus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Upgrade to kubernetes version to v1.7.7 on ACS

Upgrade master

tldr;

1. ssh to master DNS FQDN (Should be $DNS_PREFIX.$LOCATION.cloudapp.azure.com)
2. bump hyperkube version in /etc/systemd/system/kubelet.service and static pod manifests that kubelet runs (which includes apiserver) in /etc/kubernetes/manifests/ and /etc/kubernetes/addons
3. restart daemons

Run this script to upgrade the masters.

$ curl https://gist.githubusercontent.com/seanknox/6eff8703ba7ae25dcf77e1a8e65a633a/raw/32d681ac7988c38e55845d103ce40e9d223875f7/upgrade_acs_k8s_masters.sh | sudo bash

Troubleshoot issues by viewing logs: journalctl -f;

Upgrade agents (aka nodes)

tldr; For each agent node, one at a time:

1. ssh -A to master DNS FQDN
2. drain and cordon agent
3. From the master, ssh to the agent node. The node hostname is the same as the name in kubectl get nodes.
4. bump hyperkube version in /etc/systemd/system/kubelet.service and static pod manifests that kubelet runs (which includes apiserver) in /etc/kubernetes/manifests/ and /etc/kubernetes/addons
5. restart daemons
6. uncordon node

Run this script on each agent to upgrade:

$ curl https://gist.githubusercontent.com/seanknox/6eff8703ba7ae25dcf77e1a8e65a633a/raw/32d681ac7988c38e55845d103ce40e9d223875f7/upgrade_acs_k8s_nodes.sh | sudo bash

Troubleshoot issues by viewing logs: journalctl -f;

If you don't want to curl | bash (understandable), the scripts are here: https://gist.github.com/seanknox/6eff8703ba7ae25dcf77e1a8e65a633a