Skip to content

Instantly share code, notes, and snippets.

@pivotaljohn

pivotaljohn/k8s-notes.md

Last active October 21, 2019 14:47
Show Gist options
  • Save pivotaljohn/d49752e50df7fec3a942eb655e743bc2 to your computer and use it in GitHub Desktop.
Save pivotaljohn/d49752e50df7fec3a942eb655e743bc2 to your computer and use it in GitHub Desktop.
Download ZIP
Kubernetes Notes
Raw
k8s-notes.md

Architecture

Master

  • this is the control plane of the cluster
  • etcd is the default backing store; others can be configured (q: what are the other viable options for a cluster backing store?)

Uncategorized Notes

  • "Resource" -- the thing (pod, replicaset, deployment, ...)
    • "Controller" -- a process that brings the actual state to the desired state
      • "Operators" -- a scripted use of controllers (?)

etcd

  • implementation of the RAFT algorithm. This is a consensus-based distributed DB.

Questions

Raw
k8s-the-hard-way-notes.md

Summary

  1. Installed command-line tools
    • gcloud -- GCP CLI
    • cfssl -- creating PKI stuffs
  2. Created a VPC on GCP.
  3. Created VM instances for what will be Control Plane nodes and Worker Nodes.
  4. Created a slew of certificates (all signed by a self-signed CA) to secure communications between components.
  5. Created kubeconfig for all of the API Service clients.
  6. Established a shared secret for encryption of data at rest.
  7. Installed etcd (a clustered key-value DB) on the controller nodes (required configuration).
  8. Attempted to install the control plane software (apiserver, controller-manager, scheduler,...) but all three control plane VMs jammed-up processing etcd traffic. I chose undersized instances and they could not, in fact, handle even idle loads.

... so I started over from "Create VM instances" ...

... 1.

Details

Prep

  • installed gcloud (via brew cask install google-cloud-sdk) so that we can do IaaS work from the command-line (and likely so that other tooling has programmatic access to the infrastructure?)
  • installed cfssl (via brew install cfssl in order to generate certificates (why not use openssl?) (see also https://github.com/cloudflare/cfssl)

Side trips

(stuff I did that wasn't in the tutorial)

fix the kubectl install

$ brew reinstall kubectl
==> Reinstalling kubernetes-cli
==> Downloading https://homebrew.bintray.com/bottles/kubernetes-cli-1.16.1.mojave.bottle.tar.gz
Already downloaded: /Users/jryan/Library/Caches/Homebrew/downloads/65c8ffb92651c32e96cc7eb2e36e58ea97b3a1148a579c3fb9a287df498ea59e--kubernetes-cli-1.16.1.mojave.bottle.tar.gz
==> Pouring kubernetes-cli-1.16.1.mojave.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/kubectl
Target /usr/local/bin/kubectl
already exists. You may want to remove it:
  rm '/usr/local/bin/kubectl'

...

Possible conflicting files are:
/usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl
...

So, I:

$ $ brew link --overwrite kubernetes-cli
Linking /usr/local/Cellar/kubernetes-cli/1.16.1... 227 symlinks created

Paving (manual)

Create VPC

$ gcloud compute networks create k8s-the-hard-way --subnet-mode custom
Created [https://www.googleapis.com/compute/v1/projects/cf-sandbox-release-engineering/global/networks/k8s-the-hard-way].
NAME              SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
k8s-the-hard-way  CUSTOM       REGIONAL

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

  $ gcloud compute firewall-rules create <FIREWALL_NAME> --network k8s-the-hard-way --allow tcp,udp,icmp --source-ranges <IP_RANGE>
  $ gcloud compute firewall-rules create <FIREWALL_NAME> --network k8s-the-hard-way --allow tcp:22,tcp:3389,icmp

Create Subnet within the VPC

$ gcloud compute networks subnets create k8s-cluster-subnet --network k8s-the-hard-way --range 10.240.0.0/24
Created [https://www.googleapis.com/compute/v1/projects/cf-sandbox-release-engineering/regions/us-central1/subnetworks/k8s-cluster-subnet].
NAME                REGION       NETWORK           RANGE
k8s-cluster-subnet  us-central1  k8s-the-hard-way  10.240.0.0/24

Create Firewall Rules

$ gcloud compute firewall-rules create k8s-thw--fwr-allow-internal --allow tcp,udp,icmp --network k8s-the-hard-way --source-ranges 10.240.0.0/24,10.200.0.0/16
Creating firewall...⠛Created [https://www.googleapis.com/compute/v1/projects/cf-sandbox-release-engineering/global/firewalls/k8s-thw--fwr-allow-internal].
Creating firewall...done.
NAME                         NETWORK           DIRECTION  PRIORITY  ALLOW         DENY  DISABLED
k8s-thw--fwr-allow-internal  k8s-the-hard-way  INGRESS    1000      tcp,udp,icmp        False

$ gcloud compute firewall-rules create k8s-thw--fwr-allow-encrypted-external --allow tcp:22,tcp:6443,icmp --network k8s-the-hard-way --source-ranges 0.0.0.0/0
Creating firewall...⠏Created [https://www.googleapis.com/compute/v1/projects/cf-sandbox-release-engineering/global/firewalls/k8s-thw--fwr-allow-encrypted-external].
Creating firewall...done.
NAME                                   NETWORK           DIRECTION  PRIORITY  ALLOW                 DENY  DISABLED
k8s-thw--fwr-allow-encrypted-external  k8s-the-hard-way  INGRESS    1000      tcp:22,tcp:6443,icmp        False

Allocate an IP Address (for later use for kubernetes service)

$ gcloud compute addresses create k8s-the-hard-way--ip --region $(gcloud config get-value compute/region)
Created [https://www.googleapis.com/compute/v1/projects/cf-sandbox-release-engineering/regions/us-central1/addresses/k8s-the-hard-way--ip].

Create VMs to House the Control Plane

for i in 0 1 2; do
  gcloud compute instances create k8s-thw--controller-${i} \
    --async \
    --boot-disk-size 200GB \
    --can-ip-forward \
    --image-family ubuntu-1804-lts \
    --image-project ubuntu-os-cloud \
    --machine-type f1-micro \
    --private-network-ip 10.240.0.1${i} \
    --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
    --subnet k8s-cluster-subnet \
    --tags k8s-thw,controller
done

#!/usr/bin/env bash

for i in 0 1 2; do
  gcloud compute instances create k8s-thw--worker-${i} \
    --async \
    --boot-disk-size 200GB \
    --can-ip-forward \
    --image-family ubuntu-1804-lts \
    --image-project ubuntu-os-cloud \
    --machine-type f1-micro \
    --metadata pod-cidr=10.200.${i}.0/24 \
    --private-network-ip 10.240.0.2${i} \
    --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
    --subnet k8s-cluster-subnet \
    --tags k8s-thw,worker
done

Create PKI Bits

Certificate Authority (CA) Certificate

Created configuration and Certificate Signing Request:

ca-config.json

{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "k8s_profile": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}

ca-csr.json

{
  "CN": "KubernetesTheHardWay",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "Pivotal",
      "OU": "Cloud R&D",
      "ST": "California"
    }
  ]
}

Then ran:

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2019/10/13 06:55:43 [INFO] generating a new CA key and certificate from CSR
2019/10/13 06:55:43 [INFO] generate received request
2019/10/13 06:55:43 [INFO] received CSR
2019/10/13 06:55:43 [INFO] generating key: rsa-2048
2019/10/13 06:55:43 [INFO] encoded CSR
2019/10/13 06:55:43 [INFO] signed certificate with serial number 318904427095986789448913517096609908161556343159

which yielded this CA cert:

$ openssl x509 -in ca.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            37:dc:2b:...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=California, L=Los Angeles, O=Pivotal, OU=Cloud R&D, CN=KubernetesTheHardWay
        Validity
            Not Before: Oct 13 13:51:00 2019 GMT
            Not After : Oct 11 13:51:00 2024 GMT
        Subject: C=US, ST=California, L=Los Angeles, O=Pivotal, OU=Cloud R&D, CN=KubernetesTheHardWay
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c4:5d:...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                E2:73:E8:...
    Signature Algorithm: sha256WithRSAEncryption
         4c:a9:b5:...

and associated private key:

$ cat ca-key.pem
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

"admin" User Certificates

admin-csr.json

{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "system:masters",
      "OU": "k8s The Hard Way",
      "ST": "California"
    }
  ]
}

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s_profile admin-csr.json | cfssljson -bare admin
2019/10/13 07:18:53 [INFO] generate received request
2019/10/13 07:18:53 [INFO] received CSR
2019/10/13 07:18:53 [INFO] generating key: rsa-2048
2019/10/13 07:18:53 [INFO] encoded CSR
2019/10/13 07:18:53 [INFO] signed certificate with serial number 152182081464568438529354720500210576012060174994
2019/10/13 07:18:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

kubelet Client Certificates (for Node Authentication)

On each Node, it's the kubelet process that communicates with the API server. The API provides for what is called "Node Authentication": where if the process provides a CA-signed certificate in which the "CN" is in the format: system:node:<NODE-HOSTNAME> (where NODE-HOSTNAME matches perfectly the hostname of the node) and is in the system:nodes group (i.e. has the name "O": "system:nodes")... the API will consider calls from that node as authenticated.

#!/usr/bin/env bash

WORKERS=$(gcloud compute instances list --filter="tags.items=(worker)" --format=json | jq -r .[].name)

for worker in ${WORKERS}; do
  echo "Generating Certificate Signing Request (CSR) for node ${worker}..."
  cat >${worker}-csr.json <<EOF
{
  "CN": "system:node:${worker}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "system:nodes",
      "OU": "k8s The Hard Way",
      "ST": "California"
    }
  ]
}
EOF
  WORKER_DATA=$(gcloud compute instances describe ${worker} --format=json)
  EXTERNAL_IP=$(jq -r .networkInterfaces[0].accessConfigs[0].natIP <<< "${WORKER_DATA}")
  INTERNAL_IP=$(jq -r .networkInterfaces[0].networkIP <<< "${WORKER_DATA}")

  cfssl gencert \
    -ca=ca.pem \
    -ca-key=ca-key.pem \
    -config=ca-config.json \
    -hostname=${worker},${EXTERNAL_IP},${INTERNAL_IP} \
    -profile=k8s_profile \
    ${worker}-csr.json | cfssljson -bare ${worker}
done

which when ran:

$ ./generate-node-authorization-certs-for-all-workers.sh
Generating Certificate Signing Request (CSR) for node k8s-thw--worker-0...
2019/10/13 08:31:45 [INFO] generate received request
2019/10/13 08:31:45 [INFO] received CSR
2019/10/13 08:31:45 [INFO] generating key: rsa-2048
2019/10/13 08:31:46 [INFO] encoded CSR
2019/10/13 08:31:46 [INFO] signed certificate with serial number 574951830346648259700591950215060899705158596141
Generating Certificate Signing Request (CSR) for node k8s-thw--worker-1...
2019/10/13 08:31:47 [INFO] generate received request
2019/10/13 08:31:47 [INFO] received CSR
2019/10/13 08:31:47 [INFO] generating key: rsa-2048
2019/10/13 08:31:47 [INFO] encoded CSR
2019/10/13 08:31:47 [INFO] signed certificate with serial number 653063770009311125900902559472788059087129362860
Generating Certificate Signing Request (CSR) for node k8s-thw--worker-2...
2019/10/13 08:31:48 [INFO] generate received request
2019/10/13 08:31:48 [INFO] received CSR
2019/10/13 08:31:48 [INFO] generating key: rsa-2048
2019/10/13 08:31:48 [INFO] encoded CSR
2019/10/13 08:31:48 [INFO] signed certificate with serial number 677094462277342270594691084181088531599883348006

which generated these certificates:

$ openssl x509 -in k8s-thw--worker-0.pem
Certificate:
    ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=California, L=Los Angeles, O=Pivotal, OU=Cloud R&D, CN=KubernetesTheHardWay
        ...
        Subject: C=US, ST=California, L=Los Angeles, O=system:nodes, OU=k8s The Hard Way, CN=system:node:k8s-thw--worker-0
        ...
    X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            ...

            X509v3 Subject Alternative Name: 
                DNS:k8s-thw--worker-0, IP Address:34.69.229.83, IP Address:10.240.0.20
    ...

$ openssl x509 -in k8s-thw--worker-1.pem
Certificate:
    ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=California, L=Los Angeles, O=Pivotal, OU=Cloud R&D, CN=KubernetesTheHardWay
        ...
        Subject: C=US, ST=California, L=Los Angeles, O=system:nodes, OU=k8s The Hard Way, CN=system:node:k8s-thw--worker-1
        ...
    X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            ...

            X509v3 Subject Alternative Name: 
                DNS:k8s-thw--worker-1, IP Address:35.226.92.172, IP Address:10.240.0.21
    ...

$ openssl x509 -in k8s-thw--worker-2.pem
Certificate:
    ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=California, L=Los Angeles, O=Pivotal, OU=Cloud R&D, CN=KubernetesTheHardWay
        ...
        Subject: C=US, ST=California, L=Los Angeles, O=system:nodes, OU=k8s The Hard Way, CN=system:node:k8s-thw--worker-2
        ...
    X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            ...

            X509v3 Subject Alternative Name: 
                DNS:k8s-thw--worker-2, IP Address:35.226.201.174, IP Address:10.240.0.22
    ...

Controller Manager Certificate

kube-controller-manager-csr.json

{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "system:kube-controller-manager",
      "OU": "k8s The Hard Way",
      "ST": "California"
    }
  ]
}

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s_profile kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2019/10/13 10:57:33 [INFO] generate received request
2019/10/13 10:57:33 [INFO] received CSR
2019/10/13 10:57:33 [INFO] generating key: rsa-2048
2019/10/13 10:57:33 [INFO] encoded CSR
2019/10/13 10:57:33 [INFO] signed certificate with serial number 519343206361199162178115795663679563707198112632
2019/10/13 10:57:33 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

kube-proxy Client Certificate

kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "system:node-proxier",
      "OU": "k8s The Hard Way",
      "ST": "California"
    }
  ]
}

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s_profile kube-proxy-csr.json | cfssljson -bare kube-proxy
2019/10/13 15:54:22 [INFO] generate received request
2019/10/13 15:54:22 [INFO] received CSR
2019/10/13 15:54:22 [INFO] generating key: rsa-2048
2019/10/13 15:54:22 [INFO] encoded CSR
2019/10/13 15:54:22 [INFO] signed certificate with serial number 85244603884961202522822863434789970908800025971
2019/10/13 15:54:22 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

kube scheduler Certificate

kube-scheduler-csr.json

{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "system:kube-scheduler",
      "OU": "k8s The Hard Way",
      "ST": "California"
    }
  ]
}

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s_profile kube-scheduler-csr.json | cfssljson -bare kube-scheduler
2019/10/13 16:00:01 [INFO] generate received request
2019/10/13 16:00:01 [INFO] received CSR
2019/10/13 16:00:01 [INFO] generating key: rsa-2048
2019/10/13 16:00:01 [INFO] encoded CSR
2019/10/13 16:00:01 [INFO] signed certificate with serial number 306089489826831215456760835511263900668702419959
2019/10/13 16:00:01 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

API Server Certificate

#!/usr/bin/env bash

K8S_EXTERNAL_ADDRESS=$(gcloud compute addresses describe k8s-the-hard-way--ip --region=$(gcloud config get-value compute/region) --format=json)
K8S_PUBLIC_IP=$(jq -r .address <<< ${K8S_EXTERNAL_ADDRESS})

INTERNAL_CLUSTER_SVCS_IP=10.32.0.1
CONTROLLER_IPS=10.240.0.10,10.240.0.11,10.240.0.12
K8S_HOSTNAMES=kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.svc.cluster.local

cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "Kubernetes",
      "OU": "k8s The Hard Way",
      "ST": "California"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=k8s_profile \
  -hostname=${INTERNAL_CLUSTER_SVCS_IP},${CONTROLLER_IPS},${K8S_PUBLIC_IP},127.0.0.1,${K8S_HOSTNAMES} \
  kubernetes-csr.json | cfssljson -bare kubernetes

Service Account Certificates

service-account-csr.json

{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Los Angeles",
      "O": "Kubernetes",
      "OU": "k8s The Hard Way",
      "ST": "California"
    }
  ]
}

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s_profile service-account-csr.json | cfssljson -bare service-account
2019/10/13 16:43:48 [INFO] generate received request
2019/10/13 16:43:48 [INFO] received CSR
2019/10/13 16:43:48 [INFO] generating key: rsa-2048
2019/10/13 16:43:48 [INFO] encoded CSR
2019/10/13 16:43:48 [INFO] signed certificate with serial number 140630582793387107716676495732507467442102422418
2019/10/13 16:43:48 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

Distribute PKI Bits

distribute-certificates.sh

#!/usr/bin/env bash

WORKERS=$(gcloud compute instances list --filter="tags.items=(worker)" --format=json | jq -r .[].name)

echo "Distributing with CA cert, node cert, node private key to worker nodes:"
for worker in ${WORKERS}; do
  echo "- ${worker}"
  gcloud compute scp ca.pem ${worker}-key.pem ${worker}.pem ${worker}:~/
done
echo "done."


CONTROLLERS=$(gcloud compute instances list --filter="tags.items=(controller)" --format=json | jq -r .[].name)
echo "Distributing CA cert & private key, 'kubernetes' service cert & private key, 'service-account' cert & private key:"
for controller in ${CONTROLLERS}; do
  echo "- ${controller}"
  gcloud compute scp ca.pem ca-key.pem kubernetes.pem kubernetes-key.pem service-account.pem service-account-key.pem ${controller}:~/
done
Raw
kubectl-scratch.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment