GKE kubernetes upgrade

README.md

Node Management in GKE

This session will walk through (auto)upgrades and (auto)repairs as well as best practices on how to set up clusters and node management.

From https://www.youtube.com/watch?v=2HmDZXbfA80

Speaker(s): Kate Fitzpatrick, Kobi Magnezi, Justin Watts

Redundancy

Think N + 1

Use podAntiAffinity

Protection

Use podDisruptionBudget

Probe

Use Readiness probe (when is the service ready to receive traffic)

readinessProbe:
  exec:
    command:
    - sh
    - -c
    - "redis-cli -h $(hostname) ping"
  initialDelaySeconds: 15
  timeoutSeconds: 5

Use liveness probe (when is the service healthy enough)

livenessProve:
  exec:
    command:
    - sh
    - -c
    - "redis-cli -h $(hostname) ping"
  initialDelaySeconds: 20
  periodSeconds: 3

BIG NO-NO

Pod distruption budget

maxUnavailable = 0

Liveness probe

fails under load (=> things become worse quicker)

Termination grace period

0 or hours

break_cluster.sh

!#/bin/sh

set -x

nodes=$(kubectl get nodes -o jsonpath='{range.items[*]}{.metadata.name} ')
for node in $nodes
do
  drain="kubectl drain $node --ignore-daemonsets &"
  eval $drain
done

podAntiAffinity.yaml

spec:
affinity:
  podAntiaffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - redis-app
      topologyKey: kubernetes.io/hostname

PodDisruptionBudget.yaml

kind: PodDisruptionBudget
metadata:
  name: my-app
spec: 
  maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

undo.sh

!#/bin/sh

nodes=$(kubectl get nodes -o jsonpath='{range.items[*]}{.metadata.name} ')
for node in $nodes
do
  drain="kubectl uncordon $node &"
  eval $drain
done