Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
GKE kubernetes upgrade

Node Management in GKE

This session will walk through (auto)upgrades and (auto)repairs as well as best practices on how to set up clusters and node management.

From https://www.youtube.com/watch?v=2HmDZXbfA80

Speaker(s): Kate Fitzpatrick, Kobi Magnezi, Justin Watts

Redundancy

Think N + 1

Use podAntiAffinity

Protection

Use podDisruptionBudget

Probe

Use Readiness probe (when is the service ready to receive traffic)

readinessProbe:
  exec:
    command:
    - sh
    - -c
    - "redis-cli -h $(hostname) ping"
  initialDelaySeconds: 15
  timeoutSeconds: 5

Use liveness probe (when is the service healthy enough)

livenessProve:
  exec:
    command:
    - sh
    - -c
    - "redis-cli -h $(hostname) ping"
  initialDelaySeconds: 20
  periodSeconds: 3

BIG NO-NO

Pod distruption budget

maxUnavailable = 0

Liveness probe

fails under load (=> things become worse quicker)

Termination grace period

0 or hours

!#/bin/sh
set -x
nodes=$(kubectl get nodes -o jsonpath='{range.items[*]}{.metadata.name} ')
for node in $nodes
do
drain="kubectl drain $node --ignore-daemonsets &"
eval $drain
done
spec:
affinity:
podAntiaffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis-app
topologyKey: kubernetes.io/hostname
kind: PodDisruptionBudget
metadata:
name: my-app
spec:
maxUnavailable: 1
selector:
matchLabels:
app: my-app
!#/bin/sh
nodes=$(kubectl get nodes -o jsonpath='{range.items[*]}{.metadata.name} ')
for node in $nodes
do
drain="kubectl uncordon $node &"
eval $drain
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.