maksim-paskal

#!/bin/sh
docker rm -f $(docker ps -qa)
docker rmi -f $(docker images -q)
docker volume rm $(docker volume ls -q)
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
cleanupdirs="/etc/ceph /etc/cni /etc/kubernetes /opt/cni /opt/rke /run/secrets/kubernetes.io /run/calico /run/flannel /var/lib/calico /var/lib/etcd /var/lib/cni /var/lib/kubelet /var/lib/rancher /var/log/containers /var/log/pods /var/run/calico"
for dir in $cleanupdirs; do
  echo "Removing $dir"
  rm -rf $dir
done

maksim-paskal

# TOP CPU Usage pods

sort_desc(sum(rate(container_cpu_usage_seconds_total{image!="",container_name!="POD"}[5m])) by (pod,namespace,container_name,kubernetes_io_hostname))

maksim-paskal

kubectl delete pods --field-selector=status.phase=Failed -A
kubectl delete pods --field-selector=status.phase=Evicted -A
kubectl delete pods --field-selector=status.phase=Succeeded -A

maksim-paskal

apiVersion: apps/v1
kind: Deployment
metadata:
  name: maintance-pod
spec:
  selector:
    matchLabels:
      app: maintance-pod
  replicas: 1
  template:

maksim-paskal

Class SRE Implements DevOps

Site Reliability Engineering (SRE) and DevOps are two ideas that have different origins but the same underlying objectives.

DevOps advocates for certain practices that increase success and productivity in a team building and running software. These include:

1. Reduce organization silos
2. Accept failure as normal
3. Implement gradual change
4. Leverage automation and tooling

maksim-paskal

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.threshold_enabled": true,
    "cluster.routing.allocation.disk.watermark.low": "93%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.info.update.interval": "5m" 
  }
}

maksim-paskal

Use dns cache in kubernetes

Install

https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/

to retrieve hostIP from pod add env to daemonset - dig kubernetes-node - will return hostIP of current pod

env:
- name: POD_HOST_IP
  valueFrom:

maksim-paskal

http_filters:
- name: envoy.lua
  typed_config:
    "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
    inline_code: |
      function stringSplit(inputstr, sep)
        if sep == nil then
          sep = "%s"
        end
        local t={}

maksim-paskal

Simple envoy configuration with basic authentication and without authorization service

Sometime you need scrape prometheus metrics from external envoy that deploy not to kubernetes environment

You can use iptable or other stuff on external server to allow only trusted IP for scraping metrics - but for dynamic infrastructure some time it's hard to support it.

Envoy can expose this metrics more elegant style - using basic auth

Simple envoy.yaml

maksim-paskal

Simple bash script to change AMI in AWS EKS cluster

Sometime you need some not standart AMI in kubernetes cluster, for example you want to preload some docker images in kubernetes node

1. Fork https://github.com/awslabs/amazon-eks-ami

2. Make some changes in https://github.com/awslabs/amazon-eks-ami/blob/v20210208/scripts/install-worker.sh - for example

sudo systemctl start docker