skoenig

Google's SRE Books

Some notes on and excerpts from Google's SRE trifecta. These are just my jottings, so they may be incomplete, not detailed and structured differently from the books.

1. the original: https://sre.google/sre-book/table-of-contents/
   • philosophy and the principles of production engineering and operations at Google
2. the follow-up: https://sre.google/workbook/table-of-contents/
   • companion to the first SRE book to address questions, requests, and comments
   • frequent question: how to put these principles into practice in my team/company?
3. the security add on: https://google.github.io/building-secure-and-reliable-systems/raw/toc.html

skoenig

Optimizing Sysctl Settings for GKE Cluster Scalability

The ARP cache limits control the number of IP-to-MAC address pairs stored in the lookup table on each system. When the cache exceeds its capacity, it can lead to [severe incidents involving connection resets and timeouts][1]. If you encounter error messages like 'neighbour: arp_cache: neighbor table overflow!' in the logs, it indicates that the ARP entries have surpassed the limit.

To solve this, the gc_thresh3 value, which represents the hard maximum number of entries to keep in the ARP cache, needs to be raised above the default value of 1024 in GKE.

Unfortunately, in GKE, [most of the system settings on the cluster nodes cannot be directly configured][2]. To overcome this limitation, you can [utilize DaemonSets][3]. They provide a convenient way to apply consistent system settings to all nodes in the cluster.

To configure system settings using DaemonSets:

skoenig

Custom Kubectl Subcommands

With the introduction of custom subcommands in Kubernetes version 1.12, the kubectl command line tool became even more powerful for DevOps engineers. These plugins extend kubectl’s functionality, allowing you to streamline repetitive tasks, partially automate processes, and improve your day-to-day workflow as a Kubernetes operator.

To create or install a plugin for kubectl, simply place an executable or script in your system’s PATH or a directory specified by the $PATH environment variable. For example, kubectl-all installed in /usr/local/bin, can effortlessly invoked as a subcommand: kubectl all. No additional configuration or registration is necessary. Detailed instructions can be found in the official documentation.

An ecosystem has already developed around the Krew plugin manager, and there are [many great plugins available](https://krew.sigs.k8s.io

skoenig

extshot

Whoo backups... exiting, right? No. But it has to be done. This script helps automating offsite backups and thus more likely that they actually happen.

extshot is meant to be used as an BACKUP program invoked by cryptshot. It will copy directories with rotating names (as the ones created by rsnapshot) to a different location using stable names. Local backup directories can be easily copied this way for long-term storage to an external disk that may be moved offsite. This helps to implement the 3-2-1 backup strategy.

Setup

1. Prepare the external disk like described in this article: https://pig-monkey.com/2012/09/cryptshot-automated-encrypted-backups-rsnapshot/.
2. Install cryptshot and extshot:

skoenig

#!/usr/bin/env bash
# Small wrapper around https://youtube-dl.org/ for downloading audio contents
# from Youtube videos / playlists / channels. Inspired by https://github.com/dvehrs/podget.
# You can either list playlists and channels you want to download in CHANNELS
# below, or pass exactly one URL as a parameter.
set -euo pipefail

# ------------------------------------------------------------------------------
# some config, could be extended and moved to external file
CHANNELS=(