how-daniel-creates-chaos-with-k6.md

How Daniel creates Chaos with k6

This is my collection of approaches/ideas/things for doing Chaos Engineering with k6.

Template

When I'm doing Chaos Engineering with k6, I rely a lot on the power of scenarios.

More information about scenarios: https://k6.io/docs/using-k6/scenarios/

This is a small template that I usually use for my experiments:

import http from 'k6/http';
import { sleep } from 'k6';
export let options = {
    scenarios: {
        generateLoadWritePath: {
            executor: 'constant-vus',
            exec: 'generateLoadWritePath',
            vus: 1,
            startTime: '20s',
            duration: '4m',
        },
        generateLoadReadPath: {
            executor: 'constant-vus',
            exec: 'generateLoadReadPath',
            vus: 1,
            startTime: '25s',
            duration: '4m',
        },
	steadyStateCheck: {
            executor: 'constant-vus',
            exec: 'steadyStateCheck',
            vus: 1,
            duration: '5m',
        },
        injectFailure: {
            executor: 'per-vu-iterations',
            exec: 'injectFailure',
            vus: 1,
            iterations: 1,
            startTime: '3m',
        },
    },
	thresholds: {
		// these are examples! But depend a lot on your scenarios.
		// the rate of successful checks should be higher than 90% for the read path
		'checks{type:read}': [{ threshold: 'rate>0.9', abortOnFail: true }],
		// the rate of successful checks should be higher than 99% for the write path
		'checks{type:write}': [{ threshold: 'rate>0.99', abortOnFail: true }],
		// the rate of successful checks should be higher than 90% for the steady checks
		'checks{type:steady}': [{ threshold: 'rate>0.9', abortOnFail: true }],
		http_req_duration: ['p(99)<1500'], // 99% of requests must complete below 1.5s
	},
};

export function setup() {
     // Here I check that the state of my cluster/deployment/app is OK before runnining the experiment.
     console.log("Setup")
}

export function generateLoadWritePath() {
    // Here I generate and ship data to my app/system. Usually, I use chuck/time based seeds to randombly generate it.
    console.log("generateLoadWritePath")
}

export function generateLoadReadPath() {
    // Here randomly query the data that I've generated and I verify that both match.
    console.log("generateLoadReadPath")
}

export function steadyStateCheck() {
    // Here I check the steady state of my system.
    console.log("Check steady state")
    sleep(5)
}

export function injectFailure() {
    // Here I inject failures
    console.log("Inject failure")
}

export function teardown() {
    // Here I remove all the failures that I've created, and roll back any damages that I've caused.
    console.log("Teardown")
}

In case you're interested in a real-world-example, I used this structure (or at least part of it) on the Grafana Tempo integration tests. Reference: https://github.com/grafana/tempo/blob/master/integration/bench/smoke_test.js

How to inject X failure

Usually, for failure injection I make a heavy use of xk6 (k6 extensions).

More information about xk6: https://k6.io/blog/extending-k6-with-xk6/ and https://github.com/topics/k6-extension

CPU/Memory/Disk stress

In Kubernetes

Short answer: xk6-kubernetes-jobs + stress-ng container

Not in Kubernetes, but I can run containers

Short answer: xk6-docker or xk6-exec/xk6-ssh + stress-ng container

Host without containers

Short answer: xk6-exec/xk6-ssh + stress-ng

Shutdown / Process Kill / Time Travel / Mess with the host

Short answer: xk6-exec/xk6-ssh

Network

TBD

Kubernetes-only

Kill Pod

Short answer: xk6-chaos