Connect to a AWS EKS cluster from the Chaos Toolkit using port-exec

instructions.md

Connect to a AWS EKS cluster from the Chaos Toolkit

Until version 8.0.0a1 of the Python client for Kubernetes, we had to rely on creating a service account to allow connection via that client (basically any non-Go client) to a EKS cluster.

With this new release, this is now possible as it implements port-exec authentication.

Here are the instructions to roll:

1. Create a virtual environment (not mandatory but at least you start from a clean slate):

$ python3 -m venv .venv
$ source .venv/bin/activate
$ python3 -m pip install pip

2. Install the chaostoolkit and dependencies in there:

$ pip install -U --pre chaostoolkit chaostoolkit-kubernetes

Notice the --pre flag so we can install the alpha release of the client. Once 8.0.0 is out, this isn't needed anymore.

3. Ensure your environment is properly set

Depending on how you created your EKS cluster, the client may not find the kubeconfig automatically so make sure it knows where to locate what you want to use:

$ export KUBECONFIG=$HOME/.kube/.../config
$ export KUBERNETES_CONTEXT="..."

Set the path and the context name according to your own deployment.

Note, if you created your cluster with eksctl, it seems it set the env key to null in the kube config. This is not allowed from the Python client so you should remove that line from the config since it's not useful anyway). This may be a bug with the client.

You should also ensure to point at the right AWS profile:

$ export AWS_PROFILE=...

4. Run your experiment!

Now, you should be configured properly to run your chaostoolkit experiment without a dedicated service account!

$ chaos run list-kube-nodes.json 
[2018-10-04 18:28:06 INFO] Validating the experiment's syntax
[2018-10-04 18:28:06 INFO] Experiment looks valid
[2018-10-04 18:28:06 INFO] Running experiment: Talk to EKS using native authentication
[2018-10-04 18:28:06 INFO] No steady state hypothesis defined. That's ok, just exploring.
[2018-10-04 18:28:06 INFO] Probe: list-nodes
[2018-10-04 18:28:07 INFO] No steady state hypothesis defined. That's ok, just exploring.
[2018-10-04 18:28:07 INFO] Let's rollback...
[2018-10-04 18:28:07 INFO] No declared rollbacks, let's move on.
[2018-10-04 18:28:07 INFO] Experiment ended with status: completed

list-kube-nodes.json

{
    "version": "1.0.0",
    "title": "Talk to EKS using native authentication",
    "description": "Let's connect to AWS EKS using the port-exec authentication scheme from Kubernetes",
    "tags": [
        "kubernetes",
        "aws",
        "eks"
    ],
    "method": [
        {
            "type": "probe",
            "name": "list-nodes",
            "provider": {
                "type": "python",
                "module": "chaosk8s.node.probes",
                "func": "get_nodes"
            }
        }
    ]
}