pod-exec.md

Testing "pod/exec" in OpenShift

Overview

Below is a walthrough of how the "pod/exec", "pod/attach" etc subresources are protected with authorization in OpenShift/Kubernetes and can be accessed with websockets.

The examples all refer to "pod/exec", but the same info applies all similar subresources like "pod/attach" etc.

Walkthrough

As a cluster-admin, grant a new user the default view ClusterRole in a single namespace.

$ cat rolebinding.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: view
  namespace: demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: newuser
$ oc create -f rolebinding.yml
rolebinding.rbac.authorization.k8s.io/view created
$ oc get rolebindings -o wide | grep newuser
view                    ClusterRole/view                   2m35s   newuser

Check the pod/exec operations available to this user:

$ oc login -u newuser -p <redacted>
Login successful.

You have one project on this server: "demo"

Using project "demo".
$ oc auth can-i get pods --subresource=exec -n demo
no
$ oc auth can-i create pods --subresource=exec -n demo
no

Attempt to exec into an existing pod, as newuser.

$ oc exec -it sleep -- bash
Error from server (Forbidden): pods "sleep" is forbidden: User "newuser" cannot create resource "pods/exec" in API group "" in the namespace "demo"

Try again with more verbose output to see what the actual request is:

$ oc -v=6 exec -it sleep -- bash
I1215 21:25:32.838698   23839 loader.go:372] Config loaded from file:  /home/quicklab/.kube/config
I1215 21:25:32.900614   23839 round_trippers.go:553] GET https://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep 200 OK in 44 milliseconds
I1215 21:25:32.902600   23839 podcmd.go:88] Defaulting container name to sleep
I1215 21:25:32.908488   23839 round_trippers.go:553] POST https://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep/exec?command=bash&container=sleep&stdin=true&stdout=true&tty=true 403 Forbidden in 5 milliseconds
                          I1215 21:25:32.908883   23839 helpers.go:222] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "pods \"sleep\" is forbidden: User \"newuser\" cannot create resource \"pods/exec\" in API group \"\" in the namespace \"demo\"",
  "reason": "Forbidden",
  "details": {
    "name": "sleep",
    "kind": "pods"
  },
  "code": 403
}]
Error from server (Forbidden): pods "sleep" is forbidden: User "newuser" cannot create resource "pods/exec" in API group "" in the namespace "demo"

It's a POST request to /api/v1/namespaces/demo/pods/sleep/exec. Let's now try raw HTTP queries to this API endoint:

$ export TOKEN=$(oc whoami -t)
$ curl -X GET -H "Authorization: Bearer $TOKEN" 'https://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep/exec?command=bash&container=sleep&stdin=true&stdout=true&tty=true'
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods \"sleep\" is forbidden: User \"newuser\" cannot get resource \"pods/exec\" in API group \"\" in the namespace \"demo\"",
  "reason": "Forbidden",
  "details": {
    "name": "sleep",
    "kind": "pods"
  },
  "code": 403
}
$ curl -X POST -H "Authorization: Bearer $TOKEN" 'https://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep/exec?command=bash&container=sleep&stdin=true&stdout=true&tty=true'
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods \"sleep\" is forbidden: User \"newuser\" cannot create resource \"pods/exec\" in API group \"\" in the namespace \"demo\"",
  "reason": "Forbidden",
  "details": {
    "name": "sleep",
    "kind": "pods"
  },
  "code": 403
}

So newuser cannot make either GET (get) or POST (create) requests to pod/exec endpoints due to authorization.

Let's now test with websocket connections, using wscat.

$ wscat -n -H "Authorization: Bearer $TOKEN" -s "channel.k8s.io" -c 'wss://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep/exec?command=bash&container=sleep&stdin=true&stdout=true&tty=true'
error: Unexpected server response: 403

We are also not able to make websocket connections, because these require initialization with GET requests.

Now let's trying give neuser permission to get (but not "create") pod/exec. Do this by adding a new ClusterRole and RoleBinding:

$ cat pod-exec-rolebinding.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-exec
  namespace: demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pod-exec
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: newuser
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-exec
  namespace: demo
rules:
- apiGroups:
  - ""
  resources:
  - pods/exec
  verbs:
  - get
$ oc create -f pod-exec-rolebinding.yml
rolebinding.rbac.authorization.k8s.io/pod-exec created
clusterrole.rbac.authorization.k8s.io/pod-exec created
$ oc auth can-i get pods --subresource=exec -n demo
yes
$ oc auth can-i create pods --subresource=exec -n demo
no

Let's now see if we can complete pod/exec call.

$ oc exec -it sleep -- bash
Error from server (Forbidden): pods "sleep" is forbidden: User "newuser" cannot create resource "pods/exec" in API group "" in the namespace "demo"

We can't, but this is expected because as shown earlier the oc exec command makes a POST request to the pod/exec endpoint and newuser does not have permission to create (POST) to pod/exec.

Let's also test this with curl commands again.

$ curl -X POST -H "Authorization: Bearer $TOKEN" 'https://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep/exec?command=bash&container=sleep&stdin=true&stdout=true&tty=true'
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods \"sleep\" is forbidden: User \"newuser\" cannot create resource \"pods/exec\" in API group \"\" in the namespace \"demo\"",
  "reason": "Forbidden",
  "details": {
    "name": "sleep",
    "kind": "pods"
  },
  "code": 403
}
$ curl -X GET -H "Authorization: Bearer $TOKEN" 'https://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep/exec?command=bash&container=sleep&stdin=true&stdout=true&tty=true'
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Upgrade request required",
  "reason": "BadRequest",
  "code": 400
}

Our POST request fails with permission denied (403), while the GET request fails with 400: Upgrade request required. The GET request is allowed because newuser has the permission to get the pod/exec endpoint, however completing the request requires upgrading the initial connection to a websocket or similar (e.g. SPDY), which is beyond HTTP and curl.

So let's test now with a websocket connection

$ wscat -n -H "Authorization: Bearer $TOKEN" -s "channel.k8s.io" -c 'wss://api.<redacted>.com:6443/api/v1/namespaces/demo/pods/sleep/exec?command=bash&container=sleep&stdin=true&stdout=true&tty=true'
Connected (press CTRL+C to quit)
<
<
<
< [root@sleep /]#
>

This time the connection is succesful because the websocket connection is opened initially with a HTTP GET request, and is then upgraded.

Summary

Both the get or create verbs can allow for succesful pod/exec calls. However, kubectl exec and oc exec try use the create verb via HTTP POST requests to initiate these connections. Websocket connections are initiated with HTTP GET requests:

https://tools.ietf.org/html/rfc6455#section-1.2

To protect against usage of the "pod/exec", "pod/attach" etc all verbs for these subsresources should be removed from untrusted user roles, not only "create".