vault client counting for jwt

vault_jwt_sa_unique_client.md

Goal:

1. Leverage jwt auth backend in vault to optimize vault client counting
2. Optimise access to secrets via templated policies

Context:

Applications running accross multiple namespaces in k8s-like environments.

ie: app1 in dev, int and prod namespaces. Each pod of these application will by default consume a vault client when connecting to vault.

Whatever the namespace is, the application consuming secrets is still the same.

How could we have all this pods consume the same vault clients?

Leveraging JWT Auth Backend

référence: https://developer.hashicorp.com/vault/docs/auth/jwt/oidc-providers/kubernetes

Example

vault auth backend creation

ISSUER="$(kubectl get --raw /.well-known/openid-configuration | jq -r '.issuer')"
vault enable jwt -path jwt_sa jwt
vault write auth/jwt/config oidc_discovery_url="${ISSUER}"

vault role creation

vault role configuration: (role_def.yaml)

{
  "role_type": "jwt",
  "user_claim_json_pointer": "true",
  "policies": ["default", "access_kv_app1"],
  "bound_audiences": <the value of the 'aud:' field of the default sa token>,
  "user_claim": "/kubernetes.io/serviceaccount/name"
}

create role: (app1)

curl \
--header "X-Vault-Token: <<VAULT TOKEN WITH PRIVILEGES TO CREATE THE ROLE>>" \
--header "X-Vault-Namespace: kube_app" \
--request POST \
--data @role_def.yaml \
http:///127.0.0.1:8200/v1/auth/jwt_sa/role/app1

For demo purposes create a static kvv2 secret engine at app1/webapp, with the following policy access_kv_app1:

path "secrets" {
  capabilities = ["list"]
}

path "app1/data/webapp" {
	capabilities = ["read","list"]
}

now that the environment is created, let's test it:

create namespaces, sa and pods:

  k create ns dev
  k create ns int
  k create ns prod
  
  k create sa app1 -n dev
  k create sa app1 -n int
  k create sa app1 -n prod
  k create sa app2 -n dev
  k create sa app2 -n int
  k create sa app2 -n prod

then, create pods: (same content, change pod name, namespace and sa)

example: nginx-app1-dev.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx-app1	#<---CHANGE FOR APP2
  namespace: dev	#<---CHANGE FOR NAMESPACE INT,PROD
spec:
  automountServiceAccountToken: false
  containers:
  - image: nginx
    name: nginx-app1	#<---CHANGE FOR APP2
    volumeMounts:
    - name: custom-token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  serviceAccountName: app1	#<---CHANGE FOR APP2
  volumes:
  - name: custom-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          path: token
          expirationSeconds: 3600
      - configMap:
          name: kube-root-ca.crt
          items:
          - key: ca.crt
            path: ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status: {}

then create the pod:

k apply -f nginx-app1-dev.yaml

... and so on for other pods

once created run

$k exec nging-app1 -n dev -it -- /bin/sh
$TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)  (copy the resulting token)
$curl \
--fail \
--request POST \
--header "X-Vault-Namespace: kube_app" \
--header "X-Vault-Request: true" \
--data '{"jwt":"$TOKEN","role":"app1"}' \
"http://127.0.0.1:8200/v1/auth/jwt_sa/login"

example output:

{
  "request_id": "c050c05d-d853-d576-2fe7-fe66353090ab",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "<REDACTED>",
    "accessor": "<REDACTED>",
    "policies": [
      "access_kv_app1",
      "default"
    ],
    "token_policies": [
      "access_kv_app1",
      "default"
    ],
    "metadata": {
      "role": "app1_role6"
    },
    "lease_duration": 2764800,
    "renewable": true,
    "entity_id": "<REDACTED>",
    "token_type": "service",
    "orphan": true,
    "mfa_requirement": null,
    "num_uses": 0
  }
}

from the output copy the client token, and issue the following curl command:

 curl \
-H "X-Vault-Token: <<client_token_here>>" \
-H "X-Vault-Namespace: kube_app" \
-X GET \
http://127.0.0.1:8200/v1/app1/data/webapp

You should get the secrets you have configured in app1/webapp Repeat the same operation for pod nginx-app1-int.

Note that you don't create addional client (yet, the lease count increases)

Repeat again for pod nginx-app2-dev: client count changes

What Happens?

with the user_claim field in the role, we instruct vault to use that name for the identity entity alias created due to successfull login.

As user_claim is equal to the sa used by the pod, and there are only 3 sa, then you will only consume 3 clients, irrespective the number of namespaces the pod are created in

Additional ideas

It is also possible to use the bound_claims to limit the scope of the token generated. The following role will only allow for delivering token to app in the dev or int kubernetes namespaces

{
"role_type": "jwt",
"user_claim_json_pointer": "true",
"policies": ["default", "access_kv_app1"],
"bound_audiences": "https://kubernetes.default.svc.cluster.local",
"user_claim": "/kubernetes.io/serviceaccount/name",
"bound_claims": {
  "/kubernetes.io/namespace": ["dev","int"]
},
"claim_mappings": {
  "/kubernetes.io/namespace": "namespace"
	}
}

And with a policy like the following (using templating) you can achieve easier fine grained control

path "secrets" {
  capabilities = ["list"]
}

path "app1/data/{{identity.entity.aliases.auth_jwt_55c8d99a.metadata.namespace}}/webapp" {
	capabilities = ["read","list"]
}

In the above role, I created a claim_mapping. The /kubernetes.io/namespacefield of the auth token of the jwt method is mapped to a metadata key namespacein this jwt auth method. This metadata key is then referenced in the policy. (note: you get the auth_jwt_ value from the command: vault auth list under the accessor) column.

Putting it all together

The the last role we created we allowed for pods in namespaces dev or ìnt to authenticate.

We can create a secret at app1/dev/webapp and another one at app1/int/webapp.

With the above policy, pods from dev namespace will access secrets in app1/dev/webapp.

And pods from int namespace will access secrets in app1/int/webapp