vault-sidecar-workflow.md

Vault Kubernetes - sidecar integration step by step guide

• documentation
• source code

mirror example code

git clone https://github.com/hashicorp/vault-guides.git

workflow

1. create service account
2. associate required RBAC policy
3. create Vault policy to allow secret access
4. mount and create a kv secret v1
5. create userpass user with above policy
6. test user can read secret
7. set env variables & configure k8s auth method
8. create a role to map k8s service account to policies
9. test using vault image

Create and update a service account, 'vault-auth'

cat vault-auth-service-account.yml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default

kubectl create serviceaccount vault-auth
kubectl apply --filename vault-auth-service-account.yml

Create a policy

# Create a policy file, myapp-kv-ro.hcl
$ tee myapp-kv-ro.hcl <<EOF
# If working with K/V v1
path "secret/myapp/*" {
    capabilities = ["read", "list"]
}

# If working with K/V v2
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
EOF

Create some secret

vault kv put secret/myapp/config username='appuser' \
    password='suP3rsec(et!' \
    ttl='30s'

or

vault kv put kv/k8s-secret/config username='appuser' \
    password='suP3rsec(et!' \
    ttl='30s'

Configure Kubernetes auth method

vault auth enable kubernetes

export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)

How to communicate with k8s cluster

vault write auth/kubernetes/config \
    token_reviewer_jwt="$SA_JWT_TOKEN" \
    kubernetes_host="https://<FQDN_K8s_API>:8443" \
    kubernetes_ca_cert="$SA_CA_CRT"

Create a role which map Service Account to Vault policies and TTL

vault write auth/kubernetes/role/example \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces=default \
    policies=myapp-kv-ro \
    ttl=24h

Testing k8s auth.

kubectl run test --rm -i --tty \
--env="VAULT_ADDR=https://v1.prod.yet.org:8200" \
--image alpine:3.7 -- /bin/sh

apk update; apk add curl jq
curl -s $VAULT_ADDR/v1/sys/health | jq
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl --request POST \
    --data '{"jwt": "'"$JWT"'", "role": "k8s-role"}' \
    $VAULT_ADDR/v1/auth/kubernetes/login | jq

or

kubectl run vault-shell --rm -i --tty \
--env="VAULT_ADDR=https://<VAULT_API_ADDRESS>" \
--image <VAULT_IMG> -- /bin/sh

then fetch a Vault Token to authenticate

JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
vault write -tls-skip-verify auth/kubernetes/login role=<ROLE> jwt=$JWT

Sidecar example deployment

kubectl create configmap example-vault-agent-config-gcp --from-file=./configs-k8s/
kubectl get configmap example-vault-agent-config-gcp -o yaml

Update Vault API URL in example-k8s-spec.yml and provision the pod

kubectl apply -f example-k8s-spec-gcp.yml

Debug

uncomment debug in manifest

args:
    [
      "agent",
      "-config=/etc/vault/vault-agent-config-gcp.hcl",
      "-log-level=debug",
    ]

kubectl logs vault-agent-example consul-template
kubectl get pod vault-agent-example --template '{{.status.initContainerStatuses}}'
kubectl logs vault-agent-example -c vault-agent-auth

Check

kubectl port-forward pod/vault-agent-example 8080:80

Access http://localhost:8080