| id | name | description | products_used | redirects | default_collection_context | |||
|---|---|---|---|---|---|---|---|---|
6a84ab7d-e947-4b4f-80c0-b7559c0acb3c |
Mount Vault Secrets through Container Storage Interface (CSI) Volume |
Mount Vault secrets in your pods and deployments through a Container Storage Interface (CSI) Volume |
|
|
vault/kubernetes |
Kubernetes application pods that rely on Vault to manage their secrets can retrieve them directly via network requests or maintained on a mounted file system through the Vault Injector service via annotations or attached as ephemeral volumes. This approach of employing ephemeral volumes to store secrets is a feature of the Secrets Store extension to the Kubernetes Container Storage Interface (CSI) driver.
In this tutorial, you will setup Vault and its dependencies with a Helm chart. Then enable and configure the secrets store CSI driver to create a volume that contains a secret that you will mount to an application pod.
This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, and additional configuration to bring it all together.
This tutorial was last tested 25 Apr 2022 on macOS 11.6.1 using this configuration.
Docker version.
$ docker version
Client:
Cloud integration: v1.0.22
Version: 20.10.13
...snip...
Server: Docker Desktop 4.6.1 (76265)
Engine:
Version: 20.10.13
...snip...
Minikube version.
$ minikube version
minikube version: v1.25.2
commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7
Helm version.
$ helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.8"}
These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.
First, follow the directions to install Minikube, including VirtualBox or similar.
Next, install kubectl CLI and helm CLI.
Install kubectl with Homebrew.
$ brew install kubernetes-cli
Install helm with Homebrew.
$ brew install helm
Install kubectl with Chocolatey.
$ choco install kubernetes-cli
Install helm with Chocolatey.
$ choco install kubernetes-helm
Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters. These clusters are run locally inside Virtual Machines (VM).
Start a Kubernetes cluster.
$ minikube start
😄 minikube v1.25.2 on Darwin 11.6.1
✨ Using the docker driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔄 Restarting existing docker container for "minikube" ...
🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
▪ kubelet.housekeeping-interval=5m
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Display the version of the Kubernetes cluster.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:51:05Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Kubernetes version 1.19.0 and lower requires a local Kubernetes cluster started with two additional arguments to set the apiserver service account credentials.
Delete the current cluster.
$ minikube delete
Start a new Kubernetes cluster with additional arguments.
$ minikube start \
--extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key \
--extra-config=apiserver.service-account-issuer=https://kubernetes.default.svc.cluster.local
Verify the status of the Minikube cluster.
$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
~> Additional waiting: Even if this last command completed successfully, you may have to wait for Minikube to be available. If an error is displayed, try again after a few minutes.
The host, kubelet, apiserver report that they are running. The kubectl, a
command line interface (CLI) for running commands against Kubernetes cluster, is
also configured to communicate with this recently started cluster.
Vault manages the secrets that are written to these mountable volumes. To provide these secrets a single Vault server is required. For this demonstration Vault can be run in development mode to automatically handle initialization, unsealing, and setup of a KV secrets engine.
Add the HashiCorp Helm repository.
$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
Update all the repositories to ensure helm is aware of the latest versions.
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈
Install the latest version of the Vault Helm chart running in development mode with the injector service disabled and CSI enabled.
$ helm install vault hashicorp/vault \
--set "server.dev.enabled=true" \
--set "injector.enabled=false" \
--set "csi.enabled=true"
Example output:
NAME: vault
LAST DEPLOYED: Mon Apr 25 17:06:20 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
https://www.vaultproject.io/docs/
Your release is named vault. To learn more about the release, try:
$ helm status vault
$ helm get manifest vault
The Vault server runs in development mode on a single pod
server.dev.enabled=true. The Vault Agent Injector pod is disabled
injector.enabled=false and the Vault CSI Provider pod csi.enabled=true is
enabled.
Display all the pods within the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 58s
vault-csi-provider-t874l 1/1 Running 0 58s
Wait until the vault-0 pod is running and ready (1/1).
The volume mounted to the pod in the Create a pod with secret
mounted section expects a secret stored at
the path secret/data/db-pass. When Vault is run in development a KV
secret engine is
enabled at the path /secret.
First, start an interactive shell session on the vault-0 pod.
$ kubectl exec -it vault-0 -- /bin/sh
/ $
Your system prompt is replaced with a new prompt / $. Commands issued at this
prompt are executed on the vault-0 container.
Create a secret at the path secret/db-pass with a password.
$ vault kv put secret/db-pass password="db-secret-password"
Key Value
--- -----
created_time 2020-05-30T16:58:54.295890646Z
deletion_time n/a
destroyed false
version 1
Verify that the secret is readable at the path secret/db-pass.
$ vault kv get secret/db-pass
====== Metadata ======
Key Value
--- -----
created_time 2020-05-30T16:58:54.295890646Z
deletion_time n/a
destroyed false
version 1
====== Data ======
Key Value
--- -----
password db-secret-password
Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. The Kubernetes resources that access the secret and create the volume authenticate through this method through a role.
Enable the Kubernetes authentication method.
$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
Configure the Kubernetes authentication method with the Kubernetes API address. It will automatically use the Vault pod's own service account token.
$ vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Successful output:
Success! Data written to: auth/kubernetes/config
The environment variable KUBERNETES_PORT_443_TCP_ADDR
references the internal network address of the Kubernetes host.
Create a policy named internal-app. This will be used to give
the webapp-sa service account permission to read the kv secret
created earlier.
$ vault policy write internal-app - <<EOF
path "secret/data/db-pass" {
capabilities = ["read"]
}
EOF
The data of
kv-v2 requires that
an additional path element of data is included after its mount path (in this
case, secret/).
Finally, create a Kubernetes authentication role named database that binds
this policy with a Kubernetes service account named webapp-sa.
$ vault write auth/kubernetes/role/database \
bound_service_account_names=webapp-sa \
bound_service_account_namespaces=default \
policies=internal-app \
ttl=20m
Successful output:
Success! Data written to: auth/kubernetes/role/database
The role connects the Kubernetes service account, webapp-sa, in
the namespace, default, with the Vault policy, internal-app. The tokens
returned after authentication are valid for 20 minutes. This Kubernetes service
account name, webapp-sa, will be created below.
Lastly, exit the vault-0 pod.
$ exit
The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container's file system.
Add the Secrets Store CSI driver Helm repository.
$ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
"secrets-store-csi-driver" has been added to your repositories
Install the latest version of the Kubernetes Secrets Store CSI Driver.
$ helm install csi secrets-store-csi-driver/secrets-store-csi-driver \
--set syncSecret.enabled=true
Example output:
NAME: csi
LAST DEPLOYED: Mon Apr 25 17:12:21 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Secrets Store CSI Driver is getting deployed to your cluster.
To verify that Secrets Store CSI Driver has started, run:
kubectl --namespace=default get pods -l "app=secrets-store-csi-driver"
Now you can follow these steps https://secrets-store-csi-driver.sigs.k8s.io/getting-started/usage.html
to create a SecretProviderClass resource, and a deployment using the SecretProviderClass.
The Secrets Store CSI driver enables extension through providers. A provider is launched as a Kubernetes DaemonSet alongside of Secrets Store CSI driver DaemonSet.
The Vault CSI provider was installed above alongside Vault by the Vault Helm chart.
This DaemonSet launches its own provider pod and runs a gRPC server which the Secrets Store CSI Driver connects to to make volume mount requests.
Get all the pods within the default namespace to check that the Vault CSI provider is running.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
csi-secrets-store-csi-driver-vkppq 3/3 Running 0 20s
vault-0 1/1 Running 0 3m10s
vault-csi-provider-t874l 1/1 Running 0 3m10s
Wait until the vault-csi-provider pod is running and ready (1/1).
The Kubernetes Secrets Store CSI Driver Helm chart creates a definition for a SecretProviderClass resource. This resource describes the parameters that are given to the Vault CSI provider. To configure it requires the address of the Vault server, the name of the Vault Kubernetes authentication role, and the secrets.
Define a SecretProviderClass named vault-database.
$ cat > spc-vault-database.yaml <<EOF
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: vault-database
spec:
provider: vault
parameters:
vaultAddress: "http://vault.default:8200"
roleName: "database"
objects: |
- objectName: "db-password"
secretPath: "secret/data/db-pass"
secretKey: "password"
EOF
Create the vault-database SecretProviderClass.
$ kubectl apply --filename spc-vault-database.yaml
The vault-database SecretProviderClass describes one secret object:
objectNameis a symbolic name for that secret, and the file name to write to.secretPathis the path to the secret defined in Vault.secretKeyis a key name within that secret.
Verify that the SecretProviderClass, named vault-database has been defined in
the default namespace.
$ kubectl describe SecretProviderClass vault-database
Name: vault-database
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"secrets-store.csi.x-k8s.io/v1","kind":"SecretProviderClass","metadata":{"annotations":{},"name":"vault-database","namespace...
API Version: secrets-store.csi.x-k8s.io/v1
Kind: SecretProviderClass
## ...
With the secret stored in Vault, the authentication configured and role created,
the provider-vault extension installed and the SecretProviderClass defined
it is finally time to create a pod that mounts the desired secret.
Create a service account named webapp-sa.
$ kubectl create serviceaccount webapp-sa
Define the webapp pod that mounts the secrets volume.
$ cat > webapp-pod.yaml <<EOF
kind: Pod
apiVersion: v1
metadata:
name: webapp
spec:
serviceAccountName: webapp-sa
containers:
- image: jweissig/app:0.0.1
name: webapp
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "vault-database"
EOF
The webapp pod defines and mounts a read-only volume to /mnt/secrets-store.
The objects defined in the vault-database SecretProviderClass are written as
files within that path.
Create the webapp pod.
$ kubectl apply --filename webapp-pod.yaml
Get all the pods within the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
csi-secrets-store-csi-driver-6rf2k 3/3 Running 0 13m
csi-secrets-store-provider-vault-qm44g 1/1 Running 0 8m
webapp 1/1 Running 0 5m
vault-0 1/1 Running 0 27m
Wait until the webapp pod is running and ready (1/1).
Display the password secret written to the file system at
/mnt/secrets-store/db-password on the webapp pod.
$ kubectl exec webapp -- cat /mnt/secrets-store/db-password
db-secret-password
The value displayed matches the password value for the secret
secret/db-pass.
The Secrets Store CSI Driver also supports syncing to Kubernetes secret objects. Kubernetes secrets are populated with the contents of files from your CSI volume, and their lifetime is closely tied to the lifetime of the pod they are created for.
To add secret syncing for your webapp pod, update the
SecretProviderClass to add a secretObjects entry:
$ cat > spc-vault-database.yaml <<EOF
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: vault-database
spec:
provider: vault
secretObjects:
- data:
- key: password
objectName: db-password
secretName: dbpass
type: Opaque
parameters:
vaultAddress: "http://vault.default:8200"
roleName: "database"
objects: |
- objectName: "db-password"
secretPath: "secret/data/db-pass"
secretKey: "password"
EOF
Apply the change:
$ kubectl apply --filename spc-vault-database.yaml
When a pod references this SecretProviderClass, the CSI driver will create a Kubernetes secret called "dbpass" with the "password" field set to the contents of the "db-password" object from the parameters. The pod will wait for the secret to be created before starting, and the secret will be deleted when the pod stops.
Next, update the pod to reference the new secret:
$ cat > webapp-pod.yaml <<EOF
kind: Pod
apiVersion: v1
metadata:
name: webapp
spec:
serviceAccountName: webapp-sa
containers:
- image: jweissig/app:0.0.1
name: webapp
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: dbpass
key: password
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "vault-database"
EOF
Notice there is now an env entry, referencing a secret. Delete and redeploy
the pod:
$ kubectl delete pod webapp && kubectl apply --filename webapp-pod.yaml
Deploy the updated configs and wait until the webapp pod has come up again.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
csi-secrets-store-csi-driver-w2xxv 3/3 Running 0 4m28s
vault-0 1/1 Running 0 5m57s
vault-csi-provider-qxz8d 1/1 Running 0 5m57s
webapp 1/1 Running 0 36s
You can now verify the Kubernetes secret has been created:
$ kubectl get secret dbpass
NAME TYPE DATA AGE
dbpass Opaque 1 89s
And you can also verify the secret is available in the pod's environment:
$ kubectl exec webapp -- env | grep DB_PASSWORD
DB_PASSWORD=db-secret-password
The Kubernetes Container Storage Interface (CSI) is an extensible approach to the management of storage alongside the lifecycle of containers. Learn more about the Secrets Store CSI driver and the Vault provider in this tutorial to accomplish the secrets management for the container.
Secrets mounted on ephemeral volumes is one approach to manage secrets for applications pods. Explore how pods can retrieve them directly via network requests and through the Vault Injector service via annotations.