The two documents above outline the basic secrets framework provided by kubernetes and its limitations. Make special note that:
- Secrets are not encrypted by default from kubernetes.
- Secrets are not reloaded upon change.
- Kubernetes is designed to keep secrets safe primarily through RBAC policies.
Before you start investigating the level at which you'd like to encrypt your secrets, you should encrypt the secrets as they are stored in the datastore. Unencrypted snapshots are easy to read. RKE Encryption Config
Secrets can also be taken off of the wire as they hop from node to node. By default RKE will encrypt etcd communication with ssl certificates. Be sure to rotate them. Rotating RKE Certs
Below are a few notes of best practice:
- Your secrets are only as secure as the weakest section of your CI/CD pipeline. (Often your application itself)
- Mounting secrets is preferable to storing them in environment variables.
- Take care to scrub secrets from your logs and traces.
- The solutions below handle kubernetes level security, your application solution can and should be composed within one of the following solutions. (Decrypting a secret using application keys themselves)
Sorted by increasing complexity:
-
- Encrypts using private keys stored in cluster
- Decrypted into kubernetes secret resources.
-
- Encrypts using a service account token
- Encrypts/decrypts using a service endpoint
- Stores decrypted passwords in a separate volume in the pod
-
- Authenticates with external store
- Leverages an init container
- Stores secrets in shared volume
-
- Authenticates with service account and vault role
- Leverages annotations to pull from external vault
- Stores secrets in shared volume
In a true zero-trust environment you would want to guarantee that the application using given secrets is actually the intended application. This would rely less on secret infrastructure however, and more on supply chain trust. See Notary