Pulumi Kubernetes Operator with Vault Secret Manager

OperatorWithVaultSecretManager.md

1. Enable transit encryption and create a key: https://www.vaultproject.io/docs/secrets/transit#setup
2. Create a policy to grant encryption key access:

   vault policy write transit -<<EOF
   path "transit/encrypt/my-key" {
      capabilities = [ "update" ]
   }
   path "transit/decrypt/my-key" {
      capabilities = [ "update" ]
   }
   EOF

3. Use Vault Helm to install vault injector in external server mode https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault#install-the-vault-helm-chart-configured-to-address-an-external-vault
4. Enable kubernetes auth using instructions here: https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault#configure-kubernetes-authentication Lets call the role in those instructions (devweb-app) pulumi-operator instead and ensure it is assigned the transit policy as well, i.e.:

   vault write auth/kubernetes/role/pulumi-operator \
        bound_service_account_names=pulumi-kubernetes-operator \
        bound_service_account_namespaces=default \
        policies=transit \
        ttl=60s
   

5. Update the deployment spec for the operator to: a. Inject a vault agent in caching mode and enable auto-auth to vault using the pulumi-operator role b. Inform the agent to handle token renewals c. Open a proxy listener which allows the operator to use the agent to enable token-less auth to vault

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pulumi-kubernetes-operator
spec:
  # Currently only 1 replica supported, until leader election: https://github.com/pulumi/pulumi-kubernetes-operator/issues/33
  replicas: 1
  selector:
    matchLabels:
      name: pulumi-kubernetes-operator
  template:
    metadata:
      annotations:
        # Agent injector settings
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        # This is the role 
        vault.hashicorp.com/role: "pulumi-operator"
        # The following instructs vault-agent to create and refresh vault tokens
        # using automatic authentication using the above role
        # and open a listener (port 8200)
        vault.hashicorp.com/agent-cache-enable: "true"
        vault.hashicorp.com/agent-cache-use-auto-auth-token: "force"
      labels:
        name: pulumi-kubernetes-operator
    spec:
      serviceAccountName: pulumi-kubernetes-operator
      imagePullSecrets:
        - name: pulumi-kubernetes-operator
      containers:
        - name: pulumi-kubernetes-operator
          image: pulumi/pulumi-kubernetes-operator:v0.0.10
          args:
          - "--zap-level=debug"
          imagePullPolicy: Always
          env:
            # VAULT_SERVER_URL is the environment variable that gocloud CDK currently uses:
            # https://gocloud.dev/howto/secrets/#vault. Here we tell the CDK client to connect to
            # the vault agent sidecar at the specified port - which essentially creates a token-less
            # proxy locally for the operator container. Note that VAULT_SERVER_TOKEN is unset.
            - name: VAULT_SERVER_URL
              value: "http://localhost:8200"
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "pulumi-kubernetes-operator"

6. Update stack to use secretsProvider: "hashivault://my-key"