Skip to content

Instantly share code, notes, and snippets.

@willbicks

willbicks/sops-secrets-operator-example.md

Created November 6, 2023 01:58
Show Gist options
  • Save willbicks/39e1292365cb512cfeab49fe764d7a02 to your computer and use it in GitHub Desktop.
Save willbicks/39e1292365cb512cfeab49fe764d7a02 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
sops-secrets-operator-example.md

Encrypting Kubernetes Secrets with Age and SOPS on k3s

SOPS is a handy utility for encrypting sensitive content within files while making it easy to edit and track them with standard developer tools (git, diff, vi, etc.). Using the sops-secret-operator, you can deploy encrypted Kubernetes secrets as SopsSecret custom resources, which are decrypted by the operator and made available as standard Secrets for general consumption. This allows secrets to be tracked securely in version control, deployed with standard CI/CD tools, and edited securely by developers.

Comparison to Alternatives

KSOPS

KSOPS is a Kustomize plugin that supports decrypting SOPS files and applying them to your cluster.

While popular than sops-secret-operator, KSOPS relies on every deployer running Kustomize to install and manage their binary plugin, and does not provide in-cluster decryption, hampering compatability with off the shelf CI/CD tools. Additionally, I do not currently have a need for Kustomize in many of my deployments, which would mean additional lift would be required to add secrets encrpytion to non-Kustomize projects.

Sealed Secrets

Sealed Secrets is a Kubernetes operator which works similarly to sops-secret-operator, allowing the user to deploy encrypted custom resources that will be decrypted in-cluster and available as standard Secrets. Sealed secrets however can only be decrypted in-cluster, meaning local editing and testing are impossible, and a connection to the cluster (or other automation) is required to encrypt new secrets.

The sops-secret-operator can be configured to mimic the functionality of sealed secrets by encrypting secrets with a public key derived from a private key that exists only inside the cluster. However, because the public key is static, it does not require a connection to the cluster in order to encrypt new secrets, and because SOPS supports multiple keys for decryption, a file can be encrypted such that both the operator and select developers can decrypt and edit the file.

Setup

For my application, Age provides a suitable balance of tried and trusted encryption with convenient ergonomics. On a public cloud, SOPS can be instead configured to use your cloud's key management service (e.g. AWS KMS).

If noy already present, Age can be installed with go install filippo.io/age/cmd/...@c6dcfa1efcaa27879762a934d5bea0d1b83a894c

The following script will generate a new age keypair, deploy the sops-secret-operator, and output the public key for encrypting secrets. The private key will be stored in a Kubernetes secret which the operator will use to decrypt secrets. In this example, the operator is deployed by the helm-controller provided by k3s, but it can also be deployed via the Helm CLI or other means.

#!/bin/bash
set -e

# Generate a new age keypair
AGE_KEY_FILE=$PWD/key.txt
AGE_PUBLIC_KEY=$( age-keygen -o $AGE_KEY_FILE 2>&1 | awk '{ print $3 }' )

# Create a namespace for the operator and add the age private key as a secret
kubectl create namespace sops-operator
kubectl create secret generic -n sops-operator age-key --from-file=$AGE_KEY_FILE

# Deploy the sops-secret operator
cat << EOF | kubectl create -f -
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: sops-operator
spec:
  repo: https://isindir.github.io/sops-secrets-operator/
  chart: sops-secrets-operator
  version: 0.17.2
  targetNamespace: sops-operator
  valuesContent: |-
    extraEnv:
      - name: SOPS_AGE_RECIPIENTS
        value: $AGE_PUBLIC_KEY
      - name: SOPS_AGE_KEY_FILE
        value: "/mnt/age/key.txt"
    secretsAsFiles:
      - name: age-key
        mountPath: /mnt/age/
        secretName: age-key
EOF

echo "Age Public Key: \n$AGE_PUBLIC_KEY"
rm -f $AGE_KEY_FILE

Usage

Once the operator is deployed, you can create SopsSecrets whose data and stringData fields are encrypted with the operator's pubic key.

When encrypting resources, SOPS can either be configured with command line flags, or with a .sops.yaml file (per the SOPS docs). The following example uses a .sops.yaml file to configure SOPS to encrypt Kubernetes secret fields with Age and the public key generated above.

# .sops.yaml
creation_rules:
  - path_regex: \.yaml$
    encrypted_regex: ^(data|stringData)$
    # The following key is the public key generated by the above script. Any additional developers who 
    # should be able to decrypt and edit the file can also have their public keys added here, but 
    # doing so is not required for "sealed secrets" one-way encryption functionality.
    age: 'your_age_public_key'

Encrypt

Now, example secrets can be created and encrypted as follows, starting with an unencrypted example:

# test-sops-secret.yaml
apiVersion: isindir.github.com/v1alpha3
kind: SopsSecret
metadata:
    name: sopssecret-sample
spec:
    secretTemplates:
        - name: test-secret-a
          labels:
            label0: value0
            labelK: valueK
          annotations:
            key0: value0
            keyN: valueN
          stringData:
            data-name0: data-value0
            data-nameL: data-valueL
        - name: test-secret-b
          data:
            data-name1: ZGF0YS12YWx1ZTE=
            data-nameM: ZGF0YS12YWx1ZU0=
        - name: test-secret-jenkins
          labels:
            jenkins.io/credentials-type: usernamePassword
          annotations:
            jenkins.io/credentials-description: credentials from Kubernetes
          stringData:
            username: myUsername
            password: Pa$$word
        - name: test-secret-docker
          type: kubernetes.io/dockerconfigjson
          stringData:
            .dockerconfigjson: '{"auths":{"index.docker.io":{"username":"imyuser","password":"mypass","email":"myuser@abc.com","auth":"aW15dXNlcjpteXBhc3M="}}}'
# Encrypt the secrets
sops --encrypt --in-place test-sops-secret.yaml

Observe that while the rest of the file is still readable, the data and stringData fields are now encrypted, and a new sops: section has been appended with the details required for decryption.

# test-sops-secret.yaml
apiVersion: isindir.github.com/v1alpha3
kind: SopsSecret
metadata:
    name: sopssecret-sample
spec:
    secretTemplates:
        - name: test-secret-a
          labels:
            label0: value0
            labelK: valueK
          annotations:
            key0: value0
            keyN: valueN
          stringData:
            data-name0: ENC[AES256_GCM,data:5Cf0Pjngo9qnhxo=,iv:Op8WfxlGFkmunUeiakVVqLqOMOvw1torG5XphThCJiU=,tag:66VrfST89yqqFH1Ux3Kwww==,type:str]
            data-nameL: ENC[AES256_GCM,data:nK6OYcX9WUFovhE=,iv:NuUOTpt4gBbUXRIW3liENohOjza/iKLNipdG4xodUbg=,tag:93rVL1CDoH2VbY4LJsYx9w==,type:str]
        - name: test-secret-b
          data:
            data-name1: ENC[AES256_GCM,data:MnJjiJZ9VFHRtz/+C9BIPQ==,iv:DWixNULVdrc5XTLPKs5mck/lUlaaXKAMhbQW47ugf9g=,tag:yVhPhH6fTs66RZ2f7NIefQ==,type:str]
            data-nameM: ENC[AES256_GCM,data:dP56M9qkPEWmCbFINnUvRw==,iv:UEnIvTr9VIT/ab+8p3NvUtwU8qxvgaiuff57+UuwKEw=,tag:cM3HhH+YRZBriFyDsNPrKg==,type:str]
        - name: test-secret-jenkins
          labels:
            jenkins.io/credentials-type: usernamePassword
          annotations:
            jenkins.io/credentials-description: credentials from Kubernetes
          stringData:
            username: ENC[AES256_GCM,data:fp/ChAp5QjqxXA==,iv:aV1hYyxWb/ATtAdexmqq1Wg0E/pi3Te+Ot2cnBZb4IU=,tag:OO6tIaZ+a9tPuQ7aADkR5w==,type:str]
            password: ENC[AES256_GCM,data:tIvwtPBQTeI=,iv:dExAPxMK/XALTT0U0iBWxPs8puB11el5KXpN+wtGp8Q=,tag:ecj5i8QodLRazX6NHiECOQ==,type:str]
        - name: test-secret-docker
          type: kubernetes.io/dockerconfigjson
          stringData:
            .dockerconfigjson: ENC[AES256_GCM,data:5QdvySkmFyvOkSYEHt/PZ7rDhU3o3rj8zt7lANECloLp+jwjIIMzmf8aiJqvMNs1ZRcvrpIWgj3+Qv7asQA/dflQOrVJX4H8xow12b5AmB4SC9UpQgAR3NpXt2zUbs9qoqFRodlV23U6UiLegteSwSediDcqYBR40SFwtFt+cg==,iv:Ui+WZdwRROEO5V7BOomjA1tAz7gYAUvXsYLPfiz12dI=,tag:stQPcr5ShIZEFheYLmPccQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1tr40zqqma4pfcwdst4dnm6mzvsp3ltlxq8mc9rglfhc2jhgvp95q6wnr5x
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYRkhiV3lJUFdSOG95cG1t
            TUhuU1U5OVlUS2FSbGhZd2JLN3MxSVptekZJCnEvQmJoTzlKL3lpUGo2c0VNUXVh
            eUxHMEhkMjVTS29iOTBuWjFvU0JMcTAKLS0tIHQ4SE80WVRrWHRUaEo3cUNCMFJ3
            eVZTVlBuWGpxZkpMbHVkQ012ZzV4cUkKGYcU31LIqDCrmkSGXSo1ygOcj8tbOs+Z
            V8OVyk9I4agxPTirLA/fW2FJB0q/jsPzdB1NbLO0v9MLAuw+HVI+MQ==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2023-11-05T23:14:22Z"
    mac: ENC[AES256_GCM,data:4WQ+FGLvWUTnGuhHpqJWciLnFFhZfaPBegdxomJAxDHrFqM2S9Mt8Gq4qVVDf0/aLrJP9b/ish+WqVQsPckUVi0OyxIB2fbIggbgMlTAmk394nmgiLCAW3i/VU4fuWzgobGh1gOAR7bJ6JUxpOJJcuLmF9q+YV6iq0+3ZwJc39s=,iv:+Gpp1mob2VMoaQape9c5FG/3CuyGKC+vPqid+NN3D1c=,tag:OsrgA9UDYKthJ9roOLeUWg==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.8.1

Decrypt

At this point, if you generated a personal developer keypair at ~/.config/sops/age/key.txt and added it when encrypting the secrets, you can decrypt the secrets locally, or edit the file interactively.

# Decrypt the file in memory and open in your $EDITOR, re-encrypting when closed
sops test-sops-secret.yaml

# Decrypt the file on disk
sops --decrypt --in-place test-sops-secret.yaml

For testing, you can also steal the private key from the cluster for use locally, but I would not recommend doing this with a private key that will be used for production secrets.

kubectl get -n sops-operator secret age-key --template="{{ index .data \"key.txt\" }}" | base64 -d > ~/.config/sops/age/keys.txt

Deploy

Once the secrets are encrypted, they can be deployed to the cluster as any other resource. The sops-secret-operator will detect the SopsSecret and decrypt it into standard Secret resources, which can then be consumed as normal.

$ kubectl apply -f test-sops-secret.yaml
sopssecret.isindir.github.com/sopssecret-sample created
$ kubectl get secrets
NAME                         TYPE                             DATA   AGE
test-secret-docker           kubernetes.io/dockerconfigjson   1      28s
test-secret-a                Opaque                           2      28s
test-secret-b                Opaque                           2      28s
test-secret-jenkins          Opaque                           2      28s
$ kubectl describe secrets test-secret-a
Name:         test-secret-a
Namespace:    default
Labels:       label0=value0
              labelK=valueK
Annotations:  key0: value0
              keyN: valueN

Type:  Opaque

Data
====
data-name0:  11 bytes
data-nameL:  11 bytes
$ kubectl get secret test-secret-jenkins --template="{{.data.password}}" | base64 -d
Pa$$word
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment