SSH keys in kubernetes as non-root

a-non-root-ssh-keys-in-kube.md

Problem: SSH in kubernetes

SSH (and git+ssh) has very particular opinions about file permissions. In Kubernetes you can set secret file permissions, but not ownership: (see the "Secret files permissions" section) https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000

There is no way to specify the fsOwner, so you need to workaround this somehow. One way is an init container.

• init container workaround

Another workaround is to use a postStart lifecycle hook to copy it into a different place:

        lifecycle:
          postStart:
            exec:
              command:
                - /bin/sh
                - -c
                - cp /var/my-app-secrets/id_rsa /var/my-app/id_rsa

Start to Finish

Create a secret containing some ssh keys:

kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub

The output is similar to:

secret "ssh-key-secret" created

Note that the filenames will be as follows:

ssh-privatekey ssh-publickey

You can also create a kustomization.yaml with a secretGenerator field containing ssh keys.

You can access the secrets this way:

kubectl get secret -n demographics ssh-key-secret -o jsonpath="{.data.ssh-privatekey}" | base64 --decode

If you want the files to use the standard names, since ssh-privatekey is required, you need do something like this:

apiVersion: v1
kind: Secret
metadata:
  name: secret-ssh-auth
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
          -
data:
   id_rsa: |
          SEVMTE9PT09PT09PT09PT09PT09PCg==

You can also mount the secret volume with specific key paths (filenames) for the items (keys).

cronjob.yaml

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  labels:
    test: liveness
    app: pull-demographics
    cluster_name: internal-services
    project_id: khan-internal-services
    location: us-central1-b
  name: pull-demographics
  namespace: demographics
spec:
  concurrencyPolicy: Forbid  # run one at a time
  schedule: "5 4 1 1 *"    # At 04:05 UTC on January 1st
  startingDeadlineSeconds: 600  # only catch up on the last missed job 10 mins ago
  jobTemplate:
    spec:
      backoffLimit: 4  # Do not retry on failure
      template:
        metadata:
          labels:
            app: pull-demographics
            cluster_name: internal-services
            project_id: khan-internal-services
            location: us-central1-b
        spec:
          restartPolicy: Never
          containers:
          - name: pull-demographics
            image: gcr.io/khan-internal-services/districts-jobs-pull-demographics:b28c786344296e16c6a87e4d1e3c0accca07e95e
            imagePullPolicy: Always
            terminationMessagePolicy: FallbackToLogsOnError
            command: ["/go/bin/main"]
            args: []
            securityContext:
              allowPrivilegeEscalation: false
              privileged: false
              readOnlyRootFilesystem: false
              runAsUser: 65534 # nobody
              runAsGroup: 65534 # nobody_group
#              fsGroup: 2000
            resources:
              requests:
                cpu: "1"
                memory: 4Gi
              limits:
                cpu: "1"
                memory: 4Gi
            env:
            - name: DEBUG
              value: "true"
            - name: LOCAL_SSHFILE
              value: "/ssh"
            - name: LOCAL_BASEPATH
              value: "/scratch"
            - name: NWEA_RESEARCH_SFTP_USER
              valueFrom:
                secretKeyRef:
                  name: secrets
                  key: NWEA_RESEARCH_SFTP_USER
            - name: NWEA_RESEARCH_SFTP_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: secrets
                  key: NWEA_RESEARCH_SFTP_PASSWORD
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: "/config/secret/service-account-credentials.json"
            volumeMounts:
              - name: ssh-key-nobody-vol
                mountPath: "/ssh"
                readOnly: true
              - name: service-account-credentials-vol
                mountPath: "/config/secret"
                readOnly: true
              - name: scratch-vol
                mountPath: "/scratch"
          initContainers:
          - name: set-key-ownership
            image: alpine:3.13
            command: ["sh", "-c", "cp /root-key/* /ssh && chown -R 65534:65534 /ssh"]
            volumeMounts:
            - mountPath: /ssh
              name: ssh-key-nobody-vol
            - mountPath: /root-key
              name: ssh-key-root-vol
          volumes:
          - name: ssh-key-root-vol
            secret:
              secretName: ssh-key-secret
              defaultMode: 0644 # 0644 = 0o420 = chmod u=rw,g=r,o=r
              items:
              - key: ssh-privatekey
                path: id_rsa
                mode: 0600 # chmod u=rw,g=,o=
              - key: ssh-publickey
                path: id_rsa.pub
                mode: 0644 # chmod u=rw,g=r,o=r
          - name: ssh-key-nobody-vol
            emptyDir:
              medium: Memory
          - name: scratch-vol
            emptyDir:
              medium: Memory
          - name: service-account-credentials-vol
            secret:
              secretName: service-account-credentials
              defaultMode: 0644