Set up helm with TLS for Weave flux

README.md

Installing Weave Flux helm-operator and Helm with TLS enabled

Installing Helm / Tiller

Generate certificates by calling the script generate-tiller-certs.sh. This will provide a CA, server certs for tiller and client certs for helm / weave flux.

Next deploy Helm with TLS and RBAC enabled;

kubectl apply -f helm-rbac.yaml

# Deploy helm with mutual TLS enabled
helm init --upgrade --service-account tiller \
    --override 'spec.template.spec.containers[0].command'='{/tiller,--storage=secret}' \
    --tiller-tls \
    --tiller-tls-cert ./tls/server.pem \
    --tiller-tls-key ./tls/server-key.pem \
    --tiller-tls-verify \
    --tls-ca-cert ./tls/ca.pem

To check if tiller installed succesfully with TLS enabled, try helm ls. This should give an error:

# Should give an error
$ helm ls
Error: transport is closing

When providing the certificates, it should work correctly:

helm --tls \
  --tls-ca-cert ./tls/ca.pem \
  --tls-cert ./tls/helm-user.pem \
  --tls-key ././tls/helm-user-key.pem \
  ls

deploy weave flux helm-operator

First create a new k8s secret for the client certs;

kubectl create secret tls helm-client --cert=tls/helm-user.pem    --key=./tls/helm-user-key.pem

note; this has to be in the same namespace as the helm-operator is deployed in.

Deploy flux with Helm;

helm repo add weaveworks https://weaveworks.github.io/flux

helm upgrade --install \
    --set helmOperator.create=true \
    --set git.url=$YOUR_GIT_REPO \
    --set helmOperator.tls.enable=true \
    --set helmOperator.tls.verify=true \    
    --set helmOperator.tls.secretName=helm-client \
    --set helmOperator.tls.caContent="$(cat ./tls/tiller-ca.pem)" \
    flux \
    ./chart/flux

Check if it worked

Perform a kubectl logs on the helm-operator and observe the helm client being created.

Debugging

Error creating helm client: failed to append certificates from file: /etc/fluxd/helm-ca/ca.crt

Your CA certificate content is not set correctly, check if your configMap contains the correct values. Example:

$ kubectl get configmaps flux-helm-tls-ca-config -o yaml
apiVersion: v1
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ....
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  creationTimestamp: 2018-07-04T15:27:25Z
  name: flux-helm-tls-ca-config
  namespace: helm-system
  resourceVersion: "1267257"
  selfLink: /api/v1/namespaces/helm-system/configmaps/flux-helm-tls-ca-config
  uid: c106f866-7f9e-11e8-904a-025000000001

generate-tiller-certs.sh

#!/bin/bash

TILLER_HOSTNAME=$1

export TILLER_HOSTNAME=${TILLER_HOSTNAME:-tiller-server}
export TILLER_SERVER=server
export USER_NAME=helm-user

cd ./tls

# Prep the configuration
echo '{"CN":"CA","key":{"algo":"rsa","size":4096}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json

# Create the tiller certificate
echo '{"CN":"'$TILLER_SERVER'","hosts":[""],"key":{"algo":"rsa","size":4096}}' | cfssl gencert \
  -config=ca-config.json -ca=ca.pem \
  -ca-key=ca-key.pem \
  -hostname="$TILLER_HOSTNAME" - | cfssljson -bare $TILLER_SERVER

# Create a client certificate
echo '{"CN":"'$USER_NAME'","hosts":[""],"key":{"algo":"rsa","size":4096}}' | cfssl gencert \
  -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem \
  -hostname="$TILLER_HOSTNAME" - | cfssljson -bare $USER_NAME

helm-rbac.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: helm-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: helm-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: helm-system

---
# Helm client serviceaccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: helm
  namespace: helm-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: tiller-user
  namespace: helm-system
rules:
- apiGroups:
  - ""
  resources:
  - pods/portforward
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: tiller-user-binding
  namespace: helm-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: tiller-user
subjects:
- kind: ServiceAccount
  name: helm
  namespace: helm-system

@thojkooi any ideas why getting this error?

» helm --tls \
  --tls-ca-cert=tls/ca.pem \
  --tls-cert=tls/helm-user.pem \
  --tls-key=tls/helm-user-key.pem \
  ls
Error: secrets is forbidden: User "system:serviceaccount:kube-system:tiller" cannot list resource "secrets" in API group "" in the namespace "kube-system"

I've used the namespace helm-system in this snippet. The default is kube-system - the error you gave indicates that the RBAC rules are missing to access the secrets in the kube-system namespace. So this snippet has some confusing (or missing) information, sorry about that.

I assume you have a helm-system namespace that's mostly empty. You can probably delete that (or install helm in it's namespace, using the flag --tiller-namespace=helm-system during the helm init). But do double check though..

Next, install the following RBAC rules in kube-system:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

---
# Helm client serviceaccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: helm
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: tiller-user
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods/portforward
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: tiller-user-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: tiller-user
subjects:
- kind: ServiceAccount
  name: helm
  namespace: kube-system

@thojkooi Thanks... yeah I figured that was part of it so I tried to modify things to use helm-system as well but still ended up with things not working correctly.
The fact that your snippet is using different namespace is okay but I think when I tried to add the proper namespace to the additional helm init and helm instlal flux commands, along with --tls flag to helm commands is where I was running into issues. You could also mention setting environment variables like TILLER_NAMESPACE and HELM_TLS_ENABLE could be used if you want to continue with using a different namespace.

Also you added the helm-client secret but didn't specify which namespace. So if the secret is in flux namespace then flux should be able to use it but since tiller is in either helm-system or kube-system wasn't sure if that's where it needed to be. By default with the above command the secret gets installed in default.