This is a short guide for setting up traefik inside of a rancher kubernetes environment with client pki authentication enabled. This allows us to only grant access to specific people who have been issued a pki certificate that was signed by our Certificate Authority. In this guide we will walk through the following
- Creating a private CA + Client Certificates for PKI Authentication
- Generating Custom SSL Certificates
- Installing Traefik via helm chart
- Configuring Traefik for custom ssl and client authentication
- Adding IngressRoutes to the kubernetes cluster
- Passing Client Certificate information to downstream apis via middlewares
** IF you have your custom SSL Keys and Client CA Set up .. Skip this section **
I chose to use EasyRSA to simplify the task of creating and mantaining a private CA and certs to be distributed to clients. If you have your certs ready to use, skip this section. Parts of this section were borrowed directly from this gist
Clone easyrsa v3:
$ cd
$ git clone https://github.com/OpenVPN/easy-rsa.gitWhen git finishes, initialize your CA:
$ cd ~/easy-rsa/easyrsa3
$ ./easyrsa init-pki
$ ./easyrsa build-ca
Don't lose your CA password! Also, it's a good idea to maintain your CA keys in a machine outside your network. DO NOT store your CA keys on your server!
Easy-rsa will store your keys and certs under the ~/pki directory. Typical structure:
- ~/pki/ca.crt <-- CA cert
- ~/pki/issued <-- Issued client & server public certs
- ~/pki/private <-- Issued keys & PKCS12 certs containing certs and keys.
This will create a server certificate under ~/pki/issued and a server key under ~/pki/private.
Create a folder on your machine for storing your kubernetes configurations + files. ./kubnertes-tutorial
Create a server certificate and replace ${your_host_name} with the url of your app. (ex localhost.com)
$ cd ~/easy-rsa/easyrsa3
$ ./easyrsa --alternate-san-name="DNS:${your_host_name}" build-server-full ${your_host_name} nopassThis will create a server certificate under ~/pki/issued and a server key
under ~/pki/private.
Copy the following files from your easy-rsa machine into
your kubernetes directory ./kubernetes directory.
~/pki/issued/proxy.foobar.com.crt~/pki/private/proxy.foobar.com.key~/pki/ca.crt
We need to create a client cert. DO NOT share the same cert across multiple users/clients! At most, one cert per person (no sharing!). Let's say person is called "kube-tutorial":
$ cd easy-rsa/easyrsa3
$ ./easyrsa --days=365 build-client-full kube-tutorial nopassThis cert is valid for one year. Next, generate a PKCS#12 cert with this cert+key (most browsers can only load PKCS#12 certs):
$ ./easyrsa export-p12 meh-clientThis will ask for a cert password. Type a non-obvious password and keep it safe. The browser will ask for this password at import time.
The ".p12" file will be saved under "~/pki/private".
- Copy the p12 file over to the client machine using a safe method (SSH).
- Navigate to the Chrome Certificate Management page.
- Click "Import" and selecte the p12 file. Type the pkcs#12 password as requested.
- Still on the Cert management page, click on "Authorities".
- Use Ctrl-F to locate "org-Easy-RSA ca". ** Optional
- Click on the line with the CA, select the "three dots" menu, then "Edit".
- Select "Trust this certificate for identifying websites"
At this point we should have the following steps completed
- ✅ Creating a private CA + Client Certificates for PKI Authentication
- ✅ Generating Custom SSL Certificates
This section assumes you already have kubernetes installed. There are great tutorials for setting up kubernetes + rancher. This section assumes you are using rancher kubernetes to manage your environment.
You can use the kubectl via the online terminal or set the kubectl config local to your machine so you can run commands agaisnt your remote cluster.
Secrets are similar to ConfigMaps but are specifically intended to hold confidential data.
Create a secret for the kubernetes cluster so it can be easily used across the cluster.
$ kubectl -n ${namespace} create secret tls traefik-tls --cert=${path_to_crt}--key=${path_to_key}This will automatically create a secret for the kubernetes cluster defined in the kubectl config.
Create a secret for the ca cert we created in the Creating a private CA + Client Certificates or point it to your pre created certs.
$ kubectl create secret generic ca-secret --from-file=${path_to_ca}=ca.crtOpen rancher to verify our secrets were generated.
We will overwrite a few of the default helm chart values to achieve the following traefik deployment characteristics
- Expose the Traefik Dashboard on Port 9100 (This is critical for debugging if you are just getting started)
- Set a container directory where traefik should look for config files ( tls + ca certs, config values ). This will allow us to dynamically set the traefik configuration values without having to restart.
- Redirect ALL ingress routes to https
- Mount the kubernetes CONFIG MAP into the traefik container at a specific directory where traefik is looking for the config files.
- Mount the kubernetes SECRET which holds our CA-CERT into the traefik container.
- Mount the kubernetes SECRET which holds our TLS-CERTS into the traefik container
Overwrite the default helm chart values by adding it to this file here. Save this file in your
kubernetes folder under ./kubernetes/traefik/helm/traefik-chart-values.yaml
# Remove Dashboard access in production
# Override Helm values w/ values here.
additionalArguments:
- --api.insecure=true
- --accesslog
- --log
- --providers.file.directory=/etc/traefik/dynamic
- --providers.file.watch=true
- --metrics.prometheus=true
providers:
kubernetesCRD:
allowCrossNamespace: true
ports:
web:
redirectTo: websecure
traefik:
expose: true
#IDK if this is good or bad, not sure. need to test.
metrics:
hostPort: 9100
ingressRoute:
dashboard:
enabled: true
persistence:
enabled: true
path: /certs
size: 128Mi
volumes:
- mountPath: /etc/traefik/dynamic
name: traefik-config
type: configMap
- name: traefik-tls2
mountPath: /etc/certs
type: secret
- name: ca-secret
mountPath: /etc/ca
type: secretWe want to set the traefik config values in order to set up traefiks TLS, PKI, and middlewares.
./kubernetes/traefik/config/traefik-config.yaml
# Configure Traefik with dynamic config. References Kuberenetes Secrets.
tls:
stores:
default:
defaultCertificate:
certFile: /etc/certs/tls.crt
keyFile: /etc/certs/tls.key
certificates:
# This path points to where we defined our mount path in the helm chart values for traefik-tls.
# Kuberenetes Automatically splits a TLS secret into tls.crt and tls.key when its mounted.
- certFile: /etc/certs/tls.crt
keyFile: /etc/certs/tls.key
options:
default:
# Enable PKI/ mTLS auth for everybody that connects.
clientAuth:
# in PEM format. each file can contain multiple CAs.
# This path points to where we defined our mount path in the helm chart values for ca-secret.
caFiles:
- /etc/ca/ca.crt
clientAuthType: RequireAndVerifyClientCert
http:
middlewares:
# Create middlewares for use by the ingress routes. Refer to the traefik documentation for proper values.
secure-headers:
headers:
frameDeny: true
browserXssfilter: true
stsIncludeSubdomains: true
stsPreload: true
stsSeconds: 63072000
contentTypeNosniff: true
addVaryheader: true
contentSecurityPolicy: script-src 'self'
referrerPolicy: origin-when-cross-origin
# This middleware will pass the commonname to downstream services on all ingress routes its applied to. More options available, refer to traefik docs.
client-auth-headers:
passTLSClientCert:
info:
subject:
commonname: true
api-stripper:
stripPrefix:
prefixes:
- '/api/*'
- '/api/*'
forceSlash: false
compress-headers:
compress: {}
Add the traefik config file to our cluster as a Kubernetes ConfigMap Environment Variable
$ kubectl create configmap traefik-config --from-file=./kubernetes/traefik/config/traefik-config.yaml$ helm repo add traefik https://helm.traefik.io/traefik
$ helm repo update