timroster/lets-encrypt-free-iks.md

## lets-encrypt-free-iks.md

      
    Raw
  

              lets-encrypt-free-iks.md
            
          
    Creating Let's Encrypt certificates for IBM free Kubernetes clusters

The IBM Kubernetes service free clusters consist of a single worker node with 2 CPU and 4 GB of memory for experimenting with Kubernetes. Unlike the fee-based service, these clusters do not include capabilities for application load balancing using ingress out-of-the-box. However, if you manage a DNS domain (any provider will suffice) and can add an A record, it's possible for you to configure your own ingress that can provide http and https session termination for your containerized applications. Getting a TLS-enabled website or simply an external REST API couldn't be easier!
Prerequisites


Free IBM Kubernetes Cluster (IKS) - upgrade your account from Lite plan to create one. In the example commands, we'll assume that this cluster is named mycluster
kubectl - match your cluster API version (as of 12/5/20 - this is ~1.18.12)
helm v3
DNS domain that you can edit to configure an A record, we'll just refer to the fully qualified domain name (fqdn) for this a record as yourapp.example.com in the examples
an email address to receive notifications about certification expiration, etc from Let's Encrypt. The examples will use youremail@example.com
Log in to IBM Cloud and configure kubectl using the ibmcloud ks cluster config --cluster mycluster command


Updated on 12/5/20 to reflect changes in kubernetes 1.18 and nginxc 1.9 which implement support for IngressClass and the ingressClassName field in the ingress spec. See Improvements to the Ingress API in Kubernetes 1.18 for more information.

Components

This quick-guide was inspired by a more complex example from Bitnami. Unlike that example, which takes a black box view of the certificate creation process, this guide goes in depth with a more simple scenario to help you understand how to create application ingress resources with tls certificates.
On the IKS cluster, you will install helm charts for a nginx ingress controller from NGINX, and the cert-manager from Jetstack.
After these are installed, you will install some additional manifests to set up the CA issuer (Let's Encrypt) for cert-manager and then create a sample application that will automatically request a certificate for TLS.
Set up the ingress controller

Only do this on a free IKS instance These steps assume facts that only apply to free IKS instances:

a single worker where the cluster administrator can create pods that bind to host ports
no pre-existing ingress controller or application load balancer

Using the following steps with a paid instance can cause issues. See the IBM Cloud containers documentation for information on exposing applications with the ingress/alb services for paid clusters. You have been warned


Create a namespace for the NGINX ingress controller.
kubectl create namespace nginx-ingress


Install the NGINX ingress controller with helm using a daemonset and no service resource (which will result in a single pod that binds to ports 80 and 443 on the worker node and will skip creation of a ClusterIP, LoadBalancer, or NodePort for the daemonset).
helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update
helm install nginxc-ingress nginx-stable/nginx-ingress --set controller.kind=daemonset --set controller.service.create=false --namespace nginx-ingress

See all of the NGINX Ingress Controller helm chart options


Determine the IP address of your worker node, use the Public IP that is returned by this command to add the A record to your DNS provider for yourapp.example.com
$ ibmcloud ks workers --cluster mycluster
OK
ID                                                     Public IP        Private IP      Flavor   State    Status   Zone    Version   
kube-bqvuo8td0d72afrml2c0-mycluster-default-00000076   173.193.92.169   10.76.202.174   free     normal   Ready    hou02   1.16.9_1531


After adding the A record to your DNS provider, verify that the ingress is running and responding with something:
$ curl -I http://yourapp.example.com
HTTP/1.1 404 Not Found
Server: nginx/1.17.10
...

A 404 is expected at this point because unlike the kubernetes nginx ingress, the NGINX version of the ingress controller does not create a default backend deployment. You don't need this and it just consumes the limited resources of the Free cluster.


If you get a timeout, check that your hostname is resolving with dig yourapp.example.com or equivalent.


Set up the cert-manager

With the ingress installed, it is time to install the cert-manager and configure it with two ClusterIssuers for Let's Encrypt. Technically you don't need to configure the staging issuer, but if this is your first time working with cert-manager, it's useful in case something is amiss, or if you just want to test certificate generation.


Create the namespace for cert-manager
kubectl create namespace cert-manager


Install cert-manager using helm
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.0 --set installCRDs=true


Create a file called letsencrypt-staging.yaml with the following content, updating youremail@example.com with your actual email address
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  labels:
    name: letsencrypt-staging
spec:
  acme:
    # update the following with your email address
    email: youremail@example.com
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx


Apply this to your cluster in the cert-manager namespace
kubectl apply -f letsencrypt-staging.yaml -n cert-manager


Create a file called letsencrypt-prod.yaml with the following content, updating youremail@example.com with your actual email address
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  labels:
    name: letsencrypt-prod
spec:
  acme:
    # update the following with your email address
    email: youremail@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx


Apply this to your cluster in the cert-manager namespace
kubectl apply -f letsencrypt-prod.yaml -n cert-manager


Deploying the sample application

Now it is time to create an application with an ingress that will automatically request a certificate for the host specified in the ingress. You will do this twice, once with the staging certificate issuer endpoint and then with the production issuer. After one pass with a cluster, you can generally skip the staging request and jump right to production.

The main difference between the Let's Encrypt staging and production environments is that there are different rate limits for processing requests. Certificates created from the staging environment won't validate in browsers, but otherwise function identically to production.


Create a kubernetes manifest file called cafe.yaml using the contents from the nginx cafe example


Apply this file to the kubernetes default namespace.
kubectl create -f cafe.yaml


Create an ingress manifest file called cafe-ingress-letls.yaml with the following content (updating yourapp.example.com with your fqdn).
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: letsencrypt-staging
    acme.cert-manager.io/http01-edit-in-place: "true"
spec:
  ingressClassName: "nginx"
  tls:
  - hosts:
    - yourapp.example.com
    secretName: tls-cafe-ingress
  rules:
  - host: yourapp.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80
The very important parts here that are different from a typical Ingress resource are the annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
acme.cert-manager.io/http01-edit-in-place: "true"

When attached to an Ingress resource, this signals to the cert-manager controller that it needs to initiate the process to get a certificate created for the host defined in the tls: spec and store the private key and certificate in the specified Secret.


Create the ingress resource
kubectl create -f cafe-ingress-letls.yaml
As the resource is created, cert-manager will go into action creating a Certificate resource in the namespace. You can check the events on this resource to follow the steps being performed to create the certificate. Initially, cert-manager will add a private key to the tls-cafe-ingress secret. Then when the certificate request has completed processing, the tls-cafe-ingress secret will be updated with the certificate value. You can inspect the base64 (and decode if you wish) the key and certificate using the command kubectl get secret tls-cafe-ingress -o yaml. When both the tls.crt and tls.key keys in the secret have values, you are ready to continue.


Test out the staging certificate using curl -k
curl -k https://yourapp.example.com/coffee
One of the two coffee pods created by cafe.yaml will respond and if you change the path to /tea you will see responses from the tea pods. If this works successfully, you can update the annotation to create a production certificate.


Remove the ingress configured to use the staging ClusterIsssuer and also the staging tls secret.
kubectl delete -f cafe-ingress-letls.yaml
kubectl delete secret tls-cafe-ingress


Edit the cafe-ingress-letls.yaml manifest file and change the annotation cert-manager.io/cluster-issuer to select your production ClusterIssuer
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: letsencrypt-prod
    acme.cert-manager.io/http01-edit-in-place: "true"
spec:
...


Install the updated manifest.
kubectl create -f cafe-ingress-letls.yaml
Now the tls-cafe-ingress Certificate resource will track the progression to issuing the production signed certificate. Check the progression with kubectl get certificate tls-cafe-ingress. When the READY field is "True" the certificate has been issued and updated in the certificate.


Verify the application with curl.
curl https://yourapp.example.com/coffee
As before, you will see the response from one of the two coffee pods. But now the endpoint is using a certificate which has a signing chain that is known to browsers and other keystore types.


Visit the url in a browser and examine the certificate information presented by clicking on the lock icon or other security emblem associated with your browser. You can examine the certificate and verify that a full chain is known to the browser starting from your fqdn up to a top level certificate authority.


Summary and clean up

At this point, your free IKS cluster is ready to be used to add more applications. Just add additional A records for the websites or REST APIs that you need to create and repeat the process of defining new Ingress resources for those applications. Or alternatively, update the paths in the cafe-ingress-letls.yaml ingress resource to point to the services for those resources.
At any time, you can export from your cluster the private key and certificate using commands like this:
kubectl get secret tls-cafe-ingress -o json | jq '.data."tls.key"' | tr -d \" | base64 -D > cafe-ingress-key.pem
kubectl get secret tls-cafe-ingress -o json | jq '.data."tls.crt"' | tr -d \" | base64 -D > cafe-ingress-crt.pem

This should be done if you need to move the ingress tls secret to another cluster (for example, the free clusters expire after 30 days). Be careful with the exported certificate and private key. Any malicious user with this data can pretend to be your host. If you need to add the certificate and key to another cluster use:
 kubectl create secret tls tls-cert-name --cert=cafe-ingress-crt.pem --key=cafe-ingress-key.pem
You can remove only the sample application, ingress and secret with the commands:
kubectl delete -f cafe-ingress-letls.yaml
kubectl delete -f cafe.yaml
kubectl delete secret tls-cafe-ingress
To also remove the helm charts and namespaces for cert-manager and the nginx ingress controller:
helm delete cert-manager -n cert-manager
kubectl delete namespace cert-manager
helm delete ingress -n nginx-ingress
kubectl delete namespace nginx-ingress