Skip to content

Instantly share code, notes, and snippets.

@timroster
Last active September 1, 2022 02:29
Show Gist options
  • Save timroster/ad6b915a46a831ee42aae7fc1676e4a3 to your computer and use it in GitHub Desktop.
Save timroster/ad6b915a46a831ee42aae7fc1676e4a3 to your computer and use it in GitHub Desktop.
Creating Let's Encrypt certificates for IBM free Kubernetes clusters

Creating Let's Encrypt certificates for IBM free Kubernetes clusters

The IBM Kubernetes service free clusters consist of a single worker node with 2 CPU and 4 GB of memory for experimenting with Kubernetes. Unlike the fee-based service, these clusters do not include capabilities for application load balancing using ingress out-of-the-box. However, if you manage a DNS domain (any provider will suffice) and can add an A record, it's possible for you to configure your own ingress that can provide http and https session termination for your containerized applications. Getting a TLS-enabled website or simply an external REST API couldn't be easier!

Prerequisites

  • Free IBM Kubernetes Cluster (IKS) - upgrade your account from Lite plan to create one. In the example commands, we'll assume that this cluster is named mycluster
  • kubectl - match your cluster API version (as of 12/5/20 - this is ~1.18.12)
  • helm v3
  • DNS domain that you can edit to configure an A record, we'll just refer to the fully qualified domain name (fqdn) for this a record as yourapp.example.com in the examples
  • an email address to receive notifications about certification expiration, etc from Let's Encrypt. The examples will use youremail@example.com
  • Log in to IBM Cloud and configure kubectl using the ibmcloud ks cluster config --cluster mycluster command

Updated on 12/5/20 to reflect changes in kubernetes 1.18 and nginxc 1.9 which implement support for IngressClass and the ingressClassName field in the ingress spec. See Improvements to the Ingress API in Kubernetes 1.18 for more information.

Components

This quick-guide was inspired by a more complex example from Bitnami. Unlike that example, which takes a black box view of the certificate creation process, this guide goes in depth with a more simple scenario to help you understand how to create application ingress resources with tls certificates.

On the IKS cluster, you will install helm charts for a nginx ingress controller from NGINX, and the cert-manager from Jetstack.

After these are installed, you will install some additional manifests to set up the CA issuer (Let's Encrypt) for cert-manager and then create a sample application that will automatically request a certificate for TLS.

Set up the ingress controller

Only do this on a free IKS instance These steps assume facts that only apply to free IKS instances:

  • a single worker where the cluster administrator can create pods that bind to host ports
  • no pre-existing ingress controller or application load balancer

Using the following steps with a paid instance can cause issues. See the IBM Cloud containers documentation for information on exposing applications with the ingress/alb services for paid clusters. You have been warned

  1. Create a namespace for the NGINX ingress controller.

    kubectl create namespace nginx-ingress
  2. Install the NGINX ingress controller with helm using a daemonset and no service resource (which will result in a single pod that binds to ports 80 and 443 on the worker node and will skip creation of a ClusterIP, LoadBalancer, or NodePort for the daemonset).

    helm repo add nginx-stable https://helm.nginx.com/stable
    helm repo update
    helm install nginxc-ingress nginx-stable/nginx-ingress --set controller.kind=daemonset --set controller.service.create=false --namespace nginx-ingress

    See all of the NGINX Ingress Controller helm chart options

  3. Determine the IP address of your worker node, use the Public IP that is returned by this command to add the A record to your DNS provider for yourapp.example.com

    $ ibmcloud ks workers --cluster mycluster
    OK
    ID                                                     Public IP        Private IP      Flavor   State    Status   Zone    Version   
    kube-bqvuo8td0d72afrml2c0-mycluster-default-00000076   173.193.92.169   10.76.202.174   free     normal   Ready    hou02   1.16.9_1531
  4. After adding the A record to your DNS provider, verify that the ingress is running and responding with something:

    $ curl -I http://yourapp.example.com
    HTTP/1.1 404 Not Found
    Server: nginx/1.17.10
    ...

    A 404 is expected at this point because unlike the kubernetes nginx ingress, the NGINX version of the ingress controller does not create a default backend deployment. You don't need this and it just consumes the limited resources of the Free cluster.

    If you get a timeout, check that your hostname is resolving with dig yourapp.example.com or equivalent.

Set up the cert-manager

With the ingress installed, it is time to install the cert-manager and configure it with two ClusterIssuers for Let's Encrypt. Technically you don't need to configure the staging issuer, but if this is your first time working with cert-manager, it's useful in case something is amiss, or if you just want to test certificate generation.

  1. Create the namespace for cert-manager

    kubectl create namespace cert-manager
  2. Install cert-manager using helm

    helm repo add jetstack https://charts.jetstack.io
    helm repo update
    helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.0 --set installCRDs=true
  3. Create a file called letsencrypt-staging.yaml with the following content, updating youremail@example.com with your actual email address

    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-staging
      labels:
        name: letsencrypt-staging
    spec:
      acme:
        # update the following with your email address
        email: youremail@example.com
        privateKeySecretRef:
          name: letsencrypt-staging
        server: https://acme-staging-v02.api.letsencrypt.org/directory
        solvers:
        - http01:
            ingress:
              class: nginx
  4. Apply this to your cluster in the cert-manager namespace

    kubectl apply -f letsencrypt-staging.yaml -n cert-manager
  5. Create a file called letsencrypt-prod.yaml with the following content, updating youremail@example.com with your actual email address

    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-prod
      labels:
        name: letsencrypt-prod
    spec:
      acme:
        # update the following with your email address
        email: youremail@example.com
        privateKeySecretRef:
          name: letsencrypt-prod
        server: https://acme-v02.api.letsencrypt.org/directory
        solvers:
        - http01:
            ingress:
              class: nginx
  6. Apply this to your cluster in the cert-manager namespace

    kubectl apply -f letsencrypt-prod.yaml -n cert-manager

Deploying the sample application

Now it is time to create an application with an ingress that will automatically request a certificate for the host specified in the ingress. You will do this twice, once with the staging certificate issuer endpoint and then with the production issuer. After one pass with a cluster, you can generally skip the staging request and jump right to production.

The main difference between the Let's Encrypt staging and production environments is that there are different rate limits for processing requests. Certificates created from the staging environment won't validate in browsers, but otherwise function identically to production.

  1. Create a kubernetes manifest file called cafe.yaml using the contents from the nginx cafe example

  2. Apply this file to the kubernetes default namespace.

    kubectl create -f cafe.yaml
  3. Create an ingress manifest file called cafe-ingress-letls.yaml with the following content (updating yourapp.example.com with your fqdn).

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: cafe-ingress
      annotations:
        # add an annotation indicating the issuer to use.
        cert-manager.io/cluster-issuer: letsencrypt-staging
        acme.cert-manager.io/http01-edit-in-place: "true"
    spec:
      ingressClassName: "nginx"
      tls:
      - hosts:
        - yourapp.example.com
        secretName: tls-cafe-ingress
      rules:
      - host: yourapp.example.com
        http:
          paths:
          - path: /tea
            backend:
              serviceName: tea-svc
              servicePort: 80
          - path: /coffee
            backend:
              serviceName: coffee-svc
              servicePort: 80

    The very important parts here that are different from a typical Ingress resource are the annotations:

    cert-manager.io/cluster-issuer: letsencrypt-staging
    acme.cert-manager.io/http01-edit-in-place: "true"
    

    When attached to an Ingress resource, this signals to the cert-manager controller that it needs to initiate the process to get a certificate created for the host defined in the tls: spec and store the private key and certificate in the specified Secret.

  4. Create the ingress resource

    kubectl create -f cafe-ingress-letls.yaml

    As the resource is created, cert-manager will go into action creating a Certificate resource in the namespace. You can check the events on this resource to follow the steps being performed to create the certificate. Initially, cert-manager will add a private key to the tls-cafe-ingress secret. Then when the certificate request has completed processing, the tls-cafe-ingress secret will be updated with the certificate value. You can inspect the base64 (and decode if you wish) the key and certificate using the command kubectl get secret tls-cafe-ingress -o yaml. When both the tls.crt and tls.key keys in the secret have values, you are ready to continue.

  5. Test out the staging certificate using curl -k

    curl -k https://yourapp.example.com/coffee

    One of the two coffee pods created by cafe.yaml will respond and if you change the path to /tea you will see responses from the tea pods. If this works successfully, you can update the annotation to create a production certificate.

  6. Remove the ingress configured to use the staging ClusterIsssuer and also the staging tls secret.

    kubectl delete -f cafe-ingress-letls.yaml
    kubectl delete secret tls-cafe-ingress
  7. Edit the cafe-ingress-letls.yaml manifest file and change the annotation cert-manager.io/cluster-issuer to select your production ClusterIssuer

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: cafe-ingress
      annotations:
        # add an annotation indicating the issuer to use.
        cert-manager.io/cluster-issuer: letsencrypt-prod
        acme.cert-manager.io/http01-edit-in-place: "true"
    spec:
    ...
  8. Install the updated manifest.

    kubectl create -f cafe-ingress-letls.yaml

    Now the tls-cafe-ingress Certificate resource will track the progression to issuing the production signed certificate. Check the progression with kubectl get certificate tls-cafe-ingress. When the READY field is "True" the certificate has been issued and updated in the certificate.

  9. Verify the application with curl.

    curl https://yourapp.example.com/coffee

    As before, you will see the response from one of the two coffee pods. But now the endpoint is using a certificate which has a signing chain that is known to browsers and other keystore types.

  10. Visit the url in a browser and examine the certificate information presented by clicking on the lock icon or other security emblem associated with your browser. You can examine the certificate and verify that a full chain is known to the browser starting from your fqdn up to a top level certificate authority.

    example-chain

Summary and clean up

At this point, your free IKS cluster is ready to be used to add more applications. Just add additional A records for the websites or REST APIs that you need to create and repeat the process of defining new Ingress resources for those applications. Or alternatively, update the paths in the cafe-ingress-letls.yaml ingress resource to point to the services for those resources.

At any time, you can export from your cluster the private key and certificate using commands like this:

kubectl get secret tls-cafe-ingress -o json | jq '.data."tls.key"' | tr -d \" | base64 -D > cafe-ingress-key.pem
kubectl get secret tls-cafe-ingress -o json | jq '.data."tls.crt"' | tr -d \" | base64 -D > cafe-ingress-crt.pem

This should be done if you need to move the ingress tls secret to another cluster (for example, the free clusters expire after 30 days). Be careful with the exported certificate and private key. Any malicious user with this data can pretend to be your host. If you need to add the certificate and key to another cluster use:

 kubectl create secret tls tls-cert-name --cert=cafe-ingress-crt.pem --key=cafe-ingress-key.pem

You can remove only the sample application, ingress and secret with the commands:

kubectl delete -f cafe-ingress-letls.yaml
kubectl delete -f cafe.yaml
kubectl delete secret tls-cafe-ingress

To also remove the helm charts and namespaces for cert-manager and the nginx ingress controller:

helm delete cert-manager -n cert-manager
kubectl delete namespace cert-manager
helm delete ingress -n nginx-ingress
kubectl delete namespace nginx-ingress
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment