tkg-ytt-overlay-additional-ca-certs

01_additional_ca_certs.yaml

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#! This ytt overlay adds additional custom CA certificates on TKG cluster nodes, so containerd and other tools trust these CA certificates.
#! It works when using Photon or Ubuntu as the TKG node template on all TKG infrastructure providers.

#! Trust your custom CA certificates on all Control Plane nodes.
#@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})
---
spec:
  kubeadmConfigSpec:
    #@overlay/match missing_ok=True
    files:
      #@overlay/append
      - content: #@ data.read("tkg-custom-ca.pem")
        owner: root:root
        permissions: "0644"
        path: /etc/ssl/certs/tkg-custom-ca.pem
    #@overlay/match missing_ok=True
    preKubeadmCommands:
      #! For Photon OS
      #@overlay/append
      - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
      #! For Ubuntu
      #@overlay/append
      - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'

#! Trust your custom CA certificates on all worker nodes.
#@overlay/match by=overlay.subset({"kind":"KubeadmConfigTemplate"})
---
spec:
  template:
    spec:
      #@overlay/match missing_ok=True
      files:
        #@overlay/append
        - content: #@ data.read("tkg-custom-ca.pem")
          owner: root:root
          permissions: "0644"
          path: /etc/ssl/certs/tkg-custom-ca.pem
      #@overlay/match missing_ok=True
      preKubeadmCommands:
        #! For Photon OS
        #@overlay/append
        - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
        #! For Ubuntu
        #@overlay/append
        - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'

tkg-custom-ca.pem

-----BEGIN CERTIFICATE-----
CA certificate content
-----END CERTIFICATE-----

This TKG ytt overlay adds additional custom CA certificates on TKG cluster nodes, so containerd and other tools trust these CA certificates, and containerd is able to pull images from the container registry that uses the CA certificate. It works when using Photon or Ubuntu as the TKG node template on all TKG infrastructure providers.

1. Download the above files and put them under TKG ytt directory ~/.tkg/providers/ytt/03_customizations/ (for TKG 1.2.x) or ~/.tanzu/tkg/providers/ytt/03_customizations/ (for TKG 1.3.x).
2. Add your CA certificates into tkg-custom-ca.pem.
3. Create TKG management clusters or workload clusters.

This ytt overlay works for TKG AMI image for AWS as the infrastructure provider: https://github.com/scottd018/tkg-configs/blob/main/additional_ca_cert_overlay.yaml