Skip to content

Instantly share code, notes, and snippets.

@andy108369

andy108369/tls-certs-akash-provider.md Secret

Last active April 28, 2023 20:41
Show Gist options
  • Save andy108369/da97a13069804004a405c342e21eb49c to your computer and use it in GitHub Desktop.
Save andy108369/da97a13069804004a405c342e21eb49c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tls-certs-akash-provider.md

TLS certs out-of-the-box in Akash provider

This is to replace the Kubernetes Ingress Controller Fake Certificate default cert ingress-nginx serves over 443/tcp (HTTPS) by default to all clients who do not have the certs explicitly set.

After following this doc, all deployments receiving their hostnames within *.ingress.<yourdomain> or *.<yourdomain> will automatically have the wildcard LE (Let's Encrypt) cert!

1. Install Let's Encrypt cert manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

###kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.11.0 \
  --set installCRDs=true

kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges -A

2. Configure the issuer

NOTE: If you want to use the namespaces then configure Issuer instead of the ClusterIssuer.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: youremail@xyz.com
    #server: https://acme-staging-v02.api.letsencrypt.org/directory
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-prod-issuer-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

For wildcard certs you have to use the DNS-01 type of the challenge.

https://cert-manager.io/docs/release-notes/release-notes-0.3/#acmev2-and-lets-encrypt-wildcard-certificates

https://letsencrypt.org/docs/faq/#does-let-s-encrypt-issue-wildcard-certificates

We will use Google Cloud DNS in this example.

  • Cloudflare

https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/

    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            key: api-token
            name: cloudflare-api-token-secret
          email: youremail@xyz.com
      selector:
        #dnsNames:
        #  - 'yourdomain.com'
        #  - '*.ingress.yourdomain.com'
        dnsZones:
          - 'yourdomain.com'
          - 'ingress.yourdomain.com'
        #matchLabels:
        #  use-cloudflare-solver: 'true'
  • Google Cloud DNS

https://cert-manager.io/docs/configuration/acme/dns01/google/

    solvers:
    - dns01:
        cloudDNS:
          # The ID of the GCP project
          project: "<your-gcp-project-id-number>"
          # This is the secret used to access the GCP service account JSON key
          serviceAccountSecretRef:
            name: clouddns-gcp-dns01-solver-sa
            key: key.json

3. Create the Google DNS Cloud service account for DNS-01 challenge

  1. create Role

https://console.cloud.google.com/iam-admin/roles

Role name: DNS Administrator Limited
ID: dns.admin.light
Description:
Created on: 2023-04-26
To use for DNS-01 ACME challenges.
https://cert-manager.io/docs/configuration/acme/dns01/google/

Permissions:
dns.resourceRecordSets.*
dns.changes.*
dns.managedZones.list
  1. create Service Account

https://console.cloud.google.com/iam-admin/serviceaccounts

SA Name: dns01-solver
SA ID: dns01-solver
  1. create SA key for dns01-solver SA

https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts

Download it in JSON format.

cat your-gcp-service-account-key.json | base64 | tr -d '\n'

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: clouddns-gcp-dns01-solver-sa
  namespace: cert-manager
type: Opaque
data:
  key.json: "<your-gcp-service-account-key-json-base64>"
EOF

Cloudflare

If using Cloudflare for DNS then request your API token and then create the following secret:

API Tokens are recommended for higher security, since they have more restrictive permissions and are more easily revocable.
Tokens can be created at User Profile > API Tokens > API Tokens. The following settings are recommended:

Permissions:
- Zone - DNS - Edit
- Zone - Zone - Read

Zone Resources:
- Include - All Zones

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <API token>
EOF

4. Request the wildcard certificate for your domain

NOTE: replace yourdomain bits with yours accordingly. Leave the *.ingress. bit (or adjust to the one you are using for the ingress address deployments receive) since wildcards aren't working for sub-sub domain (RFC 2818).

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-yourdomain-com
  namespace: ingress-nginx
spec:
  secretName: wildcard-yourdomain-com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: '*.yourdomain.com'
  dnsNames:
  - '*.yourdomain.com'
  - '*.ingress.yourdomain.com'
EOF

https://letsencrypt.org/docs/faq/#does-let-s-encrypt-issue-wildcard-certificates

5. Instruct ingress-nginx to use your wildcard certificate

This is to replace the Kubernetes Ingress Controller Fake Certificate default cert ingress-nginx serves over 443/tcp (HTTPS) by default to all clients who do not have the certs explicitly set.

Update your ingress-nginx-custom.yaml file with this extra argument, e.g.:

  extraArgs:
    default-ssl-certificate: "$(POD_NAMESPACE)/wildcard-yourdomain-com-tls"

And then use helm upgrade command (the same way how you've installed the ingress-nginx chart) to apply the changes.

After following this doc, all deployments receiving their hostnames within *.ingress.<yourdomain> or *.<yourdomain> will automatically have the wildcard LE (Let's Encrypt) cert!

Verify

$ echo "" | openssl s_client -connect rmkpiskhbhfpr3901cqok3dhrk.ingress.yourdoamin.com:443 -showcerts |& openssl x509 -issuer -subject -dates -noout -text | grep -E '(Issuer:|Subject:|Not Before:|Not After :|DNS:)'
        Issuer: C = US, O = Let's Encrypt, CN = R3
            Not Before: Apr 26 10:40:06 2023 GMT
            Not After : Jul 25 10:40:05 2023 GMT
        Subject: CN = *.yourdoamin.com
                DNS:*.ingress.yourdoamin.com, DNS:*.yourdoamin.com

If you still get Kubernetes Ingress Controller Fake Certificate at this point, it is likely the ingress-nginx did not pick-up the cert or the cert hasn't been issued by the cert-manager.

Extra: certs for custom DNS names

Pros:

  • No API key/token is required since no DNS-01 ACME challenge is used;
  • Users will be able to have their custom domains signed by the Let's Encrypt;
  • The certs will be getting automatically renewed every 45-60 days by the cert-manager;
  • Users do not have to send/share their domain API key/token with the provider at all (thanks to the HTTP-01 ACME challenge);

Cons:

  • No wildcard support since it's done via HTTP-01 ACME challenge; (the wildcard certs require DNS-01 ACME challenge and so the domain's API key/token)

Blockers:

  • This is (step 2 to be exact) is something Akash does not do out-of-the-box yet.

For this to happen certain conditions must be met:

  1. CNAME must be pointing to one of the provider worker nodes
  • SDL manifest contains tetris.decloud.pro
services:
  app:
    image: bsord/tetris
    expose:
      - port: 80
        as: 80
        to:
          - global: true
        accept:
          - "tetris.decloud.pro"
  • CNAME verification
$ dig tetris.decloud.pro

tetris.decloud.pro.  92  IN  CNAME provider.akash.pro.
provider.akash.pro. 92  IN  A 65.108.6.185
  1. the ingress should be annotated and patched with the tls attributes:

This is something Akash does not do out-of-the-box yet.

kubectl -n l71u6bbb5mqdu592el2mics5ltqvp49uojd8fn0ien3kg annotate ingress tetris.decloud.pro cert-manager.io/cluster-issuer="letsencrypt-prod"
kubectl -n l71u6bbb5mqdu592el2mics5ltqvp49uojd8fn0ien3kg patch ingress tetris.decloud.pro -p '{"spec":{"tls":[{"hosts":["tetris.decloud.pro"],"secretName":"tetris-decloud-pro-tls"}]}}'
  1. verify

Let's Encrypt issued the x509 cert to tetris.decloud.pro

$ echo "" | openssl s_client -connect tetris.decloud.pro:443 -showcerts |& openssl x509 -issuer -subject -dates -noout -text | grep -E '(Issuer:|Subject:|Not Before:|Not After :|DNS:)'
        Issuer: C = US, O = Let's Encrypt, CN = R3
            Not Before: Apr 28 19:19:44 2023 GMT
            Not After : Jul 27 19:19:43 2023 GMT
        Subject: CN = tetris.decloud.pro
                DNS:tetris.decloud.pro
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment