Setup Keycloak with OSM, mTLS between pods and SSL Let'sEncrypt certificate termination at ingress

osm-nginx-keycloak-mtls-acme.md

Let me go thru the setup :

1. Install a 1.24.0 cluster
2. Install OSM and add the namespaces:

osm install --set OpenServiceMesh.enablePermissiveTrafficPolicy=false
kubectl patch meshconfig osm-mesh-config -n osm-system -p '{"spec":{"traffic":{"enableEgress":true}}}'  --type=merge

kubectl create ns keycloak
osm namespace add keycloak
kubectl create ns ingress-nginx
osm namespace add keycloak

3. Install ingress-nginx, with fixes for this issue

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
  --set controller.service.externalTrafficPolicy=Local \
  --namespace ingress-nginx --create-namespace

4. Add the public IP to DNS for resolution:

INGRESS_IP=`kubectl get svc -n ingress-nginx ingress-nginx-controller --output=jsonpath="{.status.loadBalancer.ingress[0]['ip']}"`
az network dns record-set a add-record  -n "*.osm" -g dns -z qubernetes.com --ipv4-addres $INGRESS_IP

5. Install cert-manager:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

6. Create a clusterIssuer, with specific annotations on the resolver pods:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: alessandro.vozza@microsoft.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            metadata:
              annotations:
                "openservicemesh.io/inbound-port-exclusion-list": "8089"
EOF

7. Create the TrafficTarget and the refenced TCPRoute to allow traffic from keycloak to postgresql

kubectl apply -f - <<EOF
apiVersion: specs.smi-spec.io/v1alpha4
kind: TCPRoute
metadata:
  name: postgresql
  namespace: keycloak
spec:
  matches:
    ports:
    - 5432
---
apiVersion: access.smi-spec.io/v1alpha3
kind: TrafficTarget
metadata:
  name: postgresql
  namespace: keycloak
spec:
  destination:
    kind: ServiceAccount
    name: default
    namespace: keycloak
  rules:
  - kind: TCPRoute
    name: postgresql
  sources:
  - kind: ServiceAccount
    name: keycloak
    namespace: keycloak
EOF

8. Deploy keycloak with this values:

cat <<EOF > values.yaml
service:
  type: ClusterIP
ingress:
  enabled: true
  ingressClassName: "nginx"
  hostname: keycloak.osm.qubernetes.com
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_ssl_name "keycloak.keycloak.cluster.local";
    nginx.ingress.kubernetes.io/proxy-ssl-secret: osm-system/osm-nginx-client-cert
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
  tls: true
EOF

helm upgrade keycloak -i -n keycloak --create-namespace --values values.yaml bitnami/keycloak

9. Patch the keycloak-postgresql service with the appProtocol field; this won’t be necessary with the next release of OSM, but it is for the current (1.1.1):

kubectl patch svc -n keycloak keycloak-postgresql --type='json' -p='[{"op": "add","path": "/spec/ports/0/appProtocol","value": "TCP"}]'

10. Finally, the IngressBackend resource; note how this should list the POD port (8080) and not the service port for keycloak (80), note also that the protocol is https as it’s provided by the sidecar, while the app itself serves plain HTTP

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: ingress
  namespace: keycloak
spec:
  backends:
  - name: keycloak
    port:
      number: 8080
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    name: ingress-nginx-controller
    namespace: ingress-nginx
  - kind: AuthenticatedPrincipal
    name: ingress-nginx.ingress-nginx.cluster.local
EOF

Finally, this will setup ingress-nginx with let’s encrypt certificates terminated at the ingress and mTLS (the secret osm-system/ osm-nginx-client-cert) all the way to the keycloak pod from ingress pod and between keycloak and postgres (even if not explicitly set in neither).