Keycloak.X k8s spec

keycloakx-k8s.yaml

apiVersion: v1
kind: Service
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak
    layer: security
spec:
  ports:
    - port: 5432
  selector:
    service: keycloak-postgres
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak-postgres
    layer: security
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /var/storage/pv-keycloak-postgres
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      service: keycloak-postgres
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak-postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      service: keycloak-postgres
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: keycloak-postgres
    spec:
      containers:
        - image: postgres
          name: keycloak-postgress
          env:
            - name: POSTGRES_DB
              value: keycloak
            - name: POSTGRES_USER
              value: keycloak
            - name: POSTGRES_PASSWORD
              value: password
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: postgres-persistent-storage
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-persistent-storage
          persistentVolumeClaim:
            claimName: keycloak-postgres
---
kind: Service
apiVersion: v1
metadata:
  name: keycloak
  labels:
    service: keycloak
spec:
  ports:
    - port: 8443
      name: https
    - port: 8080
      name: http
  selector:
    service: keycloak
    layer: security
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    service: keycloak
    layer: security
spec:
  selector:
    matchLabels:
      service: keycloak
      layer: security
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: keycloak
        layer: security
    spec:
      containers:
        - image: quay.io/keycloak/keycloak-x:16.1.0
          imagePullPolicy: IfNotPresent
          args: ["-Djgroups.dns.query=keycloak-jgroups-ping.keycloak.svc.cluster.local", "start", "--auto-build", "--cache-stack=kubernetes", "--db=postgres", "--db-url=jdbc:postgresql://keycloak-postgres/keycloak", "--db-username=keycloak", "--db-password=password", "--hostname keycloak.apps.munerasoft.com", "--proxy edge", "--spi-sticky-session-encoder-infinispan-should-attach-route=false", "--hostname-strict-https=false"]
          name: keycloak
          resources:
            limits:
              cpu: 3
              memory: 512Mi
            requests:
              cpu: 500m
              memory: 512Mi
          ports:
            - containerPort: 8443
            - containerPort: 8080
            - containerPort: 4444
            - containerPort: 8888
          env:
            - name: KEYCLOAK_ADMIN
              value: admin
            - name: KEYCLOAK_ADMIN_PASSWORD
              value: admin
            - name: JAVA_OPTS
              value: -Xms128m -Xmx128m -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=128m -XX:ParallelGCThreads=2 -XX:ConcGCThreads=2 -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:/dev/./urandom -Xlog:gc* -XX:NewRatio=1 -XX:MaxGCPauseMillis=10 -Djgroups.dns.query=keycloak-jgroups-ping.keycloak.svc.cluster.local -Dquarkus.vertx.worker-pool-size=5 -Dquarkus.vertx.event-loops-pool-size=2
              
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  labels:
    service: keycloak
    layer: security
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "KC_SC"
    nginx.ingress.kubernetes.io/session-cookie-secure: "true"
    nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "false"
    nginx.ingress.kubernetes.io/affinity-mode: "balanced"
spec:
  rules:
    - host: keycloak.apps.munerasoft.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: 
                name: keycloak
                port: 
                  number: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: keycloak
  name: keycloak-jgroups-ping
spec:
  clusterIP: None
  ports:
    - port: 4444
      name: ping
      protocol: TCP
      targetPort: 4444
  selector:
    service: keycloak
  sessionAffinity: None
  type: ClusterIP

@pedroigor, I used the above yaml as a guide and I tried running keycloak-x:13.0.0 on k8s with 2 replicas and the admin UI keeps redirecting me to the login page. 1 replica works fine.

Is there any step-by-step guide to run keycloak in HA mode on k8s?

Also, are you able to explain the significance of these:

"-Djgroups.dns.query=keycloak-jgroups-ping.keycloak.svc.cluster.local"

apiVersion: v1
kind: Service
metadata:
  labels:
    service: keycloak
  name: keycloak-jgroups-ping
spec:
  clusterIP: None
  ports:
    - port: 4444
      name: ping
      protocol: TCP
      targetPort: 4444
  selector:
    service: keycloak
  sessionAffinity: None
  type: ClusterIP

The -Djgroups.dns.query=keycloak-jgroups-ping.keycloak.svc.cluster.local is a setting that defines a DNS query for discovery. See https://github.com/keycloak/keycloak-containers/blob/master/server/README.md#openshift-example-with-dnsdns_ping.

I'm not 100% sure if you need that service spec though. And I think it also works without it.

But the DNS query should be set.

You need the headless service.

Also, see https://issues.redhat.com/browse/KEYCLOAK-18086.

@pedroigor, got it, Thanks.

So basically, we need to wait for the bug in 13.0.0 to be fixed before we can run in clustered mode? Is there any config we can apply at the container level to work around it for now?

This is the Dockerfile I am using:

FROM quay.io/keycloak/keycloak-x:13.0.0

WORKDIR /opt/jboss/keycloak

RUN ./bin/kc.sh config --cluster=default --cluster-stack=kubernetes --db=postgres --metrics-enabled=true --db-pool-initial-size=1 --db-pool-min-size=1 --db-pool-max-size=10

My k8s.yaml is very similar to yours snippet above and I have the headless service.

@pedroigor, got it, Thanks.

So basically, we need to wait for the bug in 13.0.0 to be fixed before we can run in clustered mode? Is there any config we can apply at the container level to work around it for now?

Not really. Clustering works fine but it does not if you are using the default JGroups stacks from Infinispan via the cluster-stack setting.

However, you should be able to configure clustering and the kubernetes stack by changing the conf/cluster-default.xml file to define the transport stack, manually. See https://infinispan.org/docs/11.0.x/titles/embedding/embedding.html#jgroups_getting_started-cluster-transport.

Also, make sure to change the number of owners in your caches to make sure data is replicated to at least 2 pods. In the future, we should be able to configure the number of owners more easily, without necessarily changing the cluster config file.

@pedroigor, 2 questions:

• What is the easiest way to change the default background image and default background colour for the login theme? I tried the Dockerfile example documented here but that does not work for Keycloak-X.
• Is there any argument we can use to make sure the admin UI can run on a different port compared to non-admin UI?

Thanks.

@samstride We have added some fixes in this area. It should be available in v14.

Please, look at the instructions here https://github.com/keycloak/keycloak/tree/master/distribution/server-x-dist/src/main/content/themes.

Btw, now we also disable caching when running in dev-mode. See https://github.com/keycloak/keycloak/blob/master/quarkus/server/src/main/resources/META-INF/keycloak.properties#L9.

@pedroigor, Thanks, will wait for v14.

Btw, I tried this, but that does not load the theme either:

FROM registry.access.redhat.com/ubi8-minimal AS build-env

ENV KEYCLOAK_VERSION 13.0.1
ARG KEYCLOAK_DIST=https://github.com/keycloak/keycloak/releases/download/$KEYCLOAK_VERSION/keycloak.x-preview-$KEYCLOAK_VERSION.tar.gz

RUN microdnf install -y tar gzip && \
    mkdir /opt/jboss && \ 
    cd /opt/jboss && \
    curl -L $KEYCLOAK_DIST | tar zx && \
    mv keycloak.x* keycloak

ADD tools /opt/jboss/tools

COPY themes/sunrise /opt/jboss/keycloak/themes/ ##### <-- add theme here.

FROM registry.access.redhat.com/ubi8-minimal

COPY --from=build-env /opt/jboss /opt/jboss
COPY probes /probes

RUN microdnf update -y && \
    microdnf install -y java-11-openjdk-headless && microdnf clean all && rm -rf /var/cache/yum/* && \
    echo "jboss:x:0:root" >> /etc/group && \
    echo "jboss:x:1000:0:JBoss user:/opt/jboss:/sbin/nologin" >> /etc/passwd && \
    chown -R jboss:root /opt/jboss && \
    chmod -R g+rwX /opt/jboss

USER 1000

EXPOSE 8080
EXPOSE 8443

ENTRYPOINT [ "/opt/jboss/tools/docker-entrypoint.sh" ]

WORKDIR /opt/jboss/keycloak

RUN ./bin/kc.sh config --cluster=default --cluster-stack=kubernetes --db=postgres --metrics-enabled=true --db-pool-initial-size=1 --db-pool-min-size=1 --db-pool-max-size=10

@samstride You should be able to deploy as a JAR withinthe providers dir. Themes from the themes directory are fixed in v14.

@pedroigor will this work with Keycloak v17 ?
thanks

This is the spec I'm using for testing/dev:

apiVersion: v1
kind: Service
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak
    layer: security
spec:
  ports:
    - port: 5432
  selector:
    service: keycloak-postgres
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak-postgres
    layer: security
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /var/storage/pv-keycloak-postgres
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      service: keycloak-postgres
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak-postgres
  labels:
    service: keycloak-postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      service: keycloak-postgres
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: keycloak-postgres
    spec:
      containers:
        - image: postgres
          name: keycloak-postgress
          env:
            - name: POSTGRES_DB
              value: keycloak
            - name: POSTGRES_USER
              value: keycloak
            - name: POSTGRES_PASSWORD
              value: password
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: postgres-persistent-storage
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-persistent-storage
          persistentVolumeClaim:
            claimName: keycloak-postgres
---
kind: Service
apiVersion: v1
metadata:
  name: keycloak
  labels:
    service: keycloak
spec:
  ports:
    - port: 8443
      name: https
    - port: 8080
      name: http
  selector:
    service: keycloak
    layer: security
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    service: keycloak
    layer: security
spec:
  selector:
    matchLabels:
      service: keycloak
      layer: security
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: keycloak
        layer: security
    spec:
      containers:
        - image: quay.io/keycloak/keycloak:17.0.1
          imagePullPolicy: IfNotPresent
          args: ["-Djgroups.dns.query=keycloak-jgroups-ping.keycloak.svc.cluster.local", "start", "--auto-build", "--cache-stack=kubernetes", "--db=postgres", "--db-url=jdbc:postgresql://keycloak-postgres/keycloak", "--db-username=keycloak", "--db-password=password", "--hostname keycloak.apps.mydomain.com", "--proxy edge", "--spi-sticky-session-encoder-infinispan-should-attach-route=false", "--hostname-strict-https=false"]
          name: keycloak
          resources:
            limits:
              cpu: 3
              memory: 512Mi
            requests:
              cpu: 500m
              memory: 512Mi
          ports:
            - containerPort: 8443
            - containerPort: 8080
            - containerPort: 4444
            - containerPort: 8888
          env:
            - name: KEYCLOAK_ADMIN
              value: admin
            - name: KEYCLOAK_ADMIN_PASSWORD
              value: admin
              
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  labels:
    service: keycloak
    layer: security
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "KC_SC"
    nginx.ingress.kubernetes.io/session-cookie-secure: "true"
    nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "false"
    nginx.ingress.kubernetes.io/affinity-mode: "balanced"
spec:
  rules:
    - host: keycloak.apps.mydomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: 
                name: keycloak
                port: 
                  number: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: keycloak
  name: keycloak-jgroups-ping
spec:
  clusterIP: None
  ports:
    - port: 4444
      name: ping
      protocol: TCP
      targetPort: 4444
  selector:
    service: keycloak
  sessionAffinity: None
  type: ClusterIP

We should be preparing a guide for this and making sure we have a more production-ready spec.

Also, we are about to release the new operator which should help you to easily deploy the server.

@pedroigor awesome, many thanks. I noticed that when I added two replicas, the new one does not connect to the existing pod when one of the pods goes down. There appears to be a problem with infinispan connectivity in the presence of multiple replicas.

Below are logs from the 3rd pod when i delete one of the existing pods.

Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
2022-04-07 23:09:11,577 INFO [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 12931ms
Server configuration updated and persisted. Run the following command to review the configuration:
kc.sh show-config
Next time you run the server, just run:
kc.sh start --hostname=auth.codekerala.local --proxy=edge --spi-sticky-session-encoder-infinispan-should-attach-route=false --hostname-strict-https=false
2022-04-07 23:09:17,103 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: auth.codekerala.local, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: true
2022-04-07 23:09:18,583 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-04-07 23:09:18,629 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-04-07 23:09:18,678 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-04-07 23:09:19,141 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.6.Final
2022-04-07 23:09:19,307 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2022-04-07 23:09:21,454 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-5cddf9f74d-q97xk-36026: no members discovered after 2004 ms: creating cluster as coordinator
2022-04-07 23:09:21,469 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-5cddf9f74d-q97xk-36026|0] (1) [keycloak-5cddf9f74d-q97xk-36026]
2022-04-07 23:09:21,476 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak-5cddf9f74d-q97xk-36026`, physical addresses are `[172.17.0.12:7800]`
2022-04-07 23:09:22,249 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-5cddf9f74d-q97xk-36026, Site name: null
2022-04-07 23:09:23,126 ERROR [org.keycloak.services] (main) KC-SERVICES0010: Failed to add user 'admin' to realm 'master': user with username exists
2022-04-07 23:09:23,264 INFO [io.quarkus] (main) Keycloak 17.0.1 on JVM (powered by Quarkus 2.7.5.Final) started in 11.453s. Listening on: http://0.0.0.0:8080
2022-04-07 23:09:23,264 INFO [io.quarkus] (main) Profile prod activated.
2022-04-07 23:09:23,265 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]

@pedroigor After i updated -Djgroups.dns.query=keycloak-jgroups-ping.keycloak.svc.cluster.local to "-Djgroups.dns.query=keycloak-jgroups-ping"

And changed ports in keycloak-jgroups-ping to 7800 clustering worked.
I also noticed 500 gateway error when one pod goes down but works after few minutes

Thanks

@pedroigor , should we now be using the image keycloak instead of keycloak-x?

Also, when I tried the above YAML, it doesn't seem to fully work for me.

When I click on Admin in the initial UI, it seems to add :80 to the end of the hostname, i.e. https://some.host.com:80/admin/. I have TLS terminating externally.

Is there a setting I am missing?

Thanks.