Keycloak + JupyterHub + SAML v2.0 PoC

kecloak_saml_poc.md

Keycloak + JupyterHub + ingress-nginx + AWS NLB

Overview

We have successfully set up a working version of Keycloak with Kubernetes (we are using AWS EKS) with JupyterHub using the [ingress-nginx)(https://github.com/kubernetes/ingress-nginx) as a reverse proxy. Keycloak is set up with JupyterHub as a standard OIDC client (confidential) and the JupyterHub successfully redirects to the Keycloak page that prompts the user to login. (For FYI this configuration is set up with the GenericOAuthenticator).

The Keyclaok Identity Provider has been tested with multiple third-party SAML IdP's, such as Okta and Auth0. The Keycloak broker successfully connects with the IdP and the user is prompted to add their credentials. After succussfully authenticating, however, the Keycloak service returns:

14:39:41,946 WARN  [org.keycloak.events] (default task-60) type=IDENTITY_PROVIDER_RESPONSE_ERROR, realmId=illumidesk, clientId=null, userId=null, ipAddress=71.59.34.199, error=invalid_saml_response, reason=invalid_destination

From what we have researched, this log error in most cases corresponds to an Entity ID mismatch. However, other posts indicate that this error could be the result of having the ingress-nginx controller set up behind a L4 load balancer that has TLS termination. We have the ingress-nginx controller configured to work with AWS NLB.

These are the instructions we have so far:

Requirements

This document provides instructions to set up a basic working version of IllumiDesk's stack with:

• AWS EKS with Kubernetes v1.18+
• AWS NLB
• Ingress controller with ingress-nginx v0.44.0+
• Keycloak v2.0.0+ for authentication services
• JupyterHub for workspace orchestration

Setup

Configure Kubernetes Namespaces

Ensure you have access to the the AWS EKS cluster with the kubectl CLI tool. Refer to AWS's official documentation for detailed instructions.

• Create a new namespace, called ingress-nginx. This namespace is used to manage the globally accessibly ingress controller:

kubectl create namespace ingress-nginx

• You can install the stack in the default namespace or select another. If you would like to use a namespace other than default, then create the new namespace using the kubectl CLI. For example:

kubectl create namespace <my-namespace>

Ingress Controller

1. Confirm ingress-nginx annotations: ensure the annotations are inline with the example output provided by the official ingress-nginx helm chart but replace elb with nlb as currently defined. For clarity, the annotations are on these lines.
2. Confirm ingress-nginx ConfigMap: k/v's (located within the data key) are equivalent to the settings provided by this section of the official helm chart output.
3. Confirm that the ingress controller's Service has the correct target ports (located in the spec section).

NOTE: This is an important piece of the puzzle when configuring ingress-nginx using NLB with TLS termination using AWS's ACM. Essentially, we are configuring the ingress-controller service to use http when the source port is https/443.

For example:

kind: Service
apiVersion: v1
metadata:
    ... ommitted for brevity
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: tohttps <-- This is new
    - name: https
      port: 443
      protocol: TCP
      targetPort: http

1. Update the Ingress Controller's ConfigMap with the correct key/value pairs as exemplified here. Ensure the ingress IP CIDR (proxy-real-ip-cidr) reflects your setup, for example, 0.0.0.0/0.

This YAML has an example ingress-controller.yaml. Make sure you update this manifest to:

• In the Service annotations, replace elb with nlb
• Update your ACM ARN with your AWS account ACM
• Update the ConfigMap's load balancer CIDR, for example 0.0.0.0./0
• Update the ConfigMap to forward proxy headers with use-forwarded-headers: "true"

Once you have confirmed all settings, deploy the nginx-ingress controller:

kubectl apply -f ingress-controller.yaml

Ingress Resource

Update the Ingress resource in the namespace where the PoC application is running. Make sure the following settings are in place:

• The tls spec is required when terminating TLS with the external load balancer.
• The hosts/host keys should be associated to the external facing URL (the example below uses demo.illumidesk.com)
• Keycloak's service is available with the /auth path. The port is the default port for Keycloak's service (8080). The service name in this case is keycloak.
• JupyterHub's service is available with the /jupyter path. The port is the external JupyterHub port (80). The service name for this external-facing services is proxy-public.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-default
  annotations:
    # use the shared ingress-nginx
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
    - hosts:
      - demo.illumidesk.com
  rules:
  - host: demo.illumidesk.com
    http:
      paths:
      - path: /auth
        backend:
          serviceName: keycloak
          servicePort: 8080
      - path: /jupyter
        backend:
          serviceName: proxy-public
          servicePort: 80

Deploy the Ingress resource:

kubectl apply -f ingress-resource.yaml

Deploy Keycloak

kubectl create -f https://raw.githubusercontent.com/keycloak/keycloak-quickstarts/latest/kubernetes-examples/keycloak.yaml

This setup is standard fair from this official example.

Configure Keycloak

Create and Configure Keycloak Realm

1. Port forward Keycloak's admin portal to your local environment: kubectl port-forward svc/keycloak 8080:8080
2. Log into admin portal at https://localhost:8080/keycloak/auth.
3. Create new realm by navigating to Home --> Realm Drop Down (top left) --> Create New Realm.
4. Enter Realm Name, such as illumidesk.
5. Click on Configure --> Realm Settings.
6. Ensure realm is toggled to Enable.
7. (Optional) Add Display Name and HTML Display Name values.
8. Enter the external URL for the Frontend URL field, for example: https://demo.illumidesk.com/auth. Make sure to append /auth if your ingress resource defines the /auth path to your upstream keycloak service.
9. Click on Login tab.
10. Select the none setting for Require SSL.
11. Click on Save

Create Keycloak Realm Client

General Settings

1. Click on Home --> Configure --> Clients --> Create. The Create button is on the top right hand portion of the page.
2. Enter Client ID, such as illumidesk-hub
3. Ensure the Enabled option is toggled to ON.
4. (Optional) Add Name and Description.
5. Ensure the Client Protocol option is set to openid-connect (default).
6. Ensure the Access Type option is set to credentials (public is default).
7. Ensure Standard Flow Enabled is toggled to ON.
8. Ensure Direct Access Grants Enabled is toggled to ON.
9. For Root URL enter https://<my-public-ip>.
10. For Base URL enter /.
11. For Web Origins enter * (any origin).

Create External SAML v2.0 Identity Provider (IdP)

Instructions to set up a SAML v2.0 Identity Provider (IdP) vary depending on the vendor. We have provided instructions for Auth0 configured as a SAML v2.0 IdP:

Auth0

1. Create a new Application by clicking on Applications --> + Create Application
2. Enter an application name, such as IllumiDesk SAML
3. Select the Regular Web Application option
4. Click on the Create button
5. In the Application URIs section, ensure the Token Endpoint Authentication Method option has Post selected.
6. In the Application URLs section, enter the Allowed Callback URLs value. This value should have the following format (the example below assumes the host is https://<my-puyblic-ip> and the realm is illumidesk)

https://<my-public-ip>/auth/realms/illumidesk/broker/saml/endpoint

7. At the bottom of the page, click on the Advanced Settings option.
8. Click on the Endpoints tab.
9. Take note of the SAML Protocol URL. It should look similar to: https://<your-sub-domain>.auth0.com/samlp/metadata/C2Nb4pMdbeAmwLy3dPhr9uB5KMep34ct
10. Click on the Save button at the bottom of the page.
11. Click on the Addons tab.
    1. Turn on the SAML2 Web App by toggling the button to on (green).
    2. Click on the SAML2 Web App card to open the Settings and Usage modal.
    3. Click on the Settings tab and enter the Application Callback URL for your application. For example, if your host is https://<my-puyblic-ip> and your Realm is illumidesk, then your Application Callback URL should be https://<my-puyblic-ip>/auth/realms/illumidesk/broker/saml/endpoint.
12. Click on the Connections tab.
    1. Enable the Username-Password-Authentication by toggling the button so that it's green.
    2. (Optional) Enable other connections, such as other Social Authentication services.

Note: the Application Callback URL from section 11.2 is also known as the Assertion Consumer Service URL, the Post-back URL, or Callback URL.

Create Keycloak Realm Identity Provider with SAML v2.0

Once you have setup your third party IdP, proceed to create and configure a Keycloak SAML v2.0 Identity Provider:

1. Click on Home --> Configure --> Identity Providers
2. Create a new SAML v2.0 provider by selecting the User-defined --> SAML v2.0
3. Enter saml for the Alias
4. (Optional) Enter a Display Name, such as IllumiDesk SAML v2.0 Identity Provider (IdP)
5. Ensure the Enabled option is toggled to ON.
6. Ensure the Trust Email option is toggled to ON.
7. The Service Provider Entity ID is populated by default. The value should append the /auth/realms/illumidesk to the root URL. For example, https://<my-puyblic-ip>/auth/realms/illumidesk.
8. In the SAML Config section, add the Service Provider Entity ID to reflect the Keycloak realm you set up in section 1 above.
9. In the SAML Config section, add the Single Sign-On Service URL. This value should match the value for the SAML IdP protocol URL. For example, with Auth0 this setting is called SAML Protocol URL in Applications --> <SAML Application Name> --> Settings --> Advanced Settings --> Endpoints --> SAML. The value should be similar to https://auth.illumidesk.com/samlp/C2Nb4pMdbeAmwLy3dPhr9uB5KMep34ct.
10. Select Unspecified for NameID Policy Format.
11. For Principal Type select Subject NameID.
12. Ensure HTTP-POST Binding Response, HTTP-POST Binding for AuthnRequest, and HTTP-POST Binding Logout are all toggled to ON.
13. Add a reasonable clock skew tolerance window in the Allowed clock skew field, such as 60.
14. Click on Save at the bottom of the page.

Configure Networking with Ingress Controller and Ingress Resources

To recap, the following services should be installed and configured:

• Kubernetes cluster
• Keycloak service running in Kubernetes cluster
• Keycloak Application Client with OIDC
• External SAML v2.0 Identity Provider (IdP)
• Keyclaok Realm Identity Provider configured to use SAML v2.0

Now we set up networking (basically through the use of the Ingress Controller and Ingress Resources) to access the application.

Deploy JupyterHub

Deploy JupyterHub using the Helm Chart. However, for the purposes of this test, any upstream service should do, including the hello-kubernetes service.

To proceed with JupyterHub, install the helm repo and then install it in your Kubernetes cluster:

helm repo add jupyterhub jupyterhub/jupyterhub
helm upgrade --install jupyterhub jupyterhub/jupyterhub --namespace default --version 0.11.1 --values hub.yaml --debug

The following custom config is a working example of the JupyterHub helm-chart custom config:

hub:
  image:
    pullPolicy: Always
  config:
    GenericOAuthenticator:
      auto_login: true
      enable_auth_state: true
      admin_users:
        - foo@example.com
      login_service: keycloak
      client_id: illumidesk-hub
      client_secret: <client-secret-from-keycloak-application-client>
      oauth_callback_url: <the-frontend-url>/jupyter/hub/oauth_callback
      authorize_url: <the-frontend-url>/auth/realms/illumidesk/protocol/openid-connect/auth
      token_url: <the-frontend-url>/auth/realms/illumidesk/protocol/openid-connect/token
      userdata_url: <the-frontend-url>/auth/realms/illumidesk/protocol/openid-connect/userinfo
      username_key: preferred_username
      userdata_params:
        state: state
      userdata_method: 'GET'
      scope:
        - openid
      redirectToServer: true
    JupyterHub:
      authenticator_class: generic-oauth
      tornado_settings:
        headers:
          Content-Security-Policy: "frame-ancestors 'self' *"
        cookie_options:
          SameSite: "None"
          Secure: "True"
      tls_verify: false
  baseUrl: /jupyter
  service:
    type: ClusterIP
  extraEnv:
    # required with enable_auth_state = true. Create a random value with: openssl rand -hex 32.
    JUPYTERHUB_CRYPT_KEY: "8fbcb011ea01333be5ec09bedba50c986bcc62000022926a475e8c1657a0649b"
  extraConfig:
    # logoConfig: |
    #   c.JupyterHub.logo_file = '/usr/local/share/jupyterhub/static/images/illumidesk-80.png'
proxy:
  # used as the api key between the hub and the proxy. Create a random value with: openssl rand -hex 32.
  secretToken: "7b2204c6386c563412ae761bb73b83ecb3776e122d41c096c78f58b44970ebb1"

ingress:
  enabled: true
  annotations:
    # use the shared ingress-nginx
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For, X-Forwarded-Proto, DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization"
  hosts:
    - <your-external-domain>
  # pathSuffix: /jupyter

debug:
  enabled: true