Install Kubeflow on AKS (k8s v1.21.7)

install.md

Kubeflow on AKS - Project KFC

Deploy k8s cluster v1.21.7 or lower, but not lower than v1.20.

Clone the manifests repo -

$ git clone -b master https://github.com/kubeflow/manifests

Download and run kustomize v3. NOTE v4 won’t work with current (v1.5) Kubeflow.

kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

$ wget https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_darwin_amd64

$ chmod +x kustomize_3.2.0_darwin_amd64

$ while ! ../kustomize_3.2.0_darwin_amd64 build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

Fix upstream TLS connect issues -

$ kubectl edit destinationrule -n kubeflow ml-pipeline

^^ Change ISTIO_MUTUAL to DISABLED

$ kubectl edit destinationrule -n kubeflow ml-pipeline-ui

^^ Change ISTIO_MUTUAL to DISABLED

Or for a programatic fix, amend these files before running kustomize -

~/kfc-playground/manifests (master) $ find . -name destination*
./apps/pipeline/upstream/base/metadata/options/istio/destination-rule.yaml
./apps/kfp-tekton/upstream/base/metadata/options/istio/destination-rule.yaml

PATCH HTTPS at cluster ingress

Which really means you need to manually fix Istio's SecureGateway

Create your own TLS cert and k8s secret because cert-manager comes broken

Use mkcert to create a self-signed cert - https://github.com/FiloSottile/mkcert

$ kubectl create secret -n istio-system tls kfc-cert --cert=kfc.snobu.org.pem --key=kfc.snobu.org-key.pem

Patch the gateway conf, add HTTPS listener

kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway

  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - kfc.snobu.org
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: kfc-cert
      mode: SIMPLE

Fix permissions for demo user (user@example.com)

Copy service account pipeline-runner to the demo user namespace:

$ kubectl get serviceaccounts -n kubeflow pipeline-runner -o yaml > pipeline-runner-sa.yaml
# Now change namespace in the YAML file to kubeflow-user-example-com
$ kubectl apply -f pipeline-runner-sa.yaml

kubectl create clusterrolebinding pipelinerunnerbinding \
  --clusterrole=cluster-admin \
  --serviceaccount=kubeflow-user-example-com:pipeline-runner