NOTE: This instruction documentation does not use Istio. But KFServing controller controls VirtualService so we must install it. Please refer to kfserving/issues/1336 for the detail.
There is no special step to install OpenShift Service Mesh for KFServing. We just follow the servicemesh installation documentation.
Install each operator by following the sections:
- Installing the Elasticsearch Operator
- Installing the Jaeger Operator
- Installing the Kiali Operator
- Installing the Red Hat OpenShift Service Mesh Operator
After that, you can create a namespaace and Istio Control Plane.
e.g. create istio-system namespace and deploy control plane.
$ oc new-project istio-system
$ cat <<EOF | oc apply -f -
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
name: basic
namespace: istio-system
spec:
version: v2.0
EOF
The Istio pods are running.
$ oc get pod -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-588b445894-5cnps 2/2 Running 0 3m37s
istio-egressgateway-85fdc5b466-rw842 1/1 Running 0 3m37s
istio-ingressgateway-844f785b79-pznz6 1/1 Running 0 3m38s
istiod-basic-57487b4974-ghb7d 1/1 Running 0 4m49s
jaeger-7f9bbbff5f-xp6mg 2/2 Running 0 3m38s
kiali-c64dcccd6-7q742 1/1 Running 0 2m41s
prometheus-69555fcf46-l5lxp 3/3 Running 0 4m7s
Once again, Istio is not used in this instruction. So the installation of Istio is not important.
Next we will install OpenShift serverless. OpenShift serverless also does not need any additional steps. We can follow the documentation.
Install serverless-operator by following the section:
Then, install Knative Serving by creating a simple CustomResource.
$ cat <<EOF | oc apply -f -
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
name: knative-serving
namespace: knative-serving
EOF
Once you created the KnativeServing CustomResource, Knative system pods start running in knative-serving namespace.
$ oc get pod -n knative-serving
NAME READY STATUS RESTARTS AGE
activator-7b949bfc7d-p5z2g 1/1 Running 0 57s
activator-7b949bfc7d-ppmlp 1/1 Running 0 42s
autoscaler-699dff8cff-pg726 1/1 Running 0 56s
autoscaler-hpa-68bfdb86ff-w5st2 1/1 Running 0 54s
autoscaler-hpa-68bfdb86ff-xzf78 1/1 Running 0 54s
controller-6d76c6cddb-6nhzs 1/1 Running 0 55s
controller-6d76c6cddb-qksrb 1/1 Running 0 37s
storage-version-migration-serving-serving-0.18.2-s7mkw 0/1 Completed 0 54s
webhook-7478668bc4-hbt46 1/1 Running 0 55s
Let's deploy KFServing now. This instruction uses the latest v0.5.0-rc2.
$ git clone git@github.com:kubeflow/kfserving.git
$ cd kfserving
$ oc apply -f install/v0.5.0-rc2/kfserving.yaml
oc apply command generates the following errors because we did not install cert-manager. You can ignore the error as we don't use it.
unable to recognize "install/v0.5.0-rc2/kfserving.yaml": no matches for kind "Certificate" in version "cert-manager.io/v1alpha2"
unable to recognize "install/v0.5.0-rc2/kfserving.yaml": no matches for kind "Issuer" in version "cert-manager.io/v1alpha2"
After deployed, kfserving-controller-manager pod is deployed with ContainerCreating status. ContainerCreating could be fixed by deploying certificate in the next step.
$ oc get pod -n kfserving-system
NAME READY STATUS RESTARTS AGE
kfserving-controller-manager-0 0/2 ContainerCreating 0 82s
cert-manager is one of the requirement of KFServing but OpenShift have an altenative feature to generate certificates. We will add annotation to generate the certificates.
First create certificates for kfserving-controller-manager.
$ oc annotate svc -n kfserving-system kfserving-webhook-server-service service.beta.openshift.io/serving-cert-secret-name=kfserving-webhook-server-cert
Above annotation crate kfserving-webhook-server-cert secret so kfserving-controller-manager-0 pod should start running. It may take a few minutes so please be patient or recreate the pod.
$ oc get pod -n kfserving-system
NAME READY STATUS RESTARTS AGE
kfserving-controller-manager-0 2/2 Running 0 2m37s
Then you also need to add service.beta.openshift.io/inject-cabundle=true annotation to custom resource and webhookconfiguration to inject CA.
$ oc annotate crd inferenceservices.serving.kubeflow.org service.beta.openshift.io/inject-cabundle=true
$ oc annotate mutatingwebhookconfiguration inferenceservice.serving.kubeflow.org service.beta.openshift.io/inject-cabundle=true
$ oc annotate validatingwebhookconfiguration inferenceservice.serving.kubeflow.org service.beta.openshift.io/inject-cabundle=true
This is an OpenShift magic. Please refer to Securing service traffic using service serving certificate secrets for the detail.
Finally you can use KFServing. KFServing has some samples so let's play with it.
$ oc new-project demo
For example, let's use lightgbm.
$ cd docs/samples/v1beta1/lightgbm
$ oc apply -f lightgbm.yaml
$ oc get ksvc
NAME URL LATESTCREATED LATESTREADY READY REASON
lightgbm-iris-predictor-default http://lightgbm-iris-predictor-default-demo.apps.knakayam-myc0.devcluster.openshift.com lightgbm-iris-predictor-default-00002 lightgbm-iris-predictor-default-00002 True
We can access to the ksvc URL.
$ MODEL_NAME=lightgbm-iris
$ INPUT_PATH=@./iris-input.json
$ URL=`oc get ksvc lightgbm-iris-predictor-default -o jsonpath='{.status.url}'`
$ curl $URL/v1/models/$MODEL_NAME:predict -d $INPUT_PATH
{"predictions": [[0.9999378629898321, 4.415799218835629e-05, 1.797901797954119e-05]]}
NOTE We cannot access the URL in isvc because we don't use Istio.
$ oc get isvc
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
lightgbm-iris http://lightgbm-iris-demo.apps.knakayam-myc0.devcluster.openshift.com True 100 lightgbm-iris-predictor-default-00002 2m30s