Run Kyverno outside of kubernetes cluster

DEMO.md

Run Kyverno Outside of Kubernetes Cluster

Create Host cluster

$ kind create cluster

Install Kyverno

$ helm repo add kyverno https://kyverno.github.io/kyverno/
$ helm repo update
$ helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Install vcluster CLI and create a vcluster within the host cluster.

NOTE: This setup greatly simplify the demo running on the developer machine. Having two phycially cluster would also work, but would require tunneling the trafic between them and it is out of the scope for this demo.

create vcluster.yaml with the following content:

# https://www.vcluster.com/docs/networking/coreDNS#fallback-to-host-dns
fallbackHostDns: true

$ vcluster create my-cluster -f vcluster.yaml

You are now connected to vcluster

$ kubectl get ns
NAME              STATUS   AGE
default           Active   113s
kube-system       Active   113s
kube-public       Active   113s
kube-node-lease   Active   113s

and there are not Kyverno CRDs installed:

$ kubectl get crd
NAME                              CREATED AT
helmcharts.helm.cattle.io         2023-12-08T11:52:08Z
helmchartconfigs.helm.cattle.io   2023-12-08T11:52:08Z
addons.k3s.cattle.io              2023-12-08T11:52:08Z

Switch to the host cluster and apply a Kyverno policy that adds a label to configmaps:

$ vcluster disconnect
$ kubectl apply -f - <<END
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-labels
  annotations:
    policies.kyverno.io/title: Add Labels
    policies.kyverno.io/category: Sample
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Label
spec:
  rules:
  - name: add-labels
    match:
      any:
      - resources:
          kinds:
          - ConfigMap
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            foo: bar
END

Export Kyverno MutatingWebhookConfiguration to a local file:

$ kubectl get mutatingwebhookconfiguration kyverno-resource-mutating-webhook-cfg -o yaml > kyverno-webhook.yaml

It looks like this:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: "2023-12-08T09:55:27Z"
  generation: 3
  labels:
    webhook.kyverno.io/managed-by: kyverno
  name: kyverno-resource-mutating-webhook-cfg
  resourceVersion: "30995"
  uid: 832df609-21e4-4476-8afa-f7911a0f2b6a
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM3VENDQWRXZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFZTVJZd0ZBWURWUVFEREEwcUxtdDUKZG1WeWJtOHVjM1pqTUI0WERUSXpNVEl3T0RBNE5UVXlObG9YRFRJME1USXdOekE1TlRVeU5sb3dHREVXTUJRRwpBMVVFQXd3TktpNXJlWFpsY201dkxuTjJZekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DCmdnRUJBS1pTNXNNcmhZL1NzVGNzYVNJeUZZMU9HeEFWUG9jbHl2NEJiN3Q5eEd0azRpbWYyU1VZVndmclM0NkQKRWRSSXVYdm9Bay9VTjBLTXpoOUpjZHNTOC9uSW1sQ3RsUENIUEdrNkxTOXIxd2RORUFMQUNLTWJieWtkZ2VyeQpPQlBBSWY2Qyt2YUNyTUxxZGRMVkVhU09pMmtseEpkaURSMEQrem96Z2p3V0hMcTZVaExyZ0lMUitwa2FDSmxyCmI5VHBsekJybFlCN0ZWcmh5R2xkdHBsK3g0Ymd5RHFTWkpkKzdFZ1JaNFY1UzJaSnpEV1RNbThwVkZvZGZYMlMKVzhDd0RhM080WXB4YThrMC9hNk44TE1rNmRTZnhQT1ZjWUlGY3hBUFlSSnNFWGVyeHZGcmNMNEdhd0xEWXVuUApGeUc1QlJhZStDWFZNSnYrUEV2bEVPVEdCaE1DQXdFQUFhTkNNRUF3RGdZRFZSMFBBUUgvQkFRREFnS2tNQThHCkExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkdGdHV2bmpRVmJmZHE0SEtQWW0ydGJQVExsRE1BMEcKQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUExQTRrTWpJWmR1RVFDMlNaNDVSSkdpWUhPMTFSMjU1UGNnazFTL29CWgp4SG5yeExTMkdKTHp3M3ExK1kwd21VRXBiR1owNis3THNjZExwWFZuWjRweWExUmIrM2lIU1duQ0ZHclZpZ3BkCnhCbEdBNTZQcWxNRjFSWVdzRFJrNzhWQTBOS2NCbUtiT2V0b3hTaUpCWUNReFJuY1RIRnk3dFYzblViYlhSUG8KbER1ZWlTTkJnMmltTU53TElkaElqTXhJMkVZZElZUzlSeENpMjFWL2ZhREtyVmtoQVBySGRwTytuK285bmNzTQpJUkUzN2dIS1JJcjZSVUtBcnUxTHVuOERhSHFIcEIzaUtRck8vVTcrWll1TGU4amc3MG1pdmwySmRVeGZYbm5qClZaS21UNDdlK2xsUDRFaXJ0aVcyQXRLelhMS3RSZTdTTUxFUXZJb1BZSFlYCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    service:
      name: kyverno-svc
      namespace: kyverno
      path: /mutate/fail
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: mutate.kyverno.svc-fail
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
      - kyverno
  objectSelector: {}
  reinvocationPolicy: IfNeeded
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - configmaps
    scope: '*'
  sideEffects: NoneOnDryRun
  timeoutSeconds: 10

Edit the file and remove clientConfig.service block and add client.url:

.
.
  clientConfig:
    caBundle: ...
    url: "https://kyverno-svc.kyverno.svc:443/mutate/fail"

Switch to vcluster and install the edited webhook:

$ vcluster connect my-cluster
$ kubectl apply -f kyverno-webhook.yaml

Let's try now to create a configmap:

$ kubectl create ns foo
$ kubectl create configmap my-config --from-literal=foo=bar -n foo
$ kubectl get configmap my-config -n foo -o yaml
apiVersion: v1
data:
  foo: bar
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-08T12:19:09Z"
  labels:
    foo: bar  ### the label is added by Kyverno
  name: my-config
  namespace: foo
  resourceVersion: "490"
  uid: 61dcf1e7-f064-4498-b8d3-5accf17134e3

There are still no Kyverno CRDs in the vcluster:

kubectl get crd
NAME                              CREATED AT
helmcharts.helm.cattle.io         2023-12-08T11:52:08Z
helmchartconfigs.helm.cattle.io   2023-12-08T11:52:08Z
addons.k3s.cattle.io              2023-12-08T11:52:08Z

All Kyverno policies are managed in the host cluster:

$ vcluster disconnect
$ kubectl get clusterpolicies
NAME         ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE    MESSAGE
add-labels   true        true         Audit             True    146m   Ready