Fully remote control plane with Istio

README.md

Remote control plane

Example remote control plane setup. Workloads connect to a gateway exposed under istiod.howardjohn-mc.qualistio.org, which has a real LetsEncrypt certificate.

Setup - Control Plane

istioctl install -d manifests
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml
kubectl apply -f controlplane.yaml

The control plane setup deploys a standard out-of-the-box Istio install.

We then install cert-manager and provision an ACME cert for istiod.

Next we set up routing rules for various Istiod functionalities.

NOTE: in a real setup we would also need to add the remote cluster secrets. This would allow us to read the configs, and properly authenticate clients. This is intended just as a proof of concept.

Note that as a result of using a proper ACME cert, the control plane only needs to read the remote cluster - we do not need to patch webhooks nor write the root cert configmap.

Setup - Workload

kubectl apply -f $GOPATH/src/istio.io/istio/manifests/charts/base/crds/crd-all.gen.yaml
kubectl apply -f workload.yaml

This deploys only:

• the CRDs for Istio (technically not required if you want to do all config in the remote cluster)
• validation webhook
• injection webhook

Validate the validations works (should reject):

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: invalid-gateway
spec:
  selector:
    istio: ingressgateway
EOF

Deploy a workload with annotations:

annotations:
  proxy.istio.io/config: |
    discoveryAddress: istiod.howardjohn-mc.qualistio.org:443
  sidecar.istio.io/proxyImage: gcr.io/howardjohn-istio/proxyv2:1589225525

The proxy image is from a fork with some minor modifications: howardjohn/istio:istio/full-remote.

The discovery address can be set in the centralized mesh config, but we need a way to ensure the pods in the control plane cluster (such as the ingress) do not get this config. For now, the annotation shows this is possible.

controlplane.yaml

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: istio-system
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: howardjohn@google.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - selector: {}
      http01:
        ingress:
          class: istio
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: istio-system
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: howardjohn@google.com
    privateKeySecretRef:
      name: letsencrypt-pod
    solvers:
    - selector: {}
      http01:
        ingress:
          class: istio
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: istiod
  namespace: istio-system
spec:
  secretName: istiod-tls
  issuerRef:
    name: letsencrypt-prod
  commonName: istiod.howardjohn-mc.qualistio.org
  dnsNames:
  - istiod.howardjohn-mc.qualistio.org
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istiod
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: istiod-tls
    hosts:
    - istiod.howardjohn-mc.qualistio.org
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istiod
  namespace: istio-system
spec:
  hosts:
  - istiod.howardjohn-mc.qualistio.org
  gateways:
  - istiod
  http:
  - match:
    - uri:
        prefix: /envoy.service.discovery.v2.AggregatedDiscoveryService/StreamAggregatedResources
    route:
    - destination:
        host: istiod
        port:
          number: 15012
        subset: tls
  - match:
    - uri:
        prefix: /istio.v1.auth.IstioCertificateService/CreateCertificate
    route:
    - destination:
        host: istiod
        port:
          number: 15012
        subset: mtls
  - match:
    - uri:
        prefix: /inject
    route:
    - destination:
        host: istiod
        port:
          number: 443
        subset: tls
  - match:
    - uri:
        prefix: /validate
    route:
    - destination:
        host: istiod
        port:
          number: 443
        subset: tls
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: istiod
  namespace: istio-system
spec:
  host: istiod.istio-system.svc.cluster.local
  subsets:
  # Set up TLS to istiod backends
  - name: tls
    trafficPolicy:
      connectionPool:
        http:
          h2UpgradePolicy: UPGRADE
      tls:
        mode: SIMPLE
  # Set up mtls for CA
  # Note: this is horribly insecure. We are just hacking the authentication here -- any client
  # will be authenticated as the ingress gateway. DO NOT USE.
  # In real setup, we will do proper MC setup. Authentication will be done by MC TokenReview.
  - name: mtls
    trafficPolicy:
      connectionPool:
        http:
          h2UpgradePolicy: UPGRADE
      tls:
        mode: ISTIO_MUTUAL
        subjectAltNames: [istiod.istio-system.svc]

workload.yaml

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
  labels:
    app: sidecar-injector
webhooks:
  - name: sidecar-injector.istio.io
    clientConfig:
      url: https://istiod.howardjohn-mc.qualistio.org/inject
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    failurePolicy: Fail
    namespaceSelector:
      matchLabels:
        istio-injection: enabled
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: istiod-istio-system
  labels:
    app: istiod
    release: istio
    istio: istiod
webhooks:
  - name: validation.istio.io
    clientConfig:
      url: https://istiod.howardjohn-mc.qualistio.org/validate
    rules:
      - operations:
        - CREATE
        - UPDATE
        apiGroups:
        - config.istio.io
        - rbac.istio.io
        - security.istio.io
        - authentication.istio.io
        - networking.istio.io
        apiVersions:
        - "*"
        resources:
        - "*"
    failurePolicy: Fail
    sideEffects: None