Control plane routing of XDS

README.md

XDS routing by gateway

The goal of this is to make revision transparent to user. Rather than specifying a revision as part of the pod/namespace, we will do routing at a gateway based on some metadata. For this example we will user an arbitrary setting we configure on the pod - in practice we would likely use something like ISTIO_VERSION which is configured automatically for the user.

To deploy this, add the following annotation to a pod:

        annotations:
          proxy.istio.io/config: |
            discoveryAddress: istio-ingressgateway.istio-system.svc:443
            controlPlaneAuthPolicy: MUTUAL_TLS
            proxyMetadata:
              ROUTER: istiod.istio-system.svc
          sidecar.istio.io/proxyImage: gcr.io/howardjohn-istio/proxyv2:1588805810

And another pod:

        annotations:
          proxy.istio.io/config: |
            discoveryAddress: istio-ingressgateway.istio-system.svc:443
            controlPlaneAuthPolicy: MUTUAL_TLS
            proxyMetadata:
              ROUTER: istiod-canary.istio-system.svc
          sidecar.istio.io/proxyImage: gcr.io/howardjohn-istio/proxyv2:1588805810

The first one will use the default revision, the second the canary revision.

In a real setup, this would likely be derived from something automatically - no user configure here would be needed, as that defeats the purpose. To make things simple here (and allow testing with two pods running the same version) we explicitly define some metadata to drive routing decisions.

Next we need to set up config for the gateway. Apply tls.yaml, which will do SNI routing to our different control plane pods based on this.

Note that this requires a custom proxy image to:

• Set the value of ROUTER to sni.
• Change match_subject_alt_names to *.istio-system.svc. Making this actually secure is TBD.

In a real deployment, we might choose to route based on other properties:

• ISTIO_VERSION - 1.x pods go to 1.x control plane, 1.y -> 1.y
• Really fine grained - pod name, deployment name, etc. This is a bit harder with TLS than plaintext since we only have one SNI value we can set. With plaintext we can set N headers, and routing can choose to use any combination of them

change.patch

diff --git a/pkg/bootstrap/config.go b/pkg/bootstrap/config.go
index a1a042f9f..3c58705b9 100644
--- a/pkg/bootstrap/config.go
+++ b/pkg/bootstrap/config.go
@@ -140,7 +140,7 @@ func (cfg Config) toTemplateParams() (map[string]interface{}, error) {
 		return nil, err
 	}
 	opts = append(opts, getNodeMetadataOptions(meta, rawMeta, cfg.PlatEnv, cfg.Proxy)...)
-
+	opts = append(opts, option.Sni(cfg.Proxy.ProxyMetadata["ROUTER"]))
 	// Check if nodeIP carries IPv4 or IPv6 and set up proxy accordingly
 	if isIPv6Proxy(cfg.NodeIPs) {
 		opts = append(opts,
diff --git a/pkg/bootstrap/option/instances.go b/pkg/bootstrap/option/instances.go
index e6f97761c..6d229fdb8 100644
--- a/pkg/bootstrap/option/instances.go
+++ b/pkg/bootstrap/option/instances.go
@@ -78,6 +78,10 @@ func SubZone(value string) Instance {
 	return newOptionOrSkipIfZero("sub_zone", value)
 }
 
+func Sni(value string) Instance {
+	return newOptionOrSkipIfZero("sni", value)
+}
+
 func NodeMetadata(meta *model.NodeMetadata, rawMeta map[string]interface{}) Instance {
 	return newOptionOrSkipIfZero("meta_json_str", meta).withConvert(nodeMetadataConverter(meta, rawMeta))
 }
diff --git a/tools/packaging/common/envoy_bootstrap_v2.json b/tools/packaging/common/envoy_bootstrap_v2.json
index 5a8cca471..1e5d9bc4b 100644
--- a/tools/packaging/common/envoy_bootstrap_v2.json
+++ b/tools/packaging/common/envoy_bootstrap_v2.json
@@ -214,6 +214,7 @@
           "name": "envoy.transport_sockets.tls",
           "typed_config": {
             "@type": "type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext",
+            "sni": "{{.sni}}",
             "common_tls_context": {
               "alpn_protocols": [
                 "h2"
@@ -243,7 +244,7 @@
                   "filename": "./var/run/secrets/istio/root-cert.pem"
                   {{- end }}
                 },
-                "match_subject_alt_names": {{ .pilot_SAN }}
+                "match_subject_alt_names": [{"suffix": ".istio-system.svc"}]
               }
             }
           }

tls.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: controlplane-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: PASSTHROUGH
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: controlplane
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - controlplane-gateway
  tls:
  - match:
    - sniHosts:
      - istiod-canary.istio-system.svc
    route:
    - destination:
        host: istiod-canary
        port:
          number: 15012
  - match:
    - sniHosts:
      - "*"
    route:
    - destination:
        host: istiod
        port:
          number: 15012