Skip to content

Instantly share code, notes, and snippets.

@howardjohn howardjohn/README.md

Last active May 6, 2020
Embed
What would you like to do?
Control plane routing of XDS

XDS routing by gateway

The goal of this is to make revision transparent to user. Rather than specifying a revision as part of the pod/namespace, we will do routing at a gateway based on some metadata. For this example we will user an arbitrary setting we configure on the pod - in practice we would likely use something like ISTIO_VERSION which is configured automatically for the user.

To deploy this, add the following annotation to a pod:

        annotations:
          proxy.istio.io/config: |
            discoveryAddress: istio-ingressgateway.istio-system.svc:443
            controlPlaneAuthPolicy: MUTUAL_TLS
            proxyMetadata:
              ROUTER: istiod.istio-system.svc
          sidecar.istio.io/proxyImage: gcr.io/howardjohn-istio/proxyv2:1588805810

And another pod:

        annotations:
          proxy.istio.io/config: |
            discoveryAddress: istio-ingressgateway.istio-system.svc:443
            controlPlaneAuthPolicy: MUTUAL_TLS
            proxyMetadata:
              ROUTER: istiod-canary.istio-system.svc
          sidecar.istio.io/proxyImage: gcr.io/howardjohn-istio/proxyv2:1588805810

The first one will use the default revision, the second the canary revision.

In a real setup, this would likely be derived from something automatically - no user configure here would be needed, as that defeats the purpose. To make things simple here (and allow testing with two pods running the same version) we explicitly define some metadata to drive routing decisions.

Next we need to set up config for the gateway. Apply tls.yaml, which will do SNI routing to our different control plane pods based on this.

Note that this requires a custom proxy image to:

  • Set the value of ROUTER to sni.
  • Change match_subject_alt_names to *.istio-system.svc. Making this actually secure is TBD.

In a real deployment, we might choose to route based on other properties:

  • ISTIO_VERSION - 1.x pods go to 1.x control plane, 1.y -> 1.y
  • Really fine grained - pod name, deployment name, etc. This is a bit harder with TLS than plaintext since we only have one SNI value we can set. With plaintext we can set N headers, and routing can choose to use any combination of them
diff --git a/pkg/bootstrap/config.go b/pkg/bootstrap/config.go
index a1a042f9f..3c58705b9 100644
--- a/pkg/bootstrap/config.go
+++ b/pkg/bootstrap/config.go
@@ -140,7 +140,7 @@ func (cfg Config) toTemplateParams() (map[string]interface{}, error) {
return nil, err
}
opts = append(opts, getNodeMetadataOptions(meta, rawMeta, cfg.PlatEnv, cfg.Proxy)...)
-
+ opts = append(opts, option.Sni(cfg.Proxy.ProxyMetadata["ROUTER"]))
// Check if nodeIP carries IPv4 or IPv6 and set up proxy accordingly
if isIPv6Proxy(cfg.NodeIPs) {
opts = append(opts,
diff --git a/pkg/bootstrap/option/instances.go b/pkg/bootstrap/option/instances.go
index e6f97761c..6d229fdb8 100644
--- a/pkg/bootstrap/option/instances.go
+++ b/pkg/bootstrap/option/instances.go
@@ -78,6 +78,10 @@ func SubZone(value string) Instance {
return newOptionOrSkipIfZero("sub_zone", value)
}
+func Sni(value string) Instance {
+ return newOptionOrSkipIfZero("sni", value)
+}
+
func NodeMetadata(meta *model.NodeMetadata, rawMeta map[string]interface{}) Instance {
return newOptionOrSkipIfZero("meta_json_str", meta).withConvert(nodeMetadataConverter(meta, rawMeta))
}
diff --git a/tools/packaging/common/envoy_bootstrap_v2.json b/tools/packaging/common/envoy_bootstrap_v2.json
index 5a8cca471..1e5d9bc4b 100644
--- a/tools/packaging/common/envoy_bootstrap_v2.json
+++ b/tools/packaging/common/envoy_bootstrap_v2.json
@@ -214,6 +214,7 @@
"name": "envoy.transport_sockets.tls",
"typed_config": {
"@type": "type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext",
+ "sni": "{{.sni}}",
"common_tls_context": {
"alpn_protocols": [
"h2"
@@ -243,7 +244,7 @@
"filename": "./var/run/secrets/istio/root-cert.pem"
{{- end }}
},
- "match_subject_alt_names": {{ .pilot_SAN }}
+ "match_subject_alt_names": [{"suffix": ".istio-system.svc"}]
}
}
}
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: controlplane-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: PASSTHROUGH
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: controlplane
namespace: istio-system
spec:
hosts:
- "*"
gateways:
- controlplane-gateway
tls:
- match:
- sniHosts:
- istiod-canary.istio-system.svc
route:
- destination:
host: istiod-canary
port:
number: 15012
- match:
- sniHosts:
- "*"
route:
- destination:
host: istiod
port:
number: 15012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.