using pilot on k8s for cf.md

Doc here with exact same stuff: https://docs.google.com/document/d/1mymZngAgbL1obqtsgBgVb8vlKFekKXIh73vgo7DndFU/edit

Goal: Configure BOSH-deployed copilot to send config over MCP to a Pilot deployed on a CFCR cluster and use the Istio Ingressgateway (Envoy) route to cf apps.

Steps:

1. Deploy latest CF deployment
2. Deploy CFCR cluster (don’t forget to run the apply-specs errand!) and deploy istio with a 1.1-snapshot.3 or greater (and change all the images to pull release-1.1-latest-daily)
3. Create a secret to host the cert, key, and ca needed for pilot to communicate with copilot over mutual TLS. This can be done by running the following:

kubectl create secret generic cf-mcp-certs --from-file=cert=/tmp/cert.pem --from-file=key=/tmp/key.pem --from-file=ca=/tmp/ca.pem -n istio-system

The ca, cert and key should come from credhub: (copilot.ca, pilot.cert, pilot.key). Copilot and Pilot should have the same ca, and have a cert and key generated from that ca.

4. Update the istio-pilot deployment to mount the secret to a specific location. You can pattern match from other volume_mounts with secrets

5. Update the istio configMap to include a configSources section that will let pilot know to connect to copilot. The updated mesh section of the configSource will look as follows:

  mesh: "# Set the following variable to true to disable policy checks by the Mixer.\n#
    Note that metrics will still be reported to the Mixer.\ndisablePolicyChecks: false\n\n#
    Set enableTracing to false to disable request tracing.\nenableTracing: true\n\n#
    Set accessLogFile to empty string to disable access log.\naccessLogFile: \"/dev/stdout\"\n\n#
    If accessLogEncoding is TEXT, value will be used directly as the log format\n#
    example: \"[%START_TIME%] %REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\\n\"\n#
    If AccessLogEncoding is JSON, value will be parsed as map[string]string\n# example:
    '{\"start_time\": \"%START_TIME%\", \"req_method\": \"%REQ(:METHOD)%\"}'\n# Leave
    empty to use default log format\naccessLogFormat: ''\n\n# Set accessLogEncoding
    to JSON or TEXT to configure sidecar access log\naccessLogEncoding: 'TEXT'\n\n#\n#
    Deprecated: mixer is using EDS\nmixerCheckServer: istio-policy.istio-system.svc.cluster.local:9091\nmixerReportServer:
    istio-telemetry.istio-system.svc.cluster.local:9091\n\n# policyCheckFailOpen allows
    traffic in cases when the mixer policy service cannot be reached.\n# Default is
    false which means the traffic is denied when the client is unable to connect to
    Mixer.\npolicyCheckFailOpen: false\n\n# Unix Domain Socket through which envoy
    communicates with NodeAgent SDS to get\n# key/cert for mTLS. Use secret-mount
    files instead of SDS if set to empty. \nsdsUdsPath: \n\n# This flag is used by
    secret discovery service(SDS). \n# If set to true(prerequisite: https://kubernetes.io/docs/concepts/storage/volumes/#projected),
    Istio will inject volumes mount \n# for k8s service account JWT, so that K8s API
    server mounts k8s service account JWT to envoy container, which \n# will be used
    to generate key/cert eventually. This isn't supported for non-k8s case.\nenableSdsTokenMount:
    false\n\n# This flag is used by secret discovery service(SDS). \n# If set to true,
    envoy will fetch normal k8s service account JWT from '/var/run/secrets/kubernetes.io/serviceaccount/token'
    \n# (https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod)
    \n# and pass to sds server, which will be used to request key/cert eventually.
    \n# this flag is ignored if enableSdsTokenMount is set.\n# This isn't supported
    for non-k8s case.\nsdsUseK8sSaJwt: false\n\n# The trust domain corresponds to
    the trust root of a system.\n# Refer to https://github.com/spiffe/spiffe/blob/master/standards/SPIFFE-ID.md#21-trust-domain\ntrustDomain:
    \n\n#\ndefaultConfig:\n  #\n  # TCP connection timeout between Envoy & the application,
    and between Envoys.\n  connectTimeout: 10s\n  #\n  ### ADVANCED SETTINGS #############\n
    \ # Where should envoy's configuration be stored in the istio-proxy container\n
    \ configPath: \"/etc/istio/proxy\"\n  binaryPath: \"/usr/local/bin/envoy\"\n  #
    The pseudo service name used for Envoy.\n  serviceCluster: istio-proxy\n  # These
    settings that determine how long an old Envoy\n  # process should be kept alive
    after an occasional reload.\n  drainDuration: 45s\n  parentShutdownDuration: 1m0s\n
    \ #\n  # The mode used to redirect inbound connections to Envoy. This setting\n
    \ # has no effect on outbound traffic: iptables REDIRECT is always used for\n
    \ # outbound connections.\n  # If \"REDIRECT\", use iptables REDIRECT to NAT and
    redirect to Envoy.\n  # The \"REDIRECT\" mode loses source addresses during redirection.\n
    \ # If \"TPROXY\", use iptables TPROXY to redirect to Envoy.\n  # The \"TPROXY\"
    mode preserves both the source and destination IP\n  # addresses and ports, so
    that they can be used for advanced filtering\n  # and manipulation.\n  # The \"TPROXY\"
    mode also configures the sidecar to run with the\n  # CAP_NET_ADMIN capability,
    which is required to use TPROXY.\n  #interceptionMode: REDIRECT\n  #\n  # Port
    where Envoy listens (on local host) for admin commands\n  # You can exec into
    the istio-proxy container in a pod and\n  # curl the admin port (curl http://localhost:15000/)
    to obtain\n  # diagnostic information from Envoy. See\n  # https://lyft.github.io/envoy/docs/operations/admin.html\n
    \ # for more details\n  proxyAdminPort: 15000\n  #\n  # Set concurrency to a specific
    number to control the number of Proxy worker threads.\n  # If set to 0 (default),
    then start worker thread for each CPU thread/core.\n  concurrency: 0\n  #\n  tracing:\n
    \   zipkin:\n      # Address of the Zipkin collector\n      address: zipkin.istio-system:9411\n
    \ #\n  # Mutual TLS authentication between sidecars and istio control plane.\n
    \ controlPlaneAuthPolicy: NONE\n  #\n  # Address where istio Pilot service is
    running\n  discoveryAddress: istio-pilot.istio-system:15010\nconfigSources:\n
    \ #\n  - address: copilot.service.cf.internal:9009\n    #\n    tlsSettings:\n      #\n
    \     mode: 2\n      clientCertificate: /etc/istio/cf-mcp-certs/cert\n      privateKey:
    /etc/istio/cf-mcp-certs/key\n      caCertificates: /etc/istio/cf-mcp-certs/ca"

6. Restart the pilot pod by deleting the existing one. Exec onto the pod and update the /etc/hosts file to resolve copilot.service.cf.internal to the internal ip of the istio-control vm. (OR TRY adding a service entry to resolve this address -- talk to Gabe for more info)

Right now, we see from looking at pilot logs that pilot is getting the config from copilot

curl localhost:9093/debug/configz

However, it looks like istio ingressgateway is not receiving this configuration from pilot. Once it does, it should be as simple as pointing the DNS entry for *.istio.<cf-system-domain at the istio ingressgateway (set up a load balancer to forward to a worker vm and port 31380).