See Issue #22367 on the Istio issue tracker
NOTE: All snippets are "copy-paste optimized". If you set a few environment variables (that will change if this gets run again), you can just copy-paste most commands. Additionally commands will not be prefixed with a
PS1like$; instead STDOUT / STDERR will be prefixed with a "comment"#.
- Download 1.5.0 Release
- Create A Test Namespace
- Create a Service in the Mesh
- Create a
Sidecarto Limit the Discovered Services - Spin Up Raw TCP Server
- Create a Kubernetes Service That "Breaks"
${TCP_PORT} - Verify TCP Traffic Gets Blackholed
- Using a Non-HTTP Protocol
- Removing Listener
- Clean Up
- Epilogue: Other Sources of
0.0.0.0_${PORT}Listeners
OS=osx
curl \
--location "https://github.com/istio/istio/releases/download/1.5.0/istio-1.5.0-${OS}.tar.gz" \
--output "istio-1.5.0-${OS}.tar.gz"
tar xzvf "istio-1.5.0-${OS}.tar.gz"
rm -f "istio-1.5.0-${OS}.tar.gz"
./istio-1.5.0/bin/istioctl --help
NAMESPACE=dhermes-repro
kubectl create namespace "${NAMESPACE}"
kubectl label namespace "${NAMESPACE}" istio-injection=enabled
Deploy the sleep service from the docs:
./istio-1.5.0/bin/istioctl kube-inject \
--filename ./istio-1.5.0/samples/sleep/sleep.yaml \
> sleep-injected.yml
kubectl apply --namespace "${NAMESPACE}" --filename ./sleep-injected.yml
Take note of the pod that was created:
kubectl get pods --namespace "${NAMESPACE}"
# NAME READY STATUS RESTARTS AGE
# sleep-54c989bc97-lks72 2/2 Running 0 44s
POD_NAME=sleep-54c989bc97-lks72
kubectl exec --namespace "${NAMESPACE}" "${POD_NAME}" --container sleep -- hostname
# sleep-54c989bc97-lks72
kubectl exec --namespace "${NAMESPACE}" "${POD_NAME}" --container sleep -- hostname -i
# 10.101.151.188
Before applying note the number of discovered services in the cluster
./istio-1.5.0/bin/istioctl proxy-config listeners \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--output json \
| jq '. | length'
# 468
We can limit egress with sidecar.yml
---
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
name: limited-egress
spec:
egress:
- hosts:
- istio-system/*
- ./*After applying we can see the number of listeners drop:
kubectl apply --namespace "${NAMESPACE}" --filename ./sidecar.yml
./istio-1.5.0/bin/istioctl proxy-config listeners \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--output json \
> listeners-000.json
/bin/cat listeners-000.json | jq '. | length'
# 15
/bin/cat listeners-000.json | jq '.[].name' -r
# 10.101.151.188_80 # Pod IP for sleep.${NAMESPACE} Service
# 10.101.151.188_15020 # Pod IP for sleep.${NAMESPACE} Service
# 10.101.22.111_15011 # Cluster IP for istio-pilot.istio-system Service
# 10.101.22.111_15012 # Cluster IP for istio-pilot.istio-system Service
# 10.101.28.8_15012 # Cluster IP for istiod.istio-system Service
# 10.101.28.8_443 # Cluster IP for istiod.istio-system Service
# 10.101.22.111_443 # Cluster IP for istio-pilot.istio-system Service
# 0.0.0.0_15014 #
# 0.0.0.0_8080 #
# 0.0.0.0_15010 #
# 0.0.0.0_20001 #
# 0.0.0.0_80 #
# virtualOutbound
# virtualInbound
# null
I wanted to expose a raw TCP server to the outside internet to showcase how
an Istio bug causes a BlackHoleCluster Envoy listener to swallow non-HTTP
traffic for a given port.
There are plenty of other ways to expose a raw TCP server, I chose to use
ngrok just to expose a port via nc -l 12892 to the outside internet. (There
are plenty of other ways to expose a raw TCP server, this is just an example.)
./ngrok authtoken ... # "TCP tunnels are only available after you sign up"
./ngrok tcp 12892
# ngrok by @inconshreveable (Ctrl+C to quit)
#
# Session Status online
# Account (Plan: Business)
# Version 2.3.35
# Region United States (us)
# Web Interface http://127.0.0.1:4040
# Forwarding tcp://0.tcp.ngrok.io:13602 -> localhost:12892
#
# Connections ttl opn rt1 rt5 p50 p90
# 0 0 0.00 0.00 0.00 0.00
To confirm it is running
TCP_HOSTNAME=0.tcp.ngrok.io
TCP_PORT=13602
echo "did you make it?" | nc ${TCP_HOSTNAME} ${TCP_PORT} # Client
nc -l 12892 # Server
# did you make it?
and also we want to confirm that the sleep pod in the Istio service mesh
can hit the TCP server
kubectl exec `# Client` \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--container sleep \
-- /bin/sh -c 'echo "$(hostname):$(hostname -i)" | nc '"${TCP_HOSTNAME} ${TCP_PORT}"
nc -l 12892 # Server
# sleep-54c989bc97-lks72:10.101.151.188
and with an HTTP client
kubectl exec `# Client` \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--container sleep \
-- /bin/sh -c \
'curl --silent --verbose --max-time 2 --header "pod: $(hostname)" http://'"${TCP_HOSTNAME}:${TCP_PORT}"
# * Expire in 0 ms for 6 (transfer 0x5638a5d08680)
# * Expire in 2000 ms for 8 (transfer 0x5638a5d08680)
# ...
# * Trying 3.17.202.129...
# * TCP_NODELAY set
# * Expire in 200 ms for 4 (transfer 0x5638a5d08680)
# * Connected to 0.tcp.ngrok.io (3.17.202.129) port 13602 (#0)
# > GET / HTTP/1.1
# > Host: 0.tcp.ngrok.io:13602
# > User-Agent: curl/7.64.0
# > Accept: */*
# > pod: sleep-54c989bc97-lks72
# >
# * Operation timed out after 2001 milliseconds with 0 bytes received
# * Closing connection 0
# command terminated with exit code 28
nc -l 12892 # Server
# GET / HTTP/1.1
# Host: 0.tcp.ngrok.io:13602
# User-Agent: curl/7.64.0
# Accept: */*
# pod: sleep-54c989bc97-lks72
#
We can create a service via break-template.yml
---
apiVersion: v1
kind: Service
metadata:
name: break
spec:
ports:
- name: http-break
port: ${TCP_PORT}
protocol: TCP
type: ClusterIPwhich will cause a new Envoy listener to be created:
sed s/'${TCP_PORT}'/${TCP_PORT}/g break-template.yml > break.yml
kubectl apply --namespace "${NAMESPACE}" --filename ./break.yml
./istio-1.5.0/bin/istioctl proxy-config listeners \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--output json \
> listeners-001.json
/bin/cat listeners-001.json | jq '. | length'
# 16
/bin/cat listeners-001.json | jq '.[].name' -r
# 10.101.151.188_80 # Pod IP for sleep.${NAMESPACE} Service
# ...
# virtualInbound
# 0.0.0.0_13602 # Newly added listener
# null
LISTENER_INDEX=14 # Next to last
/bin/cat listeners-001.json | jq ".[${LISTENER_INDEX}]" > 0.0.0.0_${TCP_PORT}_http.json
This listener has a BlackHoleCluster filter chain with an
envoy.tcp_proxy filter (of type tcp_proxy.v2.TcpProxy):
/bin/cat listeners-001.json | jq ".[${LISTENER_INDEX}].name" -r
# 0.0.0.0_13602
/bin/cat listeners-001.json | jq ".[${LISTENER_INDEX}].filterChains[0].filters[1]"
# {
# "name": "envoy.tcp_proxy",
# "typedConfig": {
# "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
# "statPrefix": "BlackHoleCluster",
# "cluster": "BlackHoleCluster"
# }
# }
After the 0.0.0.0_${TCP_PORT} Envoy listener gets created, any outgoing
requests to ${TCP_PORT} will match that listener. Making the same request
in the pod
kubectl exec \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--container sleep \
-- /bin/sh -c 'echo "$(hostname):$(hostname -i)" | nc '"${TCP_HOSTNAME} ${TCP_PORT}"
the client receives nothing. If instead we make an HTTP request, it goes
through just fine (even if the nc -l server can't provide an HTTP
response):
kubectl exec `# Client` \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--container sleep \
-- /bin/sh -c \
'curl --silent --verbose --max-time 2 --header "pod: $(hostname)" http://'"${TCP_HOSTNAME}:${TCP_PORT}"
# * Expire in 0 ms for 6 (transfer 0x5591e81bd680)
# * Expire in 2000 ms for 8 (transfer 0x5591e81bd680)
# ...
# * Trying 3.134.196.116...
# * TCP_NODELAY set
# * Expire in 200 ms for 4 (transfer 0x5591e81bd680)
# * Connected to 0.tcp.ngrok.io (3.134.196.116) port 13602 (#0)
# > GET / HTTP/1.1
# > Host: 0.tcp.ngrok.io:13602
# > User-Agent: curl/7.64.0
# > Accept: */*
# > pod: sleep-54c989bc97-lks72
# >
# * Operation timed out after 2001 milliseconds with 0 bytes received
# * Closing connection 0
# command terminated with exit code 28
nc -l 12892 # Server
# GET / HTTP/1.1
# host: 0.tcp.ngrok.io:13602
# user-agent: curl/7.64.0
# accept: */*
# pod: sleep-54c989bc97-lks72
# x-forwarded-for: 10.101.151.188
# x-forwarded-proto: http
# x-envoy-internal: true
# x-request-id: d9be301d-d90d-9f3b-afeb-aa9f7011fe78
# x-envoy-peer-metadata: CiAK...
# x-envoy-peer-metadata-id: sidecar~10.101.151.188~sleep-54c989bc97-lks72.dhermes-repro~dhermes-repro.svc.cluster.local
# x-b3-traceid: 875fdf0be92d99a619b41aa11660abcf
# x-b3-spanid: 19b41aa11660abcf
# x-b3-sampled: 1
# content-length: 0
#
The core of the issue is the http- prefix in the port name (http-break).
By using it, we have opted into "manual protocol selection" and this
is where the bug resides. If any of the HTTP-like prefixes are used
(http-, http2-, grpc-) a BlackHoleCluster Envoy listener will get
created on the "any IPv4 address at all" IP 0.0.0.0.
If instead we apply a patch to rename the port, the ${TCP_PORT} Envoy
listener will change
JSON_PATCH="{\"spec\":{\"ports\":[{\"name\":\"tcp-break\",\"port\":${TCP_PORT}}]}}"
echo "${JSON_PATCH}" | jq
# {
# "spec": {
# "ports": [
# {
# "name": "tcp-break",
# "port": 13602
# }
# ]
# }
# }
kubectl patch service \
--namespace "${NAMESPACE}" \
break \
--patch "${JSON_PATCH}"
./istio-1.5.0/bin/istioctl proxy-config listeners \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--output json \
> listeners-002.json
/bin/cat listeners-002.json | jq '. | length'
# 16
/bin/cat listeners-002.json | jq '.[].name' -r
# 10.101.151.188_80
# ...
# virtualInbound
# 10.101.20.10_13602 # Cluster IP for break.${NAMESPACE} Service
# null
kubectl get service --namespace "${NAMESPACE}" break
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# break ClusterIP 10.101.20.10 <none> 13602/TCP 34m
the outgoing traffic from that port will now go through
kubectl exec `# Client` \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--container sleep \
-- /bin/sh -c 'echo "$(hostname):$(hostname -i)" | nc '"${TCP_HOSTNAME} ${TCP_PORT}"
nc -l 12892 # Server
# sleep-54c989bc97-lks72:10.101.151.188
and the outgoing HTTP request will not have any of the Envoy modifications
kubectl exec `# Client` \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--container sleep \
-- /bin/sh -c \
'curl --silent --verbose --max-time 2 --header "pod: $(hostname)" http://'"${TCP_HOSTNAME}:${TCP_PORT}"
# * Expire in 0 ms for 6 (transfer 0x5570c6476680)
# * Expire in 2000 ms for 8 (transfer 0x5570c6476680)
# ...
# * Trying 3.13.191.225...
# * TCP_NODELAY set
# * Expire in 200 ms for 4 (transfer 0x5570c6476680)
# * Connected to 0.tcp.ngrok.io (3.13.191.225) port 13602 (#0)
# > GET / HTTP/1.1
# > Host: 0.tcp.ngrok.io:13602
# > User-Agent: curl/7.64.0
# > Accept: */*
# > pod: sleep-54c989bc97-lks72
# >
# * Operation timed out after 2000 milliseconds with 0 bytes received
# * Closing connection 0
# command terminated with exit code 28
nc -l 12892 # Server
# GET / HTTP/1.1
# Host: 0.tcp.ngrok.io:13602
# User-Agent: curl/7.64.0
# Accept: */*
# pod: sleep-54c989bc97-lks72
#
By deleting the break service the Envoy listener on ${TCP_PORT} will be
removed as well
kubectl delete --namespace "${NAMESPACE}" --filename ./break.yml
./istio-1.5.0/bin/istioctl proxy-config listeners \
--namespace "${NAMESPACE}" \
"${POD_NAME}" \
--output json \
> listeners-003.json
/bin/cat listeners-003.json | jq '. | length'
# 15 # i.e. the original amount
kubectl delete --namespace "${NAMESPACE}" --filename ./sidecar.yml
kubectl delete --namespace "${NAMESPACE}" --filename ./sleep-injected.yml
kubectl delete namespace "${NAMESPACE}"
rm -f \
sleep-injected.yml \
sidecar.yml \
listeners-000.json \
break-template.yml \
break.yml \
listeners-001.json \
0.0.0.0_${TCP_PORT}_http.json \
listeners-002.json \
listeners-003.json
rm -fr istio-1.5.0/ ngrok # If you must