Cluster spec:
gcloud beta container --project "dx-stefan" clusters create "istio-eu" --zone "europe-west3-a" \
--cluster-version "1.11.6-gke.6" --machine-type "n1-standard-2" --image-type "COS" \
--no-enable-basic-auth --disk-type "pd-standard" --disk-size "50" \
--num-nodes "1" --additional-zones "europe-west3-a","europe-west3-b" \
--no-enable-cloud-logging --enable-cloud-monitoring \
--enable-ip-alias --default-max-pods-per-node "110" \
--addons HorizontalPodAutoscaling,Istio --istio-config=auth=MTLS_PERMISSIVEHPA status:
kubectl -n istio-system get horizontalpodautoscalers.autoscaling
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
istio-egressgateway Deployment/istio-egressgateway <unknown>/80% 1 5 1 1d
istio-ingressgateway Deployment/istio-ingressgateway <unknown>/80% 1 5 1 1d
istio-pilot Deployment/istio-pilot <unknown>/80% 1 5 1 1d
istio-policy Deployment/istio-policy <unknown>/80% 1 5 1 1d
istio-telemetry Deployment/istio-telemetry <unknown>/80% 1 5 1 1d
HPA describe:
kubectl -n istio-system describe horizontalpodautoscalers.autoscaling istio-ingressgateway
Error from server (NotFound): the server could not find the requested resource
HPA error:
the HPA was unable to compute the replica count: missing request for cpu on container
istio-proxy in pod istio-system/istio-ingressgateway-774d77cb7c-slbgv
Egress blocked:
export REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml && \
kubectl -n test apply -f ${REPO}/artifacts/loadtester/deployment.yaml
kubectl -n test exec -it flagger-loadtester-xxx-xxx sh
/home/app $ curl -v google.com
< HTTP/1.1 404 Not Found
< date: Tue, 05 Feb 2019 17:56:02 GMT
< server: envoy
< content-length: 0
Add service entry:
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: httpbin-ext
spec:
hosts:
- httpbin.org
ports:
- number: 80
name: http
protocol: HTTP
- number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
EOF
Istio headers leaked outside the mesh:
/home/app $ curl httpbin.org/headers
{
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.61.1",
"X-B3-Sampled": "0",
"X-B3-Spanid": "6a790274908e70c3",
"X-B3-Traceid": "6a790274908e70c3",
"X-Envoy-Decorator-Operation": "httpbin.org:80/*",
"X-Istio-Attributes": "CikKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRINEgtodHRwYmluLm9yZwoqCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIJEgdkZWZhdWx0CiQKE2Rlc3RpbmF0aW9uLnNlcnZpY2USDRILaHR0cGJpbi5vcmcKRQoKc291cmNlLnVpZBI3EjVrdWJlcm5ldGVzOi8vZmxhZ2dlci1sb2FkdGVzdGVyLTc1ODU5ODc0OWYta2p4a2YudGVzdAopChhkZXN0aW5hdGlvbi5zZXJ2aWNlLmhvc3QSDRILaHR0cGJpbi5vcmc="
}
}
Istio telemetry is crashing, Prom can't scrape it due to Stackdriver errors:
2019-02-09T15:50:17.658452Z error adapters Stackdriver returned: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (zone).: timeSeries[65,66]
2019-02-09T15:50:19.318587Z info transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-02-09T15:50:19.318630Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420e123d0, TRANSIENT_FAILURE
2019-02-09T15:50:19.318647Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420e123d0, CONNECTING
2019-02-09T15:50:19.350937Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420e123d0, READY
2019-02-09T15:50:19.432513Z info transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-02-09T15:50:19.433779Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420e286b0, TRANSIENT_FAILURE
2019-02-09T15:50:19.434030Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420e286b0, CONNECTING
2019-02-09T16:15:22.115527Z error adapters Stackdriver returned: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (zone).: timeSeries[33,59]
2019-02-09T16:15:17.916663Z info OpenCensus Stackdriver exporter: failed to upload span: buffer full
2019-02-09T16:28:14.666798Z info OpenCensus Stackdriver exporter: failed to upload 970 spans: buffer full
gc 16 @112.807s 3%: 0.10+629+422 ms clock, 0.20+304/329/81+844 ms cpu, 397->412->241 MB, 418 MB goal, 2 P
2019-02-09T16:28:20.023145Z info OpenCensus Stackdriver exporter: failed to upload 1069 spans: buffer full
2019-02-09T16:28:20.879581Z error adapters Stackdriver returned: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (zone).: timeSeries[0,2,4]
Istio ingress does not preserve the client IP address, the svc/istio-ingressgateway externalTrafficPolicy: Cluster should be externalTrafficPolicy: Local see istio/istio#7607