Created
February 3, 2022 01:13
-
-
Save brkane/d14ae9e1d221a78dea475fe156c1ec1c to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
``` | |
❯ k logs -n linkerd linkerd-identity-cbd8f4795-65sq6 identity | |
time="2022-02-02T21:48:33Z" level=info msg="running version stable-2.11.1" | |
time="2022-02-02T21:48:43Z" level=fatal msg="Failed to initialize identity service: Post \"https://10.96.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": net/http: TLS handshake timeout" | |
``` | |
``` | |
❯ k logs -n linkerd linkerd-identity-cbd8f4795-65sq6 linkerd-proxy | |
[ 0.000610s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime | |
[ 0.000995s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191 | |
[ 0.001010s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143 | |
[ 0.001012s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140 | |
[ 0.001014s] INFO ThreadId(01) linkerd2_proxy: Tap DISABLED | |
[ 0.001016s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local | |
[ 0.001018s] INFO ThreadId(01) linkerd2_proxy: Identity verified via localhost:8080 | |
[ 0.001019s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local) | |
[ 0.001566s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 0.002021s] WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 0.004539s] WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 0.110848s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 0.326262s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 0.765891s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 1.266573s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 1.768262s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 2.269966s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 2.770686s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 3.272408s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 3.774148s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 4.274837s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 4.775523s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 5.023283s] WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 5.277220s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 5.777937s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 6.278652s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 6.780332s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 7.282002s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 7.782702s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 8.284187s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 8.784896s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 9.285623s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 9.787314s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 10.004435s] WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 10.289003s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 10.789695s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 11.291426s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 11.793143s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 12.293850s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 12.794543s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 13.296237s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 13.797542s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 14.299301s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 14.801015s] WARN ThreadId(02) identity:controller{addr=localhost:8080}:endpoint{addr=127.0.0.1:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 15.002254s] WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized... | |
[ 15.006727s] WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN | |
``` | |
BEFORE REMOVING `lifecycle` from `linkerd-destination` deployment: | |
``` | |
❯ k logs -n linkerd linkerd-destination-6bf7f47c5-tbd8z destination | |
Error from server (BadRequest): container "destination" in pod "linkerd-destination-6bf7f47c5-tbd8z" is waiting to start: ContainerCreating | |
❯ k logs -n linkerd linkerd-destination-6bf7f47c5-tbd8z policy | |
Error from server (BadRequest): container "policy" in pod "linkerd-destination-6bf7f47c5-tbd8z" is waiting to start: ContainerCreating | |
❯ k logs -n linkerd linkerd-destination-6bf7f47c5-tbd8z sp-validator | |
Error from server (BadRequest): container "sp-validator" in pod "linkerd-destination-6bf7f47c5-tbd8z" is waiting to start: ContainerCreating | |
❯ k logs -n linkerd linkerd-destination-6bf7f47c5-tbd8z linkerd-proxy | |
Error from server (BadRequest): container "linkerd-proxy" in pod "linkerd-destination-6bf7f47c5-tbd8z" is waiting to start: ContainerCreating | |
``` | |
AFTER REMOVING `lifecycle` from `linkerd-destination` deployment: | |
``` | |
k logs -n linkerd linkerd-destination-545949bc89-s8zxf linkerd-proxy | |
[ 0.000593s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime | |
[ 0.002014s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191 | |
[ 0.002029s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143 | |
[ 0.002031s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140 | |
[ 0.002033s] INFO ThreadId(01) linkerd2_proxy: Tap DISABLED | |
[ 0.002035s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local | |
[ 0.002037s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local) | |
[ 0.002039s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086 | |
[ 0.002352s] WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 0.003643s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 0.005645s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 0.113641s] WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 0.315799s] WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 0.751428s] WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 1.252130s] WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 1.753464s] WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 2.254190s] WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 4.042706s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 8.007896s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 12.005908s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 15.003071s] WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized... | |
[ 16.005993s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 20.030222s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 24.006444s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 28.005689s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 30.004269s] WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized... | |
[ 32.022029s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 36.023238s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 40.070962s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 40.082593s] WARN ThreadId(01) policy:watch{port=9996}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Service failed error=channel closed | |
[ 40.190116s] WARN ThreadId(01) policy:watch{port=9996}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) | |
[ 44.031032s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN | |
[ 45.005215s] WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized... | |
❯ k logs -n linkerd linkerd-destination-545949bc89-s8zxf destination | |
time="2022-02-03T00:52:21Z" level=info msg="running version stable-2.11.1" | |
time="2022-02-03T00:52:21Z" level=info msg="Using default opaque ports: map[25:{} 587:{} 3306:{} 4444:{} 5432:{} 6379:{} 9300:{} 11211:{}]" | |
time="2022-02-03T00:53:21Z" level=fatal msg="Failed to initialize K8s API: Post \"https://10.96.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": dial tcp 10.96.0.1:443: i/o timeout" | |
❯ k logs -n linkerd linkerd-destination-545949bc89-s8zxf sp-validator | |
time="2022-02-03T00:52:21Z" level=info msg="running version stable-2.11.1" | |
time="2022-02-03T00:52:21Z" level=info msg="waiting for caches to sync" | |
time="2022-02-03T00:52:21Z" level=info msg="caches synced" | |
time="2022-02-03T00:52:21Z" level=info msg="starting admin server on :9997" | |
time="2022-02-03T00:52:21Z" level=info msg="listening at :8443" | |
time="2022-02-03T00:53:21Z" level=info msg="shutting down webhook server" | |
❯ k logs -n linkerd linkerd-destination-545949bc89-s8zxf policy | |
2022-02-03T00:53:52.570182Z INFO serve{addr=0.0.0.0:9990}: linkerd_policy_controller::admin: HTTP admin server listening addr=0.0.0.0:9990 | |
2022-02-03T00:53:52.570227Z INFO grpc{addr=0.0.0.0:8090 cluster_networks=[10.0.0.0/8, 100.64.0.0/10, 172.16.0.0/12, 192.168.0.0/16]}: linkerd_policy_controller: gRPC server listening addr=0.0.0.0:8090 | |
2022-02-03T00:53:52.570628Z INFO linkerd_policy_controller: Admission controller server listening addr=0.0.0.0:9443 | |
2022-02-03T00:54:32.362292Z INFO linkerd_policy_controller: Shutting down | |
``` |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment