consul-hs66r

==> Starting Consul agent...
           Version: '1.9.0'
           Node ID: 'c2ce972c-a438-dbeb-6a8c-fc696d3228f7'
         Node name: 'worker3.k8s'
        Datacenter: 'dc1' (Segment: '')
            Server: false (Bootstrap: false)
       Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 10.233.67.183 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
    2020-12-01T14:43:27.809Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
    2020-12-01T14:43:27.809Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
    2020-12-01T14:43:27.810Z [INFO]  agent: Started gRPC server: address=[::]:8502 network=tcp
    2020-12-01T14:43:27.810Z [INFO]  agent: Joining cluster...: cluster=LAN
    2020-12-01T14:43:27.810Z [INFO]  agent: started state syncer
    2020-12-01T14:43:27.810Z [WARN]  agent.router.manager: No servers available
    2020-12-01T14:43:27.809Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
    2020-12-01T14:43:27.810Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
    2020-12-01T14:43:27.810Z [INFO]  agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
==> Consul agent running!
    2020-12-01T14:43:27.810Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
    2020-12-01T14:43:27.807Z [WARN]  agent.auto_config: Node name "worker3.k8s" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
    2020-12-01T14:43:27.809Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T14:43:27.809Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
    2020-12-01T14:43:27.809Z [INFO]  agent.router: Initializing LAN area manager
    2020-12-01T14:43:27.815Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.152
    2020-12-01T14:43:27.815Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53
    2020-12-01T14:43:27.815Z [INFO]  agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
    2020-12-01T14:43:27.815Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T14:43:27.815Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a dead message (from: worker3.k8s)
    2020-12-01T14:43:27.815Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
    2020-12-01T14:43:27.815Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
    2020-12-01T14:43:27.883Z [INFO]  agent: Synced node info
    2020-12-01T14:43:27.890Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T14:43:27.898Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T14:43:30.835Z [INFO]  agent: (LAN) joined: number_of_nodes=2
    2020-12-01T14:43:30.835Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
    2020-12-01T14:43:32.869Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
    2020-12-01T14:43:32.869Z [INFO]  agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T14:43:45.599Z [INFO]  agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
    2020-12-01T14:43:45.598Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave: consul-server-1 10.233.69.152
    2020-12-01T14:43:50.471Z [INFO]  agent.client: New leader elected: payload=consul-server-0
    2020-12-01T14:44:07.013Z [INFO]  agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-1 from 10.233.69.152:8301 to 10.233.69.201:8301
    2020-12-01T14:44:07.013Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
    2020-12-01T14:44:07.013Z [INFO]  agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T14:44:18.880Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53
    2020-12-01T14:44:18.880Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
    2020-12-01T14:44:20.948Z [INFO]  agent.client: New leader elected: payload=consul-server-2
    2020-12-01T14:44:32.547Z [INFO]  agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301
    2020-12-01T14:44:32.547Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T14:44:32.547Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T14:44:40.071Z [INFO]  agent: Deregistered service: service=vault:10.233.67.174:8200
    2020-12-01T14:44:40.603Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T14:44:40.609Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T14:45:55.068Z [INFO]  agent: Deregistered service: service=vault:10.233.67.174:8200
    2020-12-01T14:45:55.531Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T14:45:55.538Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T14:47:10.079Z [INFO]  agent: Deregistered service: service=vault:10.233.67.174:8200
    2020-12-01T14:47:10.623Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T14:47:10.630Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T14:48:25.067Z [INFO]  agent: Deregistered service: service=vault:10.233.67.174:8200
    2020-12-01T14:48:25.595Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T14:48:25.603Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T14:48:56.182Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T14:48:56.241Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T14:48:56.249Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:01:44.011Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
    2020-12-01T15:01:44.809Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect worker1.k8s has failed, no acks received
    2020-12-01T15:01:49.809Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect worker2.k8s has failed, no acks received
    2020-12-01T15:01:50.005Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:01:50.014Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:01:51.014Z [ERROR] agent.client: RPC failed to server: method=Session.Apply server=10.233.67.184:8300 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
    2020-12-01T15:01:51.014Z [ERROR] agent.http: Request error: method=PUT url=/v1/session/create from=10.233.67.174:43772 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
    2020-12-01T15:02:06.131Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:02:06.143Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:02:12.810Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect worker1.k8s has failed, no acks received
    2020-12-01T15:02:13.947Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
    2020-12-01T15:02:16.811Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect consul-server-2 has failed, no acks received
    2020-12-01T15:02:22.496Z [INFO]  agent.client.memberlist.lan: memberlist: Marking consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:02:22.496Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-2 10.233.68.21
    2020-12-01T15:02:22.496Z [INFO]  agent.client: removing server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T15:02:22.693Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
    2020-12-01T15:02:22.693Z [INFO]  agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T15:02:23.811Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to worker2.k8s but other probes failed, network may be misconfigured
    2020-12-01T15:02:25.675Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:02:25.703Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:02:28.893Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
    2020-12-01T15:02:29.809Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect consul-server-1 has failed, no acks received
    2020-12-01T15:02:35.809Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect consul-server-2 has failed, no acks received
    2020-12-01T15:02:40.555Z [ERROR] agent.client: RPC failed to server: method=KVS.Apply server=10.233.69.201:8300 error="rpc error making call: rpc error making call: invalid session "ddd01a54-e467-6a93-49fd-417a68a3a471""
    2020-12-01T15:02:40.555Z [ERROR] agent.http: Request error: method=PUT url=/v1/kv/vault/core/lock?acquire=ddd01a54-e467-6a93-49fd-417a68a3a471&flags=3304740253564472344 from=10.233.67.174:43770 error="rpc error making call: rpc error making call: invalid session "ddd01a54-e467-6a93-49fd-417a68a3a471""
    2020-12-01T15:02:40.950Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
    2020-12-01T15:02:41.096Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T15:02:43.099Z [INFO]  agent: Deregistered service: service=vault:10.233.67.174:8200
    2020-12-01T15:02:44.377Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:02:44.403Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:02:53.076Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:46846 error="rpc error making call: No cluster leader"
    2020-12-01T15:02:53.076Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.67.184:8300 error="rpc error making call: No cluster leader"
    2020-12-01T15:02:58.748Z [INFO]  agent.client: New leader elected: payload=consul-server-2
    2020-12-01T15:02:59.265Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.67.184:8300 error="rpc error making call: No cluster leader"
    2020-12-01T15:02:59.265Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:47312 error="rpc error making call: No cluster leader"
    2020-12-01T15:03:00.120Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.68.21:8300 error="rpc error making call: No cluster leader"
    2020-12-01T15:03:00.120Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:47356 error="rpc error making call: No cluster leader"
    2020-12-01T15:03:01.334Z [ERROR] agent.client: RPC failed to server: method=KVS.Get server=10.233.68.21:8300 error="rpc error making call: No cluster leader"
    2020-12-01T15:03:01.334Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/sys/uiconfig_plaintext from=10.233.67.174:47406 error="rpc error making call: No cluster leader"
    2020-12-01T15:03:04.503Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:47584 error="rpc error making call: No cluster leader"
    2020-12-01T15:03:04.503Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.68.21:8300 error="rpc error making call: No cluster leader"
    2020-12-01T15:03:04.747Z [INFO]  agent.client: New leader elected: payload=consul-server-2
    2020-12-01T15:03:55.091Z [INFO]  agent: Deregistered service: service=vault:10.233.67.174:8200
    2020-12-01T15:03:55.576Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:03:55.585Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:04:25.801Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:04:25.867Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:04:25.879Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:08:01.809Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect consul-server-1 has failed, no acks received
    2020-12-01T15:08:02.083Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
    2020-12-01T15:08:05.809Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect worker1.k8s has failed, no acks received
    2020-12-01T15:08:09.711Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:08:09.723Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
    2020-12-01T15:08:10.716Z [ERROR] agent.client: RPC failed to server: method=Session.Apply server=10.233.67.184:8300 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
    2020-12-01T15:08:10.716Z [ERROR] agent.http: Request error: method=PUT url=/v1/session/create from=10.233.67.174:50438 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
    2020-12-01T15:08:10.748Z [INFO]  agent.client.memberlist.lan: memberlist: Marking consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:08:10.748Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-1 10.233.69.201
    2020-12-01T15:08:10.748Z [INFO]  agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T15:08:10.893Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
    2020-12-01T15:08:10.893Z [INFO]  agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T15:08:12.809Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to worker1.k8s but other probes failed, network may be misconfigured
    2020-12-01T15:08:25.808Z [INFO]  agent: Synced service: service=vault:10.233.67.174:8200
    2020-12-01T15:08:25.819Z [INFO]  agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check