Skip to content

Instantly share code, notes, and snippets.

@vector623
Created March 25, 2020 18:59
Show Gist options
  • Save vector623/d193f3292790bf7f1119c57bafd4e561 to your computer and use it in GitHub Desktop.
Save vector623/d193f3292790bf7f1119c57bafd4e561 to your computer and use it in GitHub Desktop.
pulled from journalctl
-- Logs begin at Thu 2020-02-20 21:03:22 UTC. --
Mar 25 18:54:47 srv-pro-schd-05 consul[30217]: 2020-03-25T18:54:47.382Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:54:57 srv-pro-schd-05 consul[30217]: 2020-03-25T18:54:57.382Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:55:06 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:06.607Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
Mar 25 18:55:06 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:06.608Z [DEBUG] agent: Node info in sync
Mar 25 18:55:07 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:07.361Z [DEBUG] agent.server: Skipping self join check for node since the cluster is too small: node=srv-pro-svrg-01
Mar 25 18:55:07 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:07.382Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:55:17 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:17.382Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:55:18 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:18.187Z [DEBUG] agent.server.router.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="srv-pro-svrg-01.dc1 (Addr: tcp/127.0.0.1:8300) (DC: dc1)"
Mar 25 18:55:27 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:27.382Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:55:37 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:37.382Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:55:47 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:47.382Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:55:48 srv-pro-schd-05 systemd[1]: Stopping "HashiCorp Consul - A service mesh solution"...
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [INFO] agent: Caught: signal=terminated
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [INFO] agent: Graceful shutdown disabled. Exiting
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [INFO] agent: Requesting shutdown
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [INFO] agent.server: shutting down server
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [DEBUG] agent.leader: stopping routine: routine="CA root pruning"
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [WARN] agent.server.serf.lan: serf: Shutdown without a Leave
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [DEBUG] agent.leader: stopped routine: routine="CA root pruning"
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.641Z [WARN] agent.server.serf.wan: serf: Shutdown without a Leave
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.642Z [INFO] agent.server.router.manager: shutting down
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.642Z [INFO] agent: consul server down
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.642Z [INFO] agent: shutdown complete
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.642Z [INFO] agent: Stopping server: protocol=DNS address=127.0.0.1:8600 network=tcp
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.642Z [INFO] agent: Stopping server: protocol=DNS address=127.0.0.1:8600 network=udp
Mar 25 18:55:48 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:48.642Z [INFO] agent: Stopping server: protocol=HTTP address=127.0.0.1:8500 network=tcp
Mar 25 18:55:49 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:49.642Z [WARN] agent: Timeout stopping server: protocol=HTTP address=127.0.0.1:8500 network=tcp
Mar 25 18:55:49 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:49.642Z [INFO] agent: Waiting for endpoints to shut down
Mar 25 18:55:49 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:49.643Z [INFO] agent: Endpoints down
Mar 25 18:55:49 srv-pro-schd-05 consul[30217]: 2020-03-25T18:55:49.643Z [INFO] agent: Exit code: code=1
Mar 25 18:55:49 srv-pro-schd-05 systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
Mar 25 18:55:49 srv-pro-schd-05 systemd[1]: consul.service: Failed with result 'exit-code'.
Mar 25 18:55:49 srv-pro-schd-05 systemd[1]: Stopped "HashiCorp Consul - A service mesh solution".
Mar 25 18:55:49 srv-pro-schd-05 systemd[1]: Starting "HashiCorp Consul - A service mesh solution"...
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: bootstrap = true: do not enable unless necessary
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: ==> Starting Consul agent...
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Version: 'v1.7.2'
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Node ID: 'afca1171-75ff-7598-c1a2-8446f418c001'
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Node name: 'srv-pro-svrg-01'
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Datacenter: 'dc1' (Segment: '<all>')
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Server: true (Bootstrap: true)
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: ==> Log data will now stream in as it occurs:
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.726Z [DEBUG] agent.tlsutil: Update: version=1
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.727Z [DEBUG] agent.tlsutil: OutgoingRPCWrapper: version=1
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.734Z [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:afca1171-75ff-7598-c1a2-8446f418c001 Address:10.0.20.145:8300}]"
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.735Z [INFO] agent.server.raft: entering follower state: follower="Node at 127.0.0.1:8300 [Follower]" leader=
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.735Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: srv-pro-svrg-01.dc1 127.0.0.1
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.736Z [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: srv-pro-schd-05.dc1: 127.0.0.1:8302
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.736Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: srv-pro-svrg-01 127.0.0.1
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.736Z [DEBUG] agent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:8302
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.736Z [DEBUG] agent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:36494
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.736Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=udp
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent.server: Handled event for server in area: event=member-join server=srv-pro-svrg-01.dc1 area=wan
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: srv-pro-schd-05: 127.0.0.1:8301
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent.server: Adding LAN server: server="srv-pro-svrg-01 (Addr: tcp/127.0.0.1:8300) (DC: dc1)"
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=tcp
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent: Started HTTP server: address=127.0.0.1:8500 network=tcp
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent: started state syncer
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: ==> Consul agent running!
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent.server.serf.wan: serf: Re-joined to previously known node: srv-pro-schd-05.dc1: 127.0.0.1:8302
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent: Joining cluster...: cluster=LAN
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.737Z [INFO] agent: (LAN) joining: lan_addresses=[localhost]
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.738Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:8301
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.739Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:60568
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.739Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:60566
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.740Z [INFO] agent.server.serf.lan: serf: Re-joined to previously known node: srv-pro-schd-05: 127.0.0.1:8301
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.740Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:8301
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.741Z [INFO] agent: (LAN) joined: number_of_nodes=1
Mar 25 18:55:49 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:49.741Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1
Mar 25 18:55:49 srv-pro-schd-05 systemd[1]: Started "HashiCorp Consul - A service mesh solution".
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.911Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.911Z [INFO] agent.server.raft: entering candidate state: node="Node at 127.0.0.1:8300 [Candidate]" term=19
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.914Z [DEBUG] agent.server.raft: votes: needed=1
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.914Z [DEBUG] agent.server.raft: vote granted: from=afca1171-75ff-7598-c1a2-8446f418c001 term=19 tally=1
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.914Z [INFO] agent.server.raft: election won: tally=1
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.914Z [INFO] agent.server.raft: entering leader state: leader="Node at 127.0.0.1:8300 [Leader]"
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.914Z [INFO] agent.server: cluster leadership acquired
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.915Z [INFO] agent.server: New leader elected: payload=srv-pro-svrg-01
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: Processing server acl mode for: srv-pro-svrg-01 - 0
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.934Z [INFO] agent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:8300
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.934Z [INFO] agent.leader: started routine: routine="CA root pruning"
Mar 25 18:55:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:50.935Z [DEBUG] agent.server: Skipping self join check for node since the cluster is too small: node=srv-pro-svrg-01
Mar 25 18:55:51 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:51.014Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
Mar 25 18:55:51 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:51.015Z [INFO] agent: Synced node info
Mar 25 18:55:51 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:51.419Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
Mar 25 18:55:51 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:51.419Z [DEBUG] agent: Node info in sync
Mar 25 18:55:52 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:52.935Z [DEBUG] agent.tlsutil: OutgoingRPCWrapper: version=1
Mar 25 18:55:59 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:59.238Z [DEBUG] agent.http: Request finished: method=GET url=/v1/catalog/nodes?dc=dc1&stale= from=127.0.0.1:38750 latency=988.061µs
Mar 25 18:55:59 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:59.240Z [DEBUG] agent.http: Request finished: method=GET url=/v1/catalog/service/redis?dc=dc1 from=127.0.0.1:38752 latency=688.476µs
Mar 25 18:55:59 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:59.720Z [DEBUG] agent.http: Request finished: method=GET url=/v1/catalog/node/srv-pro-svrg-01?dc=dc1 from=127.0.0.1:38754 latency=517.549µs
Mar 25 18:55:59 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:59.756Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/catalog/register?dc=dc1 from=127.0.0.1:38756 latency=20.196985ms
Mar 25 18:55:59 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:59.844Z [DEBUG] agent.http: Request finished: method=GET url=/v1/catalog/service/redis?dc=dc1 from=127.0.0.1:38758 latency=277.883µs
Mar 25 18:55:59 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:59.869Z [DEBUG] agent.http: Request finished: method=GET url=/v1/health/checks/redis?dc=dc1 from=127.0.0.1:38760 latency=301.978µs
Mar 25 18:55:59 srv-pro-schd-05 consul[32576]: 2020-03-25T18:55:59.968Z [DEBUG] agent.http: Request finished: method=GET url=/v1/catalog/service/redis?dc=dc1 from=127.0.0.1:38762 latency=338.565µs
Mar 25 18:56:00 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:00.001Z [DEBUG] agent.http: Request finished: method=GET url=/v1/health/checks/redis?dc=dc1 from=127.0.0.1:38764 latency=277.105µs
Mar 25 18:56:00 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:00.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:56:08 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:08.389Z [ERROR] agent: Failed to check for updates: error="Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=db4557b9-6e10-9c45-d491-934716eb0c98&version=1.7.2: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 25 18:56:10 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:10.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:56:20 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:20.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:56:30 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:30.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:56:40 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:40.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:56:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:50.916Z [DEBUG] agent.server: Skipping self join check for node since the cluster is too small: node=srv-pro-svrg-01
Mar 25 18:56:50 srv-pro-schd-05 consul[32576]: 2020-03-25T18:56:50.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:57:00 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:00.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:57:10 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:10.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:57:20 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:20.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:57:23 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:23.181Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
Mar 25 18:57:23 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:23.181Z [DEBUG] agent: Node info in sync
Mar 25 18:57:23 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:23.183Z [INFO] agent: Deregistered service: service=redis
Mar 25 18:57:23 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:23.184Z [INFO] agent: Deregistered check: check=service:redis1
Mar 25 18:57:30 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:30.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Mar 25 18:57:40 srv-pro-schd-05 consul[32576]: 2020-03-25T18:57:40.935Z [DEBUG] agent.server.autopilot: Failed to remove dead servers: error="denied, because removing the majority of servers 1/1 is not safe"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment