Created
April 26, 2017 23:30
-
-
Save hasbro17/25bd98d3684dc58ee46c99facec88515 to your computer and use it in GitHub Desktop.
Logs of the etcd member 0004 whose health check fails first after upgrade to 3.1.6 https://github.com/coreos/etcd-operator/issues/1008
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Haseebs-MacBook-Pro:etcd-operator haseeb$ kubectl -n e2e-local logs example-etcd-cluster-with-backup-0004 -f | |
| 2017-04-26 23:23:30.651048 I | etcdmain: etcd Version: 3.1.5 | |
| 2017-04-26 23:23:30.651122 I | etcdmain: Git SHA: 20490ca | |
| 2017-04-26 23:23:30.651126 I | etcdmain: Go Version: go1.7.5 | |
| 2017-04-26 23:23:30.651129 I | etcdmain: Go OS/Arch: linux/amd64 | |
| 2017-04-26 23:23:30.651133 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2 | |
| 2017-04-26 23:23:30.651202 I | embed: listening for peers on http://0.0.0.0:2380 | |
| 2017-04-26 23:23:30.651232 I | embed: listening for client requests on 0.0.0.0:2379 | |
| 2017-04-26 23:23:30.663504 I | pkg/netutil: resolving example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380 to 10.244.2.45:2380 | |
| 2017-04-26 23:23:30.663824 I | pkg/netutil: resolving example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380 to 10.244.2.45:2380 | |
| 2017-04-26 23:23:30.663883 I | pkg/netutil: resolving example-etcd-cluster-with-backup-0004.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380 to 10.244.2.46:2380 | |
| 2017-04-26 23:23:30.663905 I | pkg/netutil: resolving example-etcd-cluster-with-backup-0004.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380 to 10.244.2.46:2380 | |
| 2017-04-26 23:23:30.664359 I | etcdserver: name = example-etcd-cluster-with-backup-0004 | |
| 2017-04-26 23:23:30.664373 I | etcdserver: data dir = /var/etcd/data | |
| 2017-04-26 23:23:30.664386 I | etcdserver: member dir = /var/etcd/data/member | |
| 2017-04-26 23:23:30.664391 I | etcdserver: heartbeat = 100ms | |
| 2017-04-26 23:23:30.664394 I | etcdserver: election = 1000ms | |
| 2017-04-26 23:23:30.664408 I | etcdserver: snapshot count = 10000 | |
| 2017-04-26 23:23:30.664419 I | etcdserver: advertise client URLs = http://example-etcd-cluster-with-backup-0004.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2379 | |
| 2017-04-26 23:23:30.678389 I | etcdserver: starting member 2f33700d5806219a in cluster b5d513d75f68b6f3 | |
| 2017-04-26 23:23:30.678502 I | raft: 2f33700d5806219a became follower at term 0 | |
| 2017-04-26 23:23:30.678532 I | raft: newRaft 2f33700d5806219a [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] | |
| 2017-04-26 23:23:30.678539 I | raft: 2f33700d5806219a became follower at term 1 | |
| 2017-04-26 23:23:30.713243 I | rafthttp: started HTTP pipelining with peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:30.713282 I | rafthttp: starting peer 3fbd082b6f67fcb4... | |
| 2017-04-26 23:23:30.713293 I | rafthttp: started HTTP pipelining with peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:30.719465 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (writer) | |
| 2017-04-26 23:23:30.719524 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (writer) | |
| 2017-04-26 23:23:30.719696 I | rafthttp: started peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:30.719783 I | rafthttp: added peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:30.719830 I | etcdserver: starting server... [version: 3.1.5, cluster version: to_be_decided] | |
| 2017-04-26 23:23:30.720373 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:30.720835 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (stream Message reader) | |
| 2017-04-26 23:23:30.722323 I | rafthttp: peer 3fbd082b6f67fcb4 became active | |
| 2017-04-26 23:23:30.722343 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:30.728780 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message reader) | |
| 2017-04-26 23:23:30.747553 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message writer) | |
| 2017-04-26 23:23:30.748082 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 writer) | |
| 2017-04-26 23:23:31.437310 I | raft: 2f33700d5806219a [term: 1] received a MsgVote message with higher term from 3fbd082b6f67fcb4 [term: 6] | |
| 2017-04-26 23:23:31.437354 I | raft: 2f33700d5806219a became follower at term 6 | |
| 2017-04-26 23:23:31.437367 I | raft: 2f33700d5806219a [logterm: 0, index: 0, vote: 0] cast MsgVote for 3fbd082b6f67fcb4 [logterm: 2, index: 5] at term 6 | |
| 2017-04-26 23:23:31.443495 I | raft: raft.node: 2f33700d5806219a elected leader 3fbd082b6f67fcb4 at term 6 | |
| 2017-04-26 23:23:31.454967 I | rafthttp: receiving database snapshot [index:5, from 3fbd082b6f67fcb4] ... | |
| 2017-04-26 23:23:31.468130 I | snap: saved database snapshot to disk [total bytes: 32768] | |
| 2017-04-26 23:23:31.468173 I | rafthttp: received and saved database snapshot [index: 5, from: 3fbd082b6f67fcb4] successfully | |
| 2017-04-26 23:23:31.468277 I | raft: 2f33700d5806219a [commit: 0, lastindex: 0, lastterm: 0] starts to restore snapshot [index: 5, term: 2] | |
| 2017-04-26 23:23:31.468302 I | raft: log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 5, term: 2] | |
| 2017-04-26 23:23:31.468323 I | raft: 2f33700d5806219a restored progress of 2f33700d5806219a [next = 6, match = 5, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0] | |
| 2017-04-26 23:23:31.468334 I | raft: 2f33700d5806219a restored progress of 3fbd082b6f67fcb4 [next = 6, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0] | |
| 2017-04-26 23:23:31.468339 I | raft: 2f33700d5806219a [commit: 5] restored snapshot [index: 5, term: 2] | |
| 2017-04-26 23:23:31.468755 I | etcdserver: applying snapshot at index 0... | |
| 2017-04-26 23:23:31.468935 I | etcdserver: recovering lessor... | |
| 2017-04-26 23:23:31.480030 I | etcdserver: finished recovering lessor | |
| 2017-04-26 23:23:31.480052 I | etcdserver: restoring mvcc store... | |
| 2017-04-26 23:23:31.482477 I | etcdserver: finished restoring mvcc store | |
| 2017-04-26 23:23:31.482507 I | etcdserver: recovering alarms... | |
| 2017-04-26 23:23:31.485043 I | etcdserver: closing old backend... | |
| 2017-04-26 23:23:31.487818 I | etcdserver: raft applied incoming snapshot at index 5 | |
| 2017-04-26 23:23:31.496810 I | etcdserver: finished closing old backend | |
| 2017-04-26 23:23:31.498893 I | etcdserver: finished recovering alarms | |
| 2017-04-26 23:23:31.498916 I | etcdserver: recovering auth store... | |
| 2017-04-26 23:23:31.498929 I | etcdserver: finished recovering auth store | |
| 2017-04-26 23:23:31.498932 I | etcdserver: recovering store v2... | |
| 2017-04-26 23:23:31.499515 I | etcdserver: finished recovering store v2 | |
| 2017-04-26 23:23:31.499534 I | etcdserver: recovering cluster configuration... | |
| 2017-04-26 23:23:31.499608 I | etcdserver/api: enabled capabilities for version 3.1 | |
| 2017-04-26 23:23:31.499623 I | etcdserver/membership: added member 3fbd082b6f67fcb4 [http://example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380] to cluster b5d513d75f68b6f3 from store | |
| 2017-04-26 23:23:31.499629 I | etcdserver/membership: added member 2f33700d5806219a [http://example-etcd-cluster-with-backup-0004.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380] to cluster b5d513d75f68b6f3 from store | |
| 2017-04-26 23:23:31.499634 I | etcdserver/membership: set the cluster version to 3.1 from store | |
| 2017-04-26 23:23:31.499637 I | etcdserver: finished recovering cluster configuration | |
| 2017-04-26 23:23:31.499640 I | etcdserver: removing old peers from network... | |
| 2017-04-26 23:23:31.499645 I | rafthttp: stopping peer 3fbd082b6f67fcb4... | |
| 2017-04-26 23:23:31.499840 I | rafthttp: closed the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 writer) | |
| 2017-04-26 23:23:31.499852 I | rafthttp: stopped streaming with peer 3fbd082b6f67fcb4 (writer) | |
| 2017-04-26 23:23:31.499982 I | rafthttp: closed the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message writer) | |
| 2017-04-26 23:23:31.499991 I | rafthttp: stopped streaming with peer 3fbd082b6f67fcb4 (writer) | |
| 2017-04-26 23:23:31.500112 I | rafthttp: stopped HTTP pipelining with peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:31.500184 W | rafthttp: lost the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:31.500197 I | rafthttp: stopped streaming with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:31.500246 W | rafthttp: lost the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message reader) | |
| 2017-04-26 23:23:31.500259 I | rafthttp: stopped streaming with peer 3fbd082b6f67fcb4 (stream Message reader) | |
| 2017-04-26 23:23:31.500264 I | rafthttp: stopped peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:31.500276 I | rafthttp: removed peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:31.500284 I | etcdserver: finished removing old peers from network | |
| 2017-04-26 23:23:31.500293 I | etcdserver: adding peers from new cluster configuration into network... | |
| 2017-04-26 23:23:31.500316 I | rafthttp: starting peer 3fbd082b6f67fcb4... | |
| 2017-04-26 23:23:31.500333 I | rafthttp: started HTTP pipelining with peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:31.500840 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (writer) | |
| 2017-04-26 23:23:31.500947 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (writer) | |
| 2017-04-26 23:23:31.501252 I | rafthttp: started peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:31.501271 I | rafthttp: added peer 3fbd082b6f67fcb4 | |
| 2017-04-26 23:23:31.501276 I | etcdserver: finished adding peers from new cluster configuration into network... | |
| 2017-04-26 23:23:31.501287 I | etcdserver: finished applying incoming snapshot at index 0 | |
| 2017-04-26 23:23:31.501371 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (stream Message reader) | |
| 2017-04-26 23:23:31.501563 I | etcdserver: published {Name:example-etcd-cluster-with-backup-0004 ClientURLs:[http://example-etcd-cluster-with-backup-0004.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2379]} to cluster b5d513d75f68b6f3 | |
| 2017-04-26 23:23:31.501685 I | embed: ready to serve client requests | |
| 2017-04-26 23:23:31.502077 I | rafthttp: started streaming with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:31.502314 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged! | |
| 2017-04-26 23:23:31.503680 I | rafthttp: peer 3fbd082b6f67fcb4 became active | |
| 2017-04-26 23:23:31.503705 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message reader) | |
| 2017-04-26 23:23:31.504200 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:31.602839 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 writer) | |
| 2017-04-26 23:23:31.602944 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message writer) | |
| 2017-04-26 23:23:40.799656 I | etcdserver/membership: added member 7922d51199b5e262 [http://example-etcd-cluster-with-backup-0005.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380] to cluster b5d513d75f68b6f3 | |
| 2017-04-26 23:23:40.799698 I | rafthttp: starting peer 7922d51199b5e262... | |
| 2017-04-26 23:23:40.799725 I | rafthttp: started HTTP pipelining with peer 7922d51199b5e262 | |
| 2017-04-26 23:23:40.801587 I | rafthttp: started streaming with peer 7922d51199b5e262 (writer) | |
| 2017-04-26 23:23:40.802299 I | rafthttp: started streaming with peer 7922d51199b5e262 (writer) | |
| 2017-04-26 23:23:40.804449 I | rafthttp: started peer 7922d51199b5e262 | |
| 2017-04-26 23:23:40.804479 I | rafthttp: added peer 7922d51199b5e262 | |
| 2017-04-26 23:23:40.804645 I | rafthttp: started streaming with peer 7922d51199b5e262 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:40.804765 I | rafthttp: started streaming with peer 7922d51199b5e262 (stream Message reader) | |
| 2017-04-26 23:23:45.804997 W | rafthttp: health check for peer 7922d51199b5e262 could not connect: dial tcp 10.244.2.47:2380: getsockopt: connection refused | |
| 2017-04-26 23:23:46.795022 I | rafthttp: peer 7922d51199b5e262 became active | |
| 2017-04-26 23:23:46.795124 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream MsgApp v2 writer) | |
| 2017-04-26 23:23:46.796369 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream Message writer) | |
| 2017-04-26 23:23:46.805432 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:46.805771 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream Message reader) | |
| 2017-04-26 23:23:46.866160 W | rafthttp: lost the TCP streaming connection with peer 7922d51199b5e262 (stream MsgApp v2 reader) | |
| 2017-04-26 23:23:46.867283 W | rafthttp: lost the TCP streaming connection with peer 7922d51199b5e262 (stream Message reader) | |
| 2017-04-26 23:23:46.874562 W | rafthttp: closed an existing TCP streaming connection with peer 7922d51199b5e262 (stream MsgApp v2 writer) | |
| 2017-04-26 23:23:46.876019 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream MsgApp v2 writer) | |
| 2017-04-26 23:23:46.876102 W | rafthttp: closed an existing TCP streaming connection with peer 7922d51199b5e262 (stream Message writer) | |
| 2017-04-26 23:23:46.876131 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream Message writer) | |
| 2017-04-26 23:23:46.968503 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream Message reader) | |
| 2017-04-26 23:23:46.969812 I | rafthttp: established a TCP streaming connection with peer 7922d51199b5e262 (stream MsgApp v2 reader) | |
| 2017-04-26 23:25:13.450420 W | rafthttp: lost the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:25:13.450855 E | rafthttp: failed to read 3fbd082b6f67fcb4 on stream MsgApp v2 (unexpected EOF) | |
| 2017-04-26 23:25:13.451483 I | rafthttp: peer 3fbd082b6f67fcb4 became inactive | |
| 2017-04-26 23:25:13.452623 W | rafthttp: lost the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message reader) | |
| 2017-04-26 23:25:14.478873 I | raft: 2f33700d5806219a is starting a new election at term 6 | |
| 2017-04-26 23:25:14.478913 I | raft: 2f33700d5806219a became candidate at term 7 | |
| 2017-04-26 23:25:14.478925 I | raft: 2f33700d5806219a received MsgVoteResp from 2f33700d5806219a at term 7 | |
| 2017-04-26 23:25:14.478935 I | raft: 2f33700d5806219a [logterm: 6, index: 9] sent MsgVote request to 3fbd082b6f67fcb4 at term 7 | |
| 2017-04-26 23:25:14.478953 I | raft: 2f33700d5806219a [logterm: 6, index: 9] sent MsgVote request to 7922d51199b5e262 at term 7 | |
| 2017-04-26 23:25:14.478965 I | raft: raft.node: 2f33700d5806219a lost leader 3fbd082b6f67fcb4 at term 7 | |
| 2017-04-26 23:25:14.486937 I | raft: 2f33700d5806219a received MsgVoteResp from 7922d51199b5e262 at term 7 | |
| 2017-04-26 23:25:14.486959 I | raft: 2f33700d5806219a [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections | |
| 2017-04-26 23:25:14.486975 I | raft: 2f33700d5806219a became leader at term 7 | |
| 2017-04-26 23:25:14.486990 I | raft: raft.node: 2f33700d5806219a elected leader 2f33700d5806219a at term 7 | |
| 2017-04-26 23:25:14.495592 W | etcdserver: failed to reach the peerURL(http://example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380) of member 3fbd082b6f67fcb4 (Get http://example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380/version: dial tcp 10.244.2.45:2380: getsockopt: connection refused) | |
| 2017-04-26 23:25:14.495622 W | etcdserver: cannot get the version of member 3fbd082b6f67fcb4 (Get http://example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380/version: dial tcp 10.244.2.45:2380: getsockopt: connection refused) | |
| 2017-04-26 23:25:14.779110 W | rafthttp: lost the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message writer) | |
| 2017-04-26 23:25:18.168118 W | rafthttp: lost the TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 writer) | |
| 2017-04-26 23:25:18.497196 W | etcdserver: failed to reach the peerURL(http://example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380) of member 3fbd082b6f67fcb4 (Get http://example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380/version: dial tcp 10.244.2.45:2380: getsockopt: connection refused) | |
| 2017-04-26 23:25:18.497224 W | etcdserver: cannot get the version of member 3fbd082b6f67fcb4 (Get http://example-etcd-cluster-with-backup-0003.example-etcd-cluster-with-backup.e2e-local.svc.cluster.local:2380/version: dial tcp 10.244.2.45:2380: getsockopt: connection refused) | |
| 2017-04-26 23:25:18.683607 I | rafthttp: peer 3fbd082b6f67fcb4 became active | |
| 2017-04-26 23:25:18.683774 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message writer) | |
| 2017-04-26 23:25:18.684193 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 writer) | |
| 2017-04-26 23:25:18.714898 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream MsgApp v2 reader) | |
| 2017-04-26 23:25:18.715086 I | rafthttp: established a TCP streaming connection with peer 3fbd082b6f67fcb4 (stream Message reader) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment