Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
$ sudo journalctl -xn10 -u 'etcd*' -f
-- Logs begin at Thu 2017-06-01 07:59:46 UTC. --
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal etcdadm[3504]: declare -x ETCDCTL_CERT_FILE="/etc/ssl/certs/etcd-client.pem"
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal etcdadm[3504]: declare -x ETCDCTL_KEY="/etc/ssl/certs/etcd-client-key.pem"
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal etcdadm[3504]: declare -x ETCDCTL_KEY_FILE="/etc/ssl/certs/etcd-client-key.pem"
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal sudo[3510]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm ]
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal sudo[3510]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3560]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm/snapshots ]
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3560]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3580]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/lib/etcd2 ]
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3580]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 03 00:15:35 ip-10-100-2-111.ec2.internal etcdadm[3504]: [ 2535.925420] etcd[5]: https://ec2-34-56-123-321.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 11.843094ms
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: [ 2541.848831] etcd[5]: https://ec2-12-34-123-321.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 3.298131ms
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_reconfigure: observing cluster state: quorum=2 healthy=2
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_is_unstarted: connecting to https://ec2-12-34-123-321.compute-1.amazonaws.com:2379
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: fffabc123badcafe: name=etcd2 peerURLs=https://ec2-12-34-123-321.compute-1.amazonaws.com:2380 clientURLs=https://ec2-12-34-123-321.compute-1.amazonaws.com:2379 isLeader=false
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: abcd123456789000[unstarted]: peerURLs=https://ec2-56-78-111-12.compute-1.amazonaws.com:2380
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: 123abc456badbeef: name=etcd0 peerURLs=https://ec2-34-56-123-321.compute-1.amazonaws.com:2380 clientURLs=https://ec2-34-56-123-321.compute-1.amazonaws.com:2379 isLeader=true
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_is_unstarted: unstarted peer for this member(etcd1) is found
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: cat: /var/run/coreos/etcdadm/status: No such file or directory
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_reconfigure: cluster is already healthy but still in bootstrap process after the disaster recovery. searching for a etcd snapshot to recover this member
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: checking existence of s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db
Jun 03 00:15:42 ip-10-100-2-111.ec2.internal etcdadm[3504]: 2017-06-03 00:15:39 42872864 snapshot.db
Jun 03 00:15:42 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db exists
Jun 03 00:15:42 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_download_snapshot: downloading /var/run/coreos/etcdadm/snapshots/etcd1.db from s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db
Jun 03 00:15:44 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:44 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:44 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: [854B blob data]
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_local_snapshot_exists: checking existence of file /var/run/coreos/etcdadm/snapshots/etcd1.db
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_local_snapshot_exists: checking existence of file /var/run/coreos/etcdadm/snapshots/etcd1.db
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_bootstrap: backup found. restoring etcd1...
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_clean_data_dir: cleaning data dir of etcd1
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_clean_data_dir: data dir /var/lib/etcd2 exists. finding files to remove
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_clean_data_dir: no files found in /var/lib/etcd2
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_restore_from_local_snapshot: restoring etcd1
Jun 03 00:15:48 ip-10-100-2-111.ec2.internal etcdadm[3504]: pod "55e765de-4f74-445b-94c0-033c0cbd0127" is already stopped
Jun 03 00:15:48 ip-10-100-2-111.ec2.internal etcdadm[3504]: "55e765de-4f74-445b-94c0-033c0cbd0127"
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_remove_snapshot: removing write protected local snapshot file: /var/run/coreos/etcdadm/snapshots/etcd1.db
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_restore_from_local_snapshot: restored etcd1
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_set_initial_cluster_state: setting initial cluster state to: new
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: _systemctl_daemon_reload: running `systemctl daemon-reload` to reload etcd-member.service
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Started periodic etcd health check.
-- Subject: Unit etcdadm-check.timer has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-check.timer has finished starting up.
--
-- The start-up result is done.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Starting etcd health check...
-- Subject: Unit etcdadm-check.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-check.service has begun starting up.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemctl[4489]: inactive
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: etcdadm-check.service: Control process exited, code=exited status=3
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Failed to start etcd health check.
-- Subject: Unit etcdadm-check.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-check.service has failed.
--
-- The result is failed.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: etcdadm-check.service: Unit entered failed state.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: etcdadm-check.service: Failed with result 'exit-code'.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Started etcdadm reconfigure runner.
-- Subject: Unit etcdadm-reconfigure.service has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-reconfigure.service has finished starting up.
--
-- The start-up result is done.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Starting etcd (System Application Container)...
-- Subject: Unit etcd-member.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcd-member.service has begun starting up.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal rkt[4504]: rm: cannot get pod: no matches found for "220fdfca-18e9-4dcf-92b4-9f4dbc2cf70e"
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal rkt[4504]: rm: failed to remove one or more pods
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemctl[4516]: active
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemctl[4533]: active
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: ++ id -u etcd
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=systemd-dir,target=/run/systemd/system --volume systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --mount volume=notify,target=/run/systemd/notify --volume notify,kind=host,source=/run/systemd/notify --set-env=NOTIFY_SOCKET=/run/systemd/notify --volume data-dir,kind=host,source=/var/lib/etcd2,readOnly=false --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume etc-hosts,kind=host,source=/etc/hosts,readOnly=true --volume etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=data-dir,target=/var/lib/etcd --mount volume=etc-ssl-certs,target=/etc/ssl/certs --mount volume=usr-share-certs,target=/usr/share/ca-certificates --mount volume=etc-hosts,target=/etc/hosts --mount volume=etc-resolv,target=/etc/resolv.conf --inherit-env --stage1-from-dir=stage1-fly.aci quay.io/coreos/etcd:v3.1.3 --user=232 --
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.051136 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://ec2-56-78-111-12.compute-1.amazonaws.com:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.051559 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.051790 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052057 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052294 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://ec2-56-78-111-12.compute-1.amazonaws.com:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052510 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd0=https://ec2-34-56-123-321.compute-1.amazonaws.com:2380,etcd1=https://ec2-56-78-111-12.compute-1.amazonaws.com:2380,etcd2=https://ec2-12-34-123-321.compute-1.amazonaws.com:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052725 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052953 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053181 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://10.100.2.111:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053415 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://10.100.2.111:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053636 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053844 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054078 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054291 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054512 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054736 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054956 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.1.3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055181 I | etcdmain: etcd Version: 3.1.3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055389 I | etcdmain: Git SHA: 21fdcc6
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055596 I | etcdmain: Go Version: go1.7.5
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055796 I | etcdmain: Go OS/Arch: linux/amd64
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.056032 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.056334 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.056556 I | embed: peerTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.083824 I | embed: listening for peers on https://10.100.2.111:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.084121 I | embed: listening for client requests on 10.100.2.111:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.089193 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.114721 I | etcdserver: recovered store from snapshot at index 3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115036 I | etcdserver: name = etcd1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115316 I | etcdserver: data dir = /var/lib/etcd
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115541 I | etcdserver: member dir = /var/lib/etcd/member
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115751 I | etcdserver: heartbeat = 100ms
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115987 I | etcdserver: election = 1000ms
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.116195 I | etcdserver: snapshot count = 10000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.116416 I | etcdserver: advertise client URLs = https://ec2-56-78-111-12.compute-1.amazonaws.com:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.134348 I | etcdserver: restarting member 1234dead123feeed in cluster 5555abcdef000000 at commit index 3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.134631 I | raft: 1234dead123feeed became follower at term 1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.134863 I | raft: newRaft 1234dead123feeed [peers: [1234dead123feeed,fffabc123badcafe,123abc456badbeef], term: 1, commit: 3, applied: 3, lastindex: 3, lastterm: 1]
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.135209 I | etcdserver/membership: added member 123abc456badbeef [https://ec2-34-56-123-321.compute-1.amazonaws.com:2380] to cluster 5555abcdef000000 from store
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.135433 I | etcdserver/membership: added member 1234dead123feeed [https://ec2-56-78-111-12.compute-1.amazonaws.com:2380] to cluster 5555abcdef000000 from store
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.135651 I | etcdserver/membership: added member fffabc123badcafe [https://ec2-12-34-123-321.compute-1.amazonaws.com:2380] to cluster 5555abcdef000000 from store
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.479538 I | mvcc: restore compact to 1233569
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.539655 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.540670 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.540929 I | rafthttp: starting peer fffabc123badcafe...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.541148 I | rafthttp: started HTTP pipelining with peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.547479 I | rafthttp: started peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.547715 I | rafthttp: added peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.547947 I | rafthttp: starting peer 123abc456badbeef...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.548152 I | rafthttp: started HTTP pipelining with peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.562579 I | rafthttp: started streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.562810 I | rafthttp: started streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.563040 I | rafthttp: started streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.563270 I | rafthttp: started streaming with peer fffabc123badcafe (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.563948 I | rafthttp: started streaming with peer fffabc123badcafe (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.564721 I | rafthttp: started peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.564964 I | rafthttp: added peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.565163 I | etcdserver: starting server... [version: 3.1.3, cluster version: to_be_decided]
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.565367 I | embed: ClientTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.566428 I | rafthttp: started streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.566690 I | rafthttp: started streaming with peer 123abc456badbeef (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.567076 I | rafthttp: started streaming with peer 123abc456badbeef (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.706257 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.706968 I | raft: 1234dead123feeed [term: 1] received a MsgHeartbeat message with higher term from 123abc456badbeef [term: 74483]
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.707194 I | raft: 1234dead123feeed became follower at term 74483
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.707403 I | raft: raft.node: 1234dead123feeed elected leader 123abc456badbeef at term 74483
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.710526 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.711799 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.768414 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.770777 E | etcdserver: the member has been permanently removed from the cluster
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771083 I | etcdserver: the data-dir used by this member must be removed.
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771314 I | etcdserver: aborting publish because server is stopped
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771575 I | rafthttp: stopping peer fffabc123badcafe...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771906 I | rafthttp: stopped streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772134 I | rafthttp: stopped streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772374 I | rafthttp: stopped HTTP pipelining with peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772607 I | rafthttp: stopped streaming with peer fffabc123badcafe (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772819 I | rafthttp: stopped streaming with peer fffabc123badcafe (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773048 I | rafthttp: stopped peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773259 I | rafthttp: stopping peer 123abc456badbeef...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773473 I | rafthttp: stopped streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773689 I | rafthttp: stopped streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775298 I | rafthttp: stopped HTTP pipelining with peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775576 I | rafthttp: stopped streaming with peer 123abc456badbeef (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775838 I | rafthttp: stopped streaming with peer 123abc456badbeef (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775853 I | rafthttp: stopped peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.814998 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.815485 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.884988 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.885436 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.888558 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.901012 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.906903 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.951568 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.978848 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.026744 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.029298 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.029632 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.091713 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.093049 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.168033 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.170214 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.197908 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.199151 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.207919 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.380200 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.381326 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.412037 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.412521 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.412879 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.423533 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.505627 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.