etcdadm-reconfigure-fail.log

$ sudo journalctl -xn10 -u 'etcd*' -f
-- Logs begin at Thu 2017-06-01 07:59:46 UTC. --
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal etcdadm[3504]: declare -x ETCDCTL_CERT_FILE="/etc/ssl/certs/etcd-client.pem"
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal etcdadm[3504]: declare -x ETCDCTL_KEY="/etc/ssl/certs/etcd-client-key.pem"
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal etcdadm[3504]: declare -x ETCDCTL_KEY_FILE="/etc/ssl/certs/etcd-client-key.pem"
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal sudo[3510]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm ]
Jun 03 00:15:32 ip-10-100-2-111.ec2.internal sudo[3510]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3560]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm/snapshots ]
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3560]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3580]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/lib/etcd2 ]
Jun 03 00:15:33 ip-10-100-2-111.ec2.internal sudo[3580]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 03 00:15:35 ip-10-100-2-111.ec2.internal etcdadm[3504]: [ 2535.925420] etcd[5]: https://ec2-34-56-123-321.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 11.843094ms
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: [ 2541.848831] etcd[5]: https://ec2-12-34-123-321.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 3.298131ms
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_reconfigure: observing cluster state: quorum=2 healthy=2
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_is_unstarted: connecting to https://ec2-12-34-123-321.compute-1.amazonaws.com:2379
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: fffabc123badcafe: name=etcd2 peerURLs=https://ec2-12-34-123-321.compute-1.amazonaws.com:2380 clientURLs=https://ec2-12-34-123-321.compute-1.amazonaws.com:2379 isLeader=false
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: abcd123456789000[unstarted]: peerURLs=https://ec2-56-78-111-12.compute-1.amazonaws.com:2380
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: 123abc456badbeef: name=etcd0 peerURLs=https://ec2-34-56-123-321.compute-1.amazonaws.com:2380 clientURLs=https://ec2-34-56-123-321.compute-1.amazonaws.com:2379 isLeader=true
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_is_unstarted: unstarted peer for this member(etcd1) is found
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: cat: /var/run/coreos/etcdadm/status: No such file or directory
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_reconfigure: cluster is already healthy but still in bootstrap process after the disaster recovery. searching for a etcd snapshot to recover this member
Jun 03 00:15:41 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: checking existence of s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db
Jun 03 00:15:42 ip-10-100-2-111.ec2.internal etcdadm[3504]: 2017-06-03 00:15:39   42872864 snapshot.db
Jun 03 00:15:42 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db exists
Jun 03 00:15:42 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_download_snapshot: downloading /var/run/coreos/etcdadm/snapshots/etcd1.db from s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db
Jun 03 00:15:44 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:44 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:44 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: [2.0K blob data]
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: [854B blob data]
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_local_snapshot_exists: checking existence of file /var/run/coreos/etcdadm/snapshots/etcd1.db
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_local_snapshot_exists: checking existence of file /var/run/coreos/etcdadm/snapshots/etcd1.db
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_bootstrap: backup found. restoring etcd1...
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_clean_data_dir: cleaning data dir of etcd1
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_clean_data_dir: data dir /var/lib/etcd2 exists. finding files to remove
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_clean_data_dir: no files found in /var/lib/etcd2
Jun 03 00:15:45 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_restore_from_local_snapshot: restoring etcd1
Jun 03 00:15:48 ip-10-100-2-111.ec2.internal etcdadm[3504]: pod "55e765de-4f74-445b-94c0-033c0cbd0127" is already stopped
Jun 03 00:15:48 ip-10-100-2-111.ec2.internal etcdadm[3504]: "55e765de-4f74-445b-94c0-033c0cbd0127"
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_remove_snapshot: removing write protected local snapshot file: /var/run/coreos/etcdadm/snapshots/etcd1.db
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_restore_from_local_snapshot: restored etcd1
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: member_set_initial_cluster_state: setting initial cluster state to: new
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcdadm[3504]: /opt/bin/etcdadm: info: _systemctl_daemon_reload: running `systemctl daemon-reload` to reload etcd-member.service
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Started periodic etcd health check.
-- Subject: Unit etcdadm-check.timer has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-check.timer has finished starting up.
--
-- The start-up result is done.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Starting etcd health check...
-- Subject: Unit etcdadm-check.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-check.service has begun starting up.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemctl[4489]: inactive
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: etcdadm-check.service: Control process exited, code=exited status=3
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Failed to start etcd health check.
-- Subject: Unit etcdadm-check.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-check.service has failed.
--
-- The result is failed.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: etcdadm-check.service: Unit entered failed state.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: etcdadm-check.service: Failed with result 'exit-code'.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Started etcdadm reconfigure runner.
-- Subject: Unit etcdadm-reconfigure.service has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-reconfigure.service has finished starting up.
--
-- The start-up result is done.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemd[1]: Starting etcd (System Application Container)...
-- Subject: Unit etcd-member.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcd-member.service has begun starting up.
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal rkt[4504]: rm: cannot get pod: no matches found for "220fdfca-18e9-4dcf-92b4-9f4dbc2cf70e"
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal rkt[4504]: rm: failed to remove one or more pods
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemctl[4516]: active
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal systemctl[4533]: active
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: ++ id -u etcd
Jun 03 00:15:49 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=systemd-dir,target=/run/systemd/system --volume systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --mount volume=notify,target=/run/systemd/notify --volume notify,kind=host,source=/run/systemd/notify --set-env=NOTIFY_SOCKET=/run/systemd/notify --volume data-dir,kind=host,source=/var/lib/etcd2,readOnly=false --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume etc-hosts,kind=host,source=/etc/hosts,readOnly=true --volume etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=data-dir,target=/var/lib/etcd --mount volume=etc-ssl-certs,target=/etc/ssl/certs --mount volume=usr-share-certs,target=/usr/share/ca-certificates --mount volume=etc-hosts,target=/etc/hosts --mount volume=etc-resolv,target=/etc/resolv.conf --inherit-env --stage1-from-dir=stage1-fly.aci quay.io/coreos/etcd:v3.1.3 --user=232 --
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.051136 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://ec2-56-78-111-12.compute-1.amazonaws.com:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.051559 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.051790 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052057 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052294 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://ec2-56-78-111-12.compute-1.amazonaws.com:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052510 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd0=https://ec2-34-56-123-321.compute-1.amazonaws.com:2380,etcd1=https://ec2-56-78-111-12.compute-1.amazonaws.com:2380,etcd2=https://ec2-12-34-123-321.compute-1.amazonaws.com:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052725 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.052953 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053181 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://10.100.2.111:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053415 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://10.100.2.111:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053636 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.053844 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054078 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054291 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054512 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054736 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.054956 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.1.3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055181 I | etcdmain: etcd Version: 3.1.3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055389 I | etcdmain: Git SHA: 21fdcc6
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055596 I | etcdmain: Go Version: go1.7.5
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.055796 I | etcdmain: Go OS/Arch: linux/amd64
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.056032 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.056334 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.056556 I | embed: peerTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.083824 I | embed: listening for peers on https://10.100.2.111:2380
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.084121 I | embed: listening for client requests on 10.100.2.111:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.089193 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.114721 I | etcdserver: recovered store from snapshot at index 3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115036 I | etcdserver: name = etcd1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115316 I | etcdserver: data dir = /var/lib/etcd
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115541 I | etcdserver: member dir = /var/lib/etcd/member
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115751 I | etcdserver: heartbeat = 100ms
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.115987 I | etcdserver: election = 1000ms
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.116195 I | etcdserver: snapshot count = 10000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.116416 I | etcdserver: advertise client URLs = https://ec2-56-78-111-12.compute-1.amazonaws.com:2379
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.134348 I | etcdserver: restarting member 1234dead123feeed in cluster 5555abcdef000000 at commit index 3
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.134631 I | raft: 1234dead123feeed became follower at term 1
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.134863 I | raft: newRaft 1234dead123feeed [peers: [1234dead123feeed,fffabc123badcafe,123abc456badbeef], term: 1, commit: 3, applied: 3, lastindex: 3, lastterm: 1]
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.135209 I | etcdserver/membership: added member 123abc456badbeef [https://ec2-34-56-123-321.compute-1.amazonaws.com:2380] to cluster 5555abcdef000000 from store
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.135433 I | etcdserver/membership: added member 1234dead123feeed [https://ec2-56-78-111-12.compute-1.amazonaws.com:2380] to cluster 5555abcdef000000 from store
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.135651 I | etcdserver/membership: added member fffabc123badcafe [https://ec2-12-34-123-321.compute-1.amazonaws.com:2380] to cluster 5555abcdef000000 from store
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.479538 I | mvcc: restore compact to 1233569
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.539655 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.540670 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.540929 I | rafthttp: starting peer fffabc123badcafe...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.541148 I | rafthttp: started HTTP pipelining with peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.547479 I | rafthttp: started peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.547715 I | rafthttp: added peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.547947 I | rafthttp: starting peer 123abc456badbeef...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.548152 I | rafthttp: started HTTP pipelining with peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.562579 I | rafthttp: started streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.562810 I | rafthttp: started streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.563040 I | rafthttp: started streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.563270 I | rafthttp: started streaming with peer fffabc123badcafe (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.563948 I | rafthttp: started streaming with peer fffabc123badcafe (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.564721 I | rafthttp: started peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.564964 I | rafthttp: added peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.565163 I | etcdserver: starting server... [version: 3.1.3, cluster version: to_be_decided]
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.565367 I | embed: ClientTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.566428 I | rafthttp: started streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.566690 I | rafthttp: started streaming with peer 123abc456badbeef (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.567076 I | rafthttp: started streaming with peer 123abc456badbeef (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.706257 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.706968 I | raft: 1234dead123feeed [term: 1] received a MsgHeartbeat message with higher term from 123abc456badbeef [term: 74483]
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.707194 I | raft: 1234dead123feeed became follower at term 74483
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.707403 I | raft: raft.node: 1234dead123feeed elected leader 123abc456badbeef at term 74483
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.710526 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.711799 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.768414 E | rafthttp: streaming request ignored (ID mismatch got abcd123456789000 want 1234dead123feeed)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.770777 E | etcdserver: the member has been permanently removed from the cluster
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771083 I | etcdserver: the data-dir used by this member must be removed.
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771314 I | etcdserver: aborting publish because server is stopped
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771575 I | rafthttp: stopping peer fffabc123badcafe...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.771906 I | rafthttp: stopped streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772134 I | rafthttp: stopped streaming with peer fffabc123badcafe (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772374 I | rafthttp: stopped HTTP pipelining with peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772607 I | rafthttp: stopped streaming with peer fffabc123badcafe (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.772819 I | rafthttp: stopped streaming with peer fffabc123badcafe (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773048 I | rafthttp: stopped peer fffabc123badcafe
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773259 I | rafthttp: stopping peer 123abc456badbeef...
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773473 I | rafthttp: stopped streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.773689 I | rafthttp: stopped streaming with peer 123abc456badbeef (writer)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775298 I | rafthttp: stopped HTTP pipelining with peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775576 I | rafthttp: stopped streaming with peer 123abc456badbeef (stream MsgApp v2 reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775838 I | rafthttp: stopped streaming with peer 123abc456badbeef (stream Message reader)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.775853 I | rafthttp: stopped peer 123abc456badbeef
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.814998 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.815485 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.884988 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.885436 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.888558 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.901012 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.906903 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.951568 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:51 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:51.978848 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.026744 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.029298 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.029632 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.091713 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.093049 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.168033 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.170214 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.197908 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.199151 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.207919 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.380200 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.381326 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.412037 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.412521 E | rafthttp: failed to find member 123abc456badbeef in cluster 5555abcdef000000
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.412879 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.423533 W | rafthttp: failed to process raft message (raft: stopped)
Jun 03 00:15:52 ip-10-100-2-111.ec2.internal etcd-wrapper[4539]: 2017-06-03 00:15:52.505627 E | rafthttp: failed to find member fffabc123badcafe in cluster 5555abcdef000000