bootkube
Jun 24 20:12:47 node1.example.com bootkube-start[1389]: [ 392.335477] bootkube[5]: All self-hosted control plane components successfully started
Jun 24 20:12:47 node1.example.com bootkube-start[1389]: [ 392.380119] bootkube[5]: Migrating to self-hosted etcd cluster...
Jun 24 20:12:47 node1.example.com bootkube-start[1389]: [ 392.381440] bootkube[5]: I0624 20:12:47.457458 5 log.go:19] warning: ignoring ServerName for user-provided CA for backwards compatibility is depre
cated
Jun 24 20:12:52 node1.example.com bootkube-start[1389]: [ 397.379177] bootkube[5]: I0624 20:12:52.487624 5 migrate.go:65] created etcd cluster TPR
Jun 24 20:12:57 node1.example.com bootkube-start[1389]: [ 402.504725] bootkube[5]: I0624 20:12:57.613136 5 migrate.go:76] etcd-service IP is 10.3.0.15
Jun 24 20:12:57 node1.example.com bootkube-start[1389]: [ 402.517627] bootkube[5]: I0624 20:12:57.621899 5 migrate.go:81] created etcd cluster for migration
Jun 24 20:13:07 node1.example.com bootkube-start[1389]: [ 412.520584] bootkube[5]: I0624 20:13:07.629341 5 migrate.go:86] etcd cluster for migration is now running
Jun 24 20:13:17 node1.example.com bootkube-start[1389]: [ 422.524262] bootkube[5]: E0624 20:13:17.633052 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:13:27 node1.example.com bootkube-start[1389]: [ 432.525721] bootkube[5]: E0624 20:13:27.634415 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:13:37 node1.example.com bootkube-start[1389]: [ 442.525476] bootkube[5]: E0624 20:13:37.633265 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:13:42 node1.example.com bootkube-start[1389]: [ 447.524887] bootkube[5]: E0624 20:13:42.633633 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:13:52 node1.example.com bootkube-start[1389]: [ 457.524317] bootkube[5]: E0624 20:13:52.633109 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:14:02 node1.example.com bootkube-start[1389]: [ 467.523775] bootkube[5]: E0624 20:14:02.632262 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:14:12 node1.example.com bootkube-start[1389]: [ 477.526430] bootkube[5]: E0624 20:14:12.635215 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:14:22 node1.example.com bootkube-start[1389]: [ 487.529450] bootkube[5]: E0624 20:14:22.638081 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:14:32 node1.example.com bootkube-start[1389]: [ 497.524622] bootkube[5]: E0624 20:14:32.633294 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:14:42 node1.example.com bootkube-start[1389]: [ 507.524873] bootkube[5]: E0624 20:14:42.633669 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:14:52 node1.example.com bootkube-start[1389]: [ 517.524710] bootkube[5]: E0624 20:14:52.633272 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:15:02 node1.example.com bootkube-start[1389]: [ 527.524195] bootkube[5]: E0624 20:15:02.632942 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:15:12 node1.example.com bootkube-start[1389]: [ 537.524712] bootkube[5]: E0624 20:15:12.633509 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:15:22 node1.example.com bootkube-start[1389]: [ 547.525794] bootkube[5]: E0624 20:15:22.634446 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:15:32 node1.example.com bootkube-start[1389]: [ 557.525783] bootkube[5]: E0624 20:15:32.633673 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:15:42 node1.example.com bootkube-start[1389]: [ 567.524276] bootkube[5]: E0624 20:15:42.633070 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:15:52 node1.example.com bootkube-start[1389]: [ 577.525784] bootkube[5]: E0624 20:15:52.634432 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:16:02 node1.example.com bootkube-start[1389]: [ 587.523705] bootkube[5]: E0624 20:16:02.632459 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:16:12 node1.example.com bootkube-start[1389]: [ 597.524062] bootkube[5]: E0624 20:16:12.632849 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:16:22 node1.example.com bootkube-start[1389]: [ 607.525037] bootkube[5]: E0624 20:16:22.633840 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:16:32 node1.example.com bootkube-start[1389]: [ 617.524146] bootkube[5]: E0624 20:16:32.632958 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:16:42 node1.example.com bootkube-start[1389]: [ 627.523774] bootkube[5]: E0624 20:16:42.632567 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:16:47 node1.example.com bootkube-start[1389]: [ 632.526321] bootkube[5]: E0624 20:16:47.634807 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:16:57 node1.example.com bootkube-start[1389]: [ 642.524995] bootkube[5]: E0624 20:16:57.633057 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:17:07 node1.example.com bootkube-start[1389]: [ 652.524613] bootkube[5]: E0624 20:17:07.633255 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:17:17 node1.example.com bootkube-start[1389]: [ 662.524062] bootkube[5]: E0624 20:17:17.632612 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:17:22 node1.example.com bootkube-start[1389]: [ 667.524113] bootkube[5]: E0624 20:17:22.632863 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:17:32 node1.example.com bootkube-start[1389]: [ 677.528803] bootkube[5]: E0624 20:17:32.637448 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:17:42 node1.example.com bootkube-start[1389]: [ 687.523763] bootkube[5]: E0624 20:17:42.632515 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:17:52 node1.example.com bootkube-start[1389]: [ 697.525246] bootkube[5]: E0624 20:17:52.632569 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:17:57 node1.example.com bootkube-start[1389]: [ 702.525416] bootkube[5]: E0624 20:17:57.633775 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:18:02 node1.example.com bootkube-start[1389]: [ 707.557258] bootkube[5]: E0624 20:18:02.665796 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:18:12 node1.example.com bootkube-start[1389]: [ 717.526567] bootkube[5]: E0624 20:18:12.635002 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:18:17 node1.example.com bootkube-start[1389]: [ 722.547379] bootkube[5]: E0624 20:18:17.656186 5 migrate.go:201] failed to create etcd client, will retry: grpc: timed out when dialing
Jun 24 20:18:30 node1.example.com bootkube-start[1389]: [ 735.618303] bootkube[5]: E0624 20:18:30.726880 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:18:44 node1.example.com bootkube-start[1389]: [ 749.584951] bootkube[5]: E0624 20:18:44.693710 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:18:58 node1.example.com bootkube-start[1389]: [ 763.595928] bootkube[5]: E0624 20:18:58.704703 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:20:36 node1.example.com bootkube-start[1389]: [ 861.607692] bootkube[5]: E0624 20:20:36.716523 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:20:50 node1.example.com bootkube-start[1389]: [ 875.617260] bootkube[5]: E0624 20:20:50.725977 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:21:04 node1.example.com bootkube-start[1389]: [ 889.638350] bootkube[5]: E0624 20:21:04.747132 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:21:18 node1.example.com bootkube-start[1389]: [ 903.628846] bootkube[5]: E0624 20:21:18.737622 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:21:32 node1.example.com bootkube-start[1389]: [ 917.637132] bootkube[5]: E0624 20:21:32.745288 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:21:46 node1.example.com bootkube-start[1389]: [ 931.643345] bootkube[5]: E0624 20:21:46.752046 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:22:00 node1.example.com bootkube-start[1389]: [ 945.620258] bootkube[5]: E0624 20:22:00.729075 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:22:14 node1.example.com bootkube-start[1389]: [ 959.637621] bootkube[5]: E0624 20:22:14.746336 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:22:28 node1.example.com bootkube-start[1389]: [ 973.641176] bootkube[5]: E0624 20:22:28.749812 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:22:42 node1.example.com bootkube-start[1389]: [ 987.644000] bootkube[5]: E0624 20:22:42.752650 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:22:56 node1.example.com bootkube-start[1389]: [ 1001.648771] bootkube[5]: E0624 20:22:56.756812 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:23:10 node1.example.com bootkube-start[1389]: [ 1015.643308] bootkube[5]: E0624 20:23:10.751995 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:23:24 node1.example.com bootkube-start[1389]: [ 1029.659880] bootkube[5]: E0624 20:23:24.768350 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:23:38 node1.example.com bootkube-start[1389]: [ 1043.661096] bootkube[5]: E0624 20:23:38.769028 5 migrate.go:230] failed to remove bootstrap-etcd-service: etcdserver: request timed out
Jun 24 20:23:38 node1.example.com bootkube-start[1389]: [ 1043.668901] bootkube[5]: E0624 20:23:38.769093 5 migrate.go:235] timed out removing bootstrap-etcd-service: timed out waiting for the condition
Jun 24 20:23:38 node1.example.com bootkube-start[1389]: [ 1043.674105] bootkube[5]: Error: failed to wait for boot-etcd to be removed: timed out waiting for the condition
Jun 24 20:23:38 node1.example.com bootkube-start[1389]: [ 1043.679957] bootkube[5]: Tearing down temporary bootstrap control plane...
Jun 24 20:23:38 node1.example.com bootkube-start[1389]: [ 1043.686663] bootkube[5]: Error: failed to wait for boot-etcd to be removed: timed out waiting for the condition
Jun 24 20:23:38 node1.example.com bootkube-start[1389]: [ 1043.692898] bootkube[5]: Error: failed to wait for boot-etcd to be removed: timed out waiting for the condition
Jun 24 20:23:38 node1.example.com bootkube-start[1389]: [ 1043.697768] bootkube[5]: failed to wait for boot-etcd to be removed: timed out waiting for the condition
Jun 24 20:23:39 node1.example.com systemd[1]: bootkube.service: Main process exited, code=exited, status=1/FAILURE
Jun 24 20:23:39 node1.example.com systemd[1]: Failed to start Bootstrap a Kubernetes control plane with a temp api-server.
Jun 24 20:23:39 node1.example.com systemd[1]: bootkube.service: Unit entered failed state.
Jun 24 20:23:39 node1.example.com systemd[1]: bootkube.service: Failed with result 'exit-code'.
bootstrap-etcd-node1.example.com
2017-06-24 20:10:45.355669 I | etcdmain: etcd Version: 3.1.8
2017-06-24 20:10:45.356721 I | etcdmain: Git SHA: d267ca9
2017-06-24 20:10:45.356726 I | etcdmain: Go Version: go1.7.5
2017-06-24 20:10:45.356736 I | etcdmain: Go OS/Arch: linux/amd64
2017-06-24 20:10:45.356739 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2017-06-24 20:10:45.356807 I | embed: peerTLS: cert = /etc/kubernetes/secrets/etcdMember/peer-crt.pem, key = /etc/kubernetes/secrets/etcdMember/peer-key.pem, ca = , trusted-ca = /etc/kubernetes/secrets/etcdMembe
r/peer-ca-crt.pem, client-cert-auth = true
2017-06-24 20:10:45.359297 I | embed: listening for peers on https://0.0.0.0:12380
2017-06-24 20:10:45.359415 I | embed: listening for client requests on 0.0.0.0:12379
2017-06-24 20:10:45.377145 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-24 20:10:45.377267 I | etcdserver: name = boot-etcd
2017-06-24 20:10:45.377291 I | etcdserver: data dir = /var/etcd/data
2017-06-24 20:10:45.377298 I | etcdserver: member dir = /var/etcd/data/member
2017-06-24 20:10:45.377303 I | etcdserver: heartbeat = 100ms
2017-06-24 20:10:45.377308 I | etcdserver: election = 1000ms
2017-06-24 20:10:45.377313 I | etcdserver: snapshot count = 10000
2017-06-24 20:10:45.377323 I | etcdserver: advertise client URLs = https://10.3.0.20:12379
2017-06-24 20:10:45.377329 I | etcdserver: initial advertise peer URLs = https://10.3.0.20:12380
2017-06-24 20:10:45.377338 I | etcdserver: initial cluster = boot-etcd=https://10.3.0.20:12380
2017-06-24 20:10:45.399293 I | etcdserver: starting member fcfe560b49a7a727 in cluster 37f107fb16c8db45
2017-06-24 20:10:45.399363 I | raft: fcfe560b49a7a727 became follower at term 0
2017-06-24 20:10:45.399398 I | raft: newRaft fcfe560b49a7a727 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2017-06-24 20:10:45.399408 I | raft: fcfe560b49a7a727 became follower at term 1
2017-06-24 20:10:45.422462 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-24 20:10:45.423203 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-24 20:10:45.423219 I | etcdserver: starting server... [version: 3.1.8, cluster version: to_be_decided]
2017-06-24 20:10:45.423236 I | embed: ClientTLS: cert = /etc/kubernetes/secrets/etcdMember/client-crt.pem, key = /etc/kubernetes/secrets/etcdMember/client-key.pem, ca = , trusted-ca = /etc/kubernetes/secrets/etc
dMember/client-ca-crt.pem, client-cert-auth = true
2017-06-24 20:10:45.424681 I | etcdserver/membership: added member fcfe560b49a7a727 [https://10.3.0.20:12380] to cluster 37f107fb16c8db45
2017-06-24 20:10:45.600491 I | raft: fcfe560b49a7a727 is starting a new election at term 1
2017-06-24 20:10:45.601238 I | raft: fcfe560b49a7a727 became candidate at term 2
2017-06-24 20:10:45.601453 I | raft: fcfe560b49a7a727 received MsgVoteResp from fcfe560b49a7a727 at term 2
2017-06-24 20:10:45.601660 I | raft: fcfe560b49a7a727 became leader at term 2
2017-06-24 20:10:45.601774 I | raft: raft.node: fcfe560b49a7a727 elected leader fcfe560b49a7a727 at term 2
2017-06-24 20:10:45.604941 I | etcdserver: setting up the initial cluster version to 3.1
2017-06-24 20:10:45.605258 I | etcdserver: published {Name:boot-etcd ClientURLs:[https://10.3.0.20:12379]} to cluster 37f107fb16c8db45
2017-06-24 20:10:45.608121 I | embed: ready to serve client requests
2017-06-24 20:10:45.609289 I | embed: serving client requests on [::]:12379
2017-06-24 20:10:45.674457 N | etcdserver/membership: set the initial cluster version to 3.1
2017-06-24 20:10:45.674560 I | etcdserver/api: enabled capabilities for version 3.1
2017-06-24 20:12:41.629725 W | wal: sync duration of 1.04478004s, expected less than 1s
2017-06-24 20:13:07.976262 I | etcdserver/membership: added member 6bafd0f2fb21ce8c [https://kube-etcd-0000.kube-etcd.kube-system.svc.cluster.local:2380] to cluster 37f107fb16c8db45
2017-06-24 20:13:07.976746 I | rafthttp: starting peer 6bafd0f2fb21ce8c...
2017-06-24 20:13:07.976780 I | rafthttp: started HTTP pipelining with peer 6bafd0f2fb21ce8c
2017-06-24 20:13:07.979649 I | rafthttp: started streaming with peer 6bafd0f2fb21ce8c (writer)
2017-06-24 20:13:07.979810 I | rafthttp: started streaming with peer 6bafd0f2fb21ce8c (writer)
2017-06-24 20:13:07.991100 I | rafthttp: started peer 6bafd0f2fb21ce8c
2017-06-24 20:13:07.992092 I | rafthttp: added peer 6bafd0f2fb21ce8c
2017-06-24 20:13:07.994884 I | rafthttp: started streaming with peer 6bafd0f2fb21ce8c (stream MsgApp v2 reader)
2017-06-24 20:13:08.008025 I | rafthttp: started streaming with peer 6bafd0f2fb21ce8c (stream Message reader)
2017-06-24 20:13:08.310523 I | rafthttp: peer 6bafd0f2fb21ce8c became active
2017-06-24 20:13:08.310606 I | rafthttp: established a TCP streaming connection with peer 6bafd0f2fb21ce8c (stream Message writer)
2017-06-24 20:13:08.313033 I | rafthttp: established a TCP streaming connection with peer 6bafd0f2fb21ce8c (stream MsgApp v2 writer)
2017-06-24 20:13:08.385206 E | rafthttp: failed to dial 6bafd0f2fb21ce8c on stream Message (dial tcp 172.18.0.21:2380: getsockopt: connection refused)
2017-06-24 20:13:08.385250 I | rafthttp: peer 6bafd0f2fb21ce8c became inactive
2017-06-24 20:13:08.599781 W | raft: fcfe560b49a7a727 stepped down to follower since quorum is not active
2017-06-24 20:13:08.599884 I | raft: fcfe560b49a7a727 became follower at term 2
2017-06-24 20:13:08.599904 I | raft: raft.node: fcfe560b49a7a727 lost leader fcfe560b49a7a727 at term 2
2017-06-24 20:13:08.605271 I | raft: fcfe560b49a7a727 no leader at term 2; dropping index reading msg
2017-06-24 20:13:10.100272 I | raft: fcfe560b49a7a727 is starting a new election at term 2
2017-06-24 20:13:10.100327 I | raft: fcfe560b49a7a727 became candidate at term 3
2017-06-24 20:13:10.100360 I | raft: fcfe560b49a7a727 received MsgVoteResp from fcfe560b49a7a727 at term 3
2017-06-24 20:13:10.100376 I | raft: fcfe560b49a7a727 [logterm: 2, index: 1304] sent MsgVote request to 6bafd0f2fb21ce8c at term 3
2017-06-24 20:13:11.299978 I | raft: fcfe560b49a7a727 is starting a new election at term 3
...
etcd-operator
time="2017-06-24T20:12:34Z" level=info msg="etcd-operator Version: 0.3.3"
time="2017-06-24T20:12:34Z" level=info msg="Git SHA: bcad5ae"
time="2017-06-24T20:12:34Z" level=info msg="Go Version: go1.8.1"
time="2017-06-24T20:12:34Z" level=info msg="Go OS/Arch: linux/amd64"
time="2017-06-24T20:12:34Z" level=info msg="Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"etcd-operator", UID:"75d29a6f-5919-11e7-b86b-525400a19cae", APIVersion:"v1", ResourceVersion$"778", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' etcd-operator-838413290-3gsvk became leader"
time="2017-06-24T20:12:37Z" level=info msg="starts running from watch version: 0" pkg=controller
time="2017-06-24T20:12:37Z" level=info msg="start watching at 0" pkg=controller
2017-06-24 20:12:57.733353 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
time="2017-06-24T20:12:57Z" level=info msg="creating cluster with
Spec:
{
"size": 1,
"baseImage": "quay.io/coreos/etcd",
"version": "3.1.8",
"pod": {
"nodeSelector": {
"node-role.kubernetes.io/master": ""
},
"resources": {},
"tolerations": [
{
"key": "node-role.kubernetes.io/master",
"operator": "Exists",
"effect": "NoSchedule"
}
]
},
"selfHosted": {
"bootMemberClientEndpoint": "https://10.3.0.20:12379"
},
"TLS": {
"static": {
"member": {
"peerSecret": "etcd-member-peer-tls",
"clientSecret": "etcd-member-client-tls"
},
"operatorSecret": "etcd-operator-client-tls"
}
}
}" cluster-name=kube-etcd pkg=cluster
time="2017-06-24T20:12:57Z" level=info msg="migrating boot member (https://10.3.0.20:12379)" cluster-name=kube-etcd pkg=cluster
time="2017-06-24T20:13:03Z" level=info msg="self-hosted cluster created with boot member (https://10.3.0.20:12379)" cluster-name=kube-etcd pkg=cluster
time="2017-06-24T20:13:03Z" level=info msg="wait 1m0s before removing the boot member" cluster-name=kube-etcd pkg=cluster
time="2017-06-24T20:13:04Z" level=info msg="start running..." cluster-name=kube-etcd pkg=cluster
E0624 20:13:17.864781 1 election.go:259] Failed to update lock: etcdserver: request timed out
time="2017-06-24T20:13:22Z" level=error msg="fail to poll pods: failed to list running pods: etcdserver: request timed out" cluster-name=kube-etcd pkg=cluster
E0624 20:13:24.882617 1 election.go:259] Failed to update lock: etcdserver: request timed out
E0624 20:13:24.883240 1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLin
k:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), L
abels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was emp
ty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'etcd-operator-838413290-3gsvk stopped leading'
time="2017-06-24T20:13:24Z" level=fatal msg="leader election lost"
kube-etcd-0000
kubectl logs kube-etcd-0000 -n kube-system -f
2017-06-24 20:48:51.250172 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_SERVICE_HOST=10.3.0.15
2017-06-24 20:48:51.250220 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_SERVICE_PORT_CLIENT=2379
2017-06-24 20:48:51.250224 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_2379_TCP_ADDR=10.3.0.15
2017-06-24 20:48:51.250227 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_SERVICE_PORT=2379
2017-06-24 20:48:51.250231 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=tcp://10.3.0.15:2379
2017-06-24 20:48:51.250276 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_2379_TCP_PORT=2379
2017-06-24 20:48:51.250279 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_2379_TCP_PROTO=tcp
2017-06-24 20:48:51.250286 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_2379_TCP=tcp://10.3.0.15:2379
2017-06-24 20:48:51.250309 I | etcdmain: etcd Version: 3.1.8
2017-06-24 20:48:51.250312 I | etcdmain: Git SHA: d267ca9
2017-06-24 20:48:51.250315 I | etcdmain: Go Version: go1.7.5
2017-06-24 20:48:51.250318 I | etcdmain: Go OS/Arch: linux/amd64
2017-06-24 20:48:51.250322 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2017-06-24 20:48:51.250366 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-06-24 20:48:51.250379 I | embed: peerTLS: cert = /etc/etcdtls/member/peer-tls/peer-crt.pem, key = /etc/etcdtls/member/peer-tls/peer-key.pem, ca = , trusted-ca = /etc/etcdtls/member/peer-tls/peer-ca-crt.pem, client-cert-auth = true
2017-06-24 20:48:51.253983 I | embed: listening for peers on https://0.0.0.0:2380
2017-06-24 20:48:51.254027 I | embed: listening for client requests on 0.0.0.0:2379
2017-06-24 20:48:51.266626 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-24 20:48:51.266760 I | etcdserver: name = kube-etcd-0000
2017-06-24 20:48:51.266766 I | etcdserver: data dir = /var/etcd/kube-system-kube-etcd-0000
2017-06-24 20:48:51.266772 I | etcdserver: member dir = /var/etcd/kube-system-kube-etcd-0000/member
2017-06-24 20:48:51.266775 I | etcdserver: heartbeat = 100ms
2017-06-24 20:48:51.266778 I | etcdserver: election = 1000ms
2017-06-24 20:48:51.266781 I | etcdserver: snapshot count = 10000
2017-06-24 20:48:51.266791 I | etcdserver: advertise client URLs = https://kube-etcd-0000.kube-etcd.kube-system.svc.cluster.local:2379
2017-06-24 20:48:51.267962 I | etcdserver: restarting member c6032ccebf550f05 in cluster 37f107fb16c8db45 at commit index 0
2017-06-24 20:48:51.267993 I | raft: c6032ccebf550f05 became follower at term 0
2017-06-24 20:48:51.268004 I | raft: newRaft c6032ccebf550f05 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2017-06-24 20:48:51.302753 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-24 20:48:51.304367 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-24 20:48:51.304459 I | etcdserver: starting server... [version: 3.1.8, cluster version: to_be_decided]
2017-06-24 20:48:51.304527 I | embed: ClientTLS: cert = /etc/etcdtls/member/client-tls/client-crt.pem, key = /etc/etcdtls/member/client-tls/client-key.pem, ca = , trusted-ca = /etc/etcdtls/member/client-tls/client-ca-crt.pem, client-cert-auth = true
2017-06-24 20:48:51.304595 I | etcdserver: skipped leadership transfer for stopping non-leader member
2017-06-24 20:48:58.306026 E | etcdserver: publish error: etcdserver: request timed out
2017-06-24 20:49:05.306399 E | etcdserver: publish error: etcdserver: request timed out
2017-06-24 20:49:12.307683 I | etcdserver: aborting publish because server is stopped
2017-06-24 20:49:12.349012 C | etcdmain: open /etc/etcdtls/member/client-tls/client-crt.pem: no such file or directory
$ kube-etcd-0000 -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
checkpointer.alpha.coreos.com/checkpoint: "true"
etcd.version: 3.1.8
pod.alpha.kubernetes.io/init-container-statuses: '[{"name":"append-hosts","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-06-24T20:48:38Z","finishedAt":"2017-06-24T20:48:38Z","containerID":"docker://ee7e12c3a6db81391bf5820149a51c04800299ce6819b30e991bbbd59196a437"}},"lastState":{},"ready":true,"restartCount":0,"image":"busybox:latest","imageID":"docker-pullable://busybox@sha256:be3c11fdba7cfe299214e46edc642e09514dbb9bbefcd0d3836c05a1e0cd0642","containerID":"docker://ee7e12c3a6db81391bf5820149a51c04800299ce6819b30e991bbbd59196a437"}]'
pod.alpha.kubernetes.io/init-containers: '[{"name":"append-hosts","image":"busybox","command":["/bin/sh","-c","[
-f /var/etcd/etcd-hosts.checkpoint ] \u0026\u0026 (cat /var/etcd/etcd-hosts.checkpoint
\u003e\u003e /etc/hosts) || true"],"resources":{},"volumeMounts":[{"name":"etcd-data","mountPath":"/var/etcd"},{"name":"default-token-jft32","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}]'
pod.beta.kubernetes.io/init-container-statuses: '[{"name":"append-hosts","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-06-24T20:48:38Z","finishedAt":"2017-06-24T20:48:38Z","containerID":"docker://ee7e12c3a6db81391bf5820149a51c04800299ce6819b30e991bbbd59196a437"}},"lastState":{},"ready":true,"restartCount":0,"image":"busybox:latest","imageID":"docker-pullable://busybox@sha256:be3c11fdba7cfe299214e46edc642e09514dbb9bbefcd0d3836c05a1e0cd0642","containerID":"docker://ee7e12c3a6db81391bf5820149a51c04800299ce6819b30e991bbbd59196a437"}]'
pod.beta.kubernetes.io/init-containers: '[{"name":"append-hosts","image":"busybox","command":["/bin/sh","-c","[
-f /var/etcd/etcd-hosts.checkpoint ] \u0026\u0026 (cat /var/etcd/etcd-hosts.checkpoint
\u003e\u003e /etc/hosts) || true"],"resources":{},"volumeMounts":[{"name":"etcd-data","mountPath":"/var/etcd"},{"name":"default-token-jft32","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}]'
creationTimestamp: 2017-06-24T20:48:35Z
labels:
app: etcd
etcd_cluster: kube-etcd
etcd_node: kube-etcd-0000
name: kube-etcd-0000
namespace: kube-system
ownerReferences:
- apiVersion: etcd.coreos.com/v1beta1
controller: true
kind: Cluster
name: kube-etcd
uid: 7dacf34a-591e-11e7-be5f-525400a19cae
resourceVersion: "923"
selfLink: /api/v1/namespaces/kube-system/pods/kube-etcd-0000
uid: 7dc3dbfe-591e-11e7-be5f-525400a19cae
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: etcd
topologyKey: kubernetes.io/hostname
containers:
- command:
- /bin/sh
- -ec
- sleep 5; flock /var/lock/etcd.lock -c "([ -d /var/etcd/kube-system-kube-etcd-0000
] || ETCDCTL_API=3 etcdctl --endpoints=https://10.3.0.20:12379 member add kube-etcd-0000
--peer-urls=https://kube-etcd-0000.kube-etcd.kube-system.svc.cluster.local:2380
--cert=/etc/etcdtls/operator/etcd-tls/etcd-crt.pem --key=/etc/etcdtls/operator/etcd-tls/etcd-key.pem
--cacert=/etc/etcdtls/operator/etcd-tls/etcd-ca-crt.pem); /usr/local/bin/etcd
--data-dir=/var/etcd/kube-system-kube-etcd-0000 --name=kube-etcd-0000 --initial-advertise-peer-urls=https://kube-etcd-0000.kube-etcd.kube-system.svc.cluster.local:2380
--listen-peer-urls=https://0.0.0.0:2380 --listen-client-urls=https://0.0.0.0:2379
--advertise-client-urls=https://kube-etcd-0000.kube-etcd.kube-system.svc.cluster.local:2379
--initial-cluster=boot-etcd=https://10.3.0.20:12380,kube-etcd-0000=https://kube-etcd-0000.kube-etcd.kube-system.svc.cluster.local:2380
--initial-cluster-state=existing --peer-client-cert-auth=true --peer-trusted-ca-file=/etc/etcdtls/member/peer-tls/peer-ca-crt.pem
--peer-cert-file=/etc/etcdtls/member/peer-tls/peer-crt.pem --peer-key-file=/etc/etcdtls/member/peer-tls/peer-key.pem
--client-cert-auth=true --trusted-ca-file=/etc/etcdtls/member/client-tls/client-ca-crt.pem
--cert-file=/etc/etcdtls/member/client-tls/client-crt.pem --key-file=/etc/etcdtls/member/client-tls/client-key.pem"
image: quay.io/coreos/etcd:v3.1.8
imagePullPolicy: IfNotPresent
name: etcd
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/etcd
name: etcd-data
- mountPath: /var/lock
name: var-lock
- mountPath: /etc/etcdtls/member/peer-tls
name: member-peer-tls
- mountPath: /etc/etcdtls/member/client-tls
name: member-client-tls
- mountPath: /etc/etcdtls/operator/etcd-tls
name: operator-etcd-tls
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-jft32
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostname: kube-etcd-0000
initContainers:
- command:
- /bin/sh
- -c
- '[ -f /var/etcd/etcd-hosts.checkpoint ] && (cat /var/etcd/etcd-hosts.checkpoint
>> /etc/hosts) || true'
image: busybox
imagePullPolicy: Always
name: append-hosts
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/etcd
name: etcd-data
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-jft32
readOnly: true
nodeName: node1.example.com
nodeSelector:
node-role.kubernetes.io/master: ""
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
subdomain: kube-etcd
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
volumes:
- hostPath:
path: /var/etcd
name: etcd-data
- hostPath:
path: /var/lock
name: var-lock
- name: member-peer-tls
secret:
defaultMode: 420
secretName: etcd-member-peer-tls
- name: member-client-tls
secret:
defaultMode: 420
secretName: etcd-member-client-tls
- name: operator-etcd-tls
secret:
defaultMode: 420
secretName: etcd-operator-client-tls
- name: default-token-jft32
secret:
defaultMode: 420
secretName: default-token-jft32
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2017-06-24T20:48:38Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2017-06-24T20:48:39Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2017-06-24T20:48:35Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://fde8a7bb856e6465adc1a1533899ef4e1419042cb2750760a13405eb6a491767
image: quay.io/coreos/etcd:v3.1.8
imageID: docker-pullable://quay.io/coreos/etcd@sha256:29a2e13e7aa8c6ca1c20023b403466f2f66b560c3b7d76aabab98f5a619daa2d
lastState: {}
name: etcd
ready: true
restartCount: 0
state:
running:
startedAt: 2017-06-24T20:48:39Z
hostIP: 172.18.0.21
initContainerStatuses:
- containerID: docker://ee7e12c3a6db81391bf5820149a51c04800299ce6819b30e991bbbd59196a437
image: busybox:latest
imageID: docker-pullable://busybox@sha256:be3c11fdba7cfe299214e46edc642e09514dbb9bbefcd0d3836c05a1e0cd0642
lastState: {}
name: append-hosts
ready: true
restartCount: 0
state:
terminated:
containerID: docker://ee7e12c3a6db81391bf5820149a51c04800299ce6819b30e991bbbd59196a437
exitCode: 0
finishedAt: 2017-06-24T20:48:38Z
reason: Completed
startedAt: 2017-06-24T20:48:38Z
phase: Running
podIP: 172.18.0.21
qosClass: BestEffort
startTime: 2017-06-24T20:48:35Z
Secrets
kubectl get secrets --all-namespaces
NAMESPACE NAME TYPE DATA AGE
default default-token-qrw11 kubernetes.io/service-account-token 3 1m
kube-public default-token-7dzqj kubernetes.io/service-account-token 3 1m
kube-system default-token-jft32 kubernetes.io/service-account-token 3 1m
kube-system etcd-member-client-tls Opaque 3 1m
kube-system etcd-member-peer-tls Opaque 3 1m
kube-system etcd-operator-client-tls Opaque 3 1m
kube-system kube-apiserver Opaque 7 1m
kube-system kube-controller-manager Opaque 2 1m