Created
December 17, 2020 21:37
-
-
Save mmerrill3/a354e9289bc44e9c9f0711f6de932fdd to your computer and use it in GitHub Desktop.
Logs from reproducing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Before upgrade master is etcd-a at 10.206.147.234: | |
| I1217 20:38:56.924145 3694 controller.go:292] etcd cluster state: etcdClusterState | |
| members: | |
| {"name":"etcd-a","peerURLs":["https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380"],"endpoints":["https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:4001"],"ID":"603088857070764962"} | |
| {"name":"etcd-c","peerURLs":["https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380"],"endpoints":["https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:4001"],"ID":"4909666646868542013"} | |
| {"name":"etcd-b","peerURLs":["https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380"],"endpoints":["https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:4001"],"ID":"13841704517881535899"} | |
| peers: | |
| etcdClusterPeerInfo{peer=peer{id:"etcd-a" endpoints:"10.206.147.234:3996" }, info=cluster_name:"etcd" node_configuration:<name:"etcd-a" peer_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:4001" quarantined_client_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:3994" > etcd_state:<cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-a" peer_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" tls_enabled:true > nodes:<name:"etcd-b" peer_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-c" peer_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > > etcd_version:"3.4.13" > } | |
| etcdClusterPeerInfo{peer=peer{id:"etcd-b" endpoints:"10.206.148.211:3996" }, info=cluster_name:"etcd" node_configuration:<name:"etcd-b" peer_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:4001" quarantined_client_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:3994" > etcd_state:<cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-a" peer_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-b" peer_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" tls_enabled:true > nodes:<name:"etcd-c" peer_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > > etcd_version:"3.4.13" > } | |
| etcdClusterPeerInfo{peer=peer{id:"etcd-c" endpoints:"10.206.149.50:3996" }, info=cluster_name:"etcd" node_configuration:<name:"etcd-c" peer_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:4001" quarantined_client_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:3994" > etcd_state:<cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-a" peer_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-b" peer_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-c" peer_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" tls_enabled:true > > etcd_version:"3.4.13" > } | |
| Terminate etcd-a, Let ASG spin up a new etcd-a: | |
| new etcd-a: 10.206.147.64 | |
| etcd-manager | |
| I1217 20:40:28.696639 4169 volumes.go:86] AWS API Request: ec2metadata/GetToken | |
| I1217 20:40:28.697293 4169 volumes.go:86] AWS API Request: ec2metadata/GetDynamicData | |
| I1217 20:40:28.697841 4169 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 20:40:28.698274 4169 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 20:40:28.698714 4169 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 20:40:28.699154 4169 main.go:279] Mounting available etcd volumes matching tags [k8s.io/etcd/main k8s.io/role/master=1 kubernetes.io/cluster/k8s.ctnror0.dev.mmerrill.net=owned]; nameTag=k8s.io/etcd/main | |
| I1217 20:40:28.700554 4169 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 20:40:28.818723 4169 boot.go:50] waiting for volumes | |
| I1217 20:41:28.818981 4169 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 20:41:28.926144 4169 mounter.go:294] Trying to mount master volume: "vol-0963c543ee9f70a6d" | |
| I1217 20:41:28.926162 4169 volumes.go:331] Trying to attach volume "vol-0963c543ee9f70a6d" at "/dev/xvdu" | |
| I1217 20:41:28.926285 4169 volumes.go:86] AWS API Request: ec2/AttachVolume | |
| W1217 20:41:29.198571 4169 volumes.go:343] Invalid value '/dev/xvdu' for unixDevice. Attachment point /dev/xvdu is already in use | |
| I1217 20:41:29.198589 4169 volumes.go:331] Trying to attach volume "vol-0963c543ee9f70a6d" at "/dev/xvdv" | |
| I1217 20:41:29.198698 4169 volumes.go:86] AWS API Request: ec2/AttachVolume | |
| I1217 20:41:29.516227 4169 volumes.go:349] AttachVolume request returned { | |
| AttachTime: 2020-12-17 20:41:29.505 +0000 UTC, | |
| Device: "/dev/xvdv", | |
| InstanceId: "i-0fef4ca9cf27f2b98", | |
| State: "attaching", | |
| VolumeId: "vol-0963c543ee9f70a6d" | |
| } | |
| I1217 20:41:29.516355 4169 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 20:41:29.631016 4169 mounter.go:308] Currently attached volumes: [0xc000198100] | |
| I1217 20:41:29.631039 4169 mounter.go:72] Master volume "vol-0963c543ee9f70a6d" is attached at "/dev/xvdv" | |
| I1217 20:41:29.631053 4169 mounter.go:86] Doing safe-format-and-mount of /dev/xvdv to /mnt/master-vol-0963c543ee9f70a6d | |
| I1217 20:41:29.631065 4169 volumes.go:234] volume vol-0963c543ee9f70a6d not mounted at /rootfs/dev/xvdv | |
| I1217 20:41:29.631079 4169 volumes.go:263] nvme path not found "/rootfs/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0963c543ee9f70a6d" | |
| I1217 20:41:29.631112 4169 volumes.go:251] volume vol-0963c543ee9f70a6d not mounted at nvme-Amazon_Elastic_Block_Store_vol0963c543ee9f70a6d | |
| I1217 20:41:29.631117 4169 mounter.go:121] Waiting for volume "vol-0963c543ee9f70a6d" to be mounted | |
| I1217 20:41:30.631206 4169 volumes.go:234] volume vol-0963c543ee9f70a6d not mounted at /rootfs/dev/xvdv | |
| I1217 20:41:30.631256 4169 volumes.go:248] found nvme volume "nvme-Amazon_Elastic_Block_Store_vol0963c543ee9f70a6d" at "/dev/nvme2n1" | |
| I1217 20:41:30.631265 4169 mounter.go:125] Found volume "vol-0963c543ee9f70a6d" mounted at device "/dev/nvme2n1" | |
| I1217 20:41:30.631902 4169 mounter.go:167] Creating mount directory "/rootfs/mnt/master-vol-0963c543ee9f70a6d" | |
| I1217 20:41:30.631995 4169 mounter.go:172] Mounting device "/dev/nvme2n1" on "/mnt/master-vol-0963c543ee9f70a6d" | |
| I1217 20:41:30.923603 4169 nsenter.go:81] nsenter mount /dev/nvme2n1 /mnt/master-vol-0963c543ee9f70a6d ext4 [defaults] | |
| I1217 20:41:30.939214 4169 nsenter.go:85] Output of mounting /dev/nvme2n1 to /mnt/master-vol-0963c543ee9f70a6d: Running scope as unit: run-r1775cd542bd044b094f85e840f39863d.scope | |
| I1217 20:41:30.940223 4169 mounter.go:252] device "/dev/nvme2n1" did not evaluate as a symlink: lstat /dev/nvme2n1: no such file or directory | |
| I1217 20:41:30.940254 4169 mounter.go:252] device "/dev/nvme2n1" did not evaluate as a symlink: lstat /dev/nvme2n1: no such file or directory | |
| I1217 20:41:30.940261 4169 mounter.go:232] matched device "/dev/nvme2n1" and "/dev/nvme2n1" via '\x00' | |
| I1217 20:41:30.940273 4169 mounter.go:94] mounted master volume "vol-0963c543ee9f70a6d" on /mnt/master-vol-0963c543ee9f70a6d | |
| I1217 20:41:30.940286 4169 main.go:294] discovered IP address: 10.206.147.64 | |
| I1217 20:41:30.940291 4169 main.go:299] Setting data dir to /rootfs/mnt/master-vol-0963c543ee9f70a6d | |
| I1217 20:41:31.118383 4169 certs.go:183] generating certificate for "etcd-manager-server-etcd-a" | |
| I1217 20:41:31.423326 4169 certs.go:183] generating certificate for "etcd-manager-client-etcd-a" | |
| I1217 20:41:31.426060 4169 server.go:87] starting GRPC server using TLS, ServerName="etcd-manager-server-etcd-a" | |
| I1217 20:41:31.493065 4169 etcdserver.go:550] starting etcd with state cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-a" peer_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" tls_enabled:true > nodes:<name:"etcd-b" peer_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-c" peer_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > > etcd_version:"3.4.13" | |
| I1217 20:41:31.493169 4169 etcdserver.go:559] starting etcd with datadir /rootfs/mnt/master-vol-0963c543ee9f70a6d/data/etcd-cluster-token-etcd | |
| I1217 20:41:31.493199 4169 etcdserver.go:573] overriding clientURLs with [https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:4001] (state had [https://0.0.0.0:4001]) | |
| I1217 20:41:31.493210 4169 etcdserver.go:577] overriding quarantinedClientURLs with [https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:3994] (state had [https://0.0.0.0:3994]) | |
| I1217 20:41:31.496068 4169 pki.go:58] generating peer keypair for etcd: {CommonName:etcd-a Organization:[] AltNames:{DNSNames:[etcd-a etcd-a.internal.k8s.ctnror0.dev.mmerrill.net] IPs:[127.0.0.1]} Usages:[2 1]} | |
| I1217 20:41:31.497378 4169 certs.go:122] existing certificate not valid after 2022-11-24T17:07:04Z; will regenerate | |
| I1217 20:41:31.497392 4169 certs.go:183] generating certificate for "etcd-a" | |
| I1217 20:41:31.500767 4169 pki.go:101] building client-serving certificate: {CommonName:etcd-a Organization:[] AltNames:{DNSNames:[etcd-a etcd-a.internal.k8s.ctnror0.dev.mmerrill.net etcd-a.internal.k8s.ctnror0.dev.mmerrill.net] IPs:[127.0.0.1 127.0.0.1]} Usages:[1 2]} | |
| I1217 20:41:31.501918 4169 certs.go:122] existing certificate not valid after 2022-11-24T17:07:04Z; will regenerate | |
| I1217 20:41:31.501985 4169 certs.go:183] generating certificate for "etcd-a" | |
| I1217 20:41:31.719012 4169 certs.go:183] generating certificate for "etcd-manager-etcd-a" | |
| I1217 20:41:31.721010 4169 server.go:105] GRPC server listening on "10.206.147.64:3996" | |
| I1217 20:41:31.721188 4169 certs.go:183] generating certificate for "etcd-a" | |
| I1217 20:41:31.721218 4169 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 20:41:31.723680 4169 etcdprocess.go:196] executing command /opt/etcd-v3.4.13-linux-amd64/etcd [/opt/etcd-v3.4.13-linux-amd64/etcd] | |
| 2020-12-17 20:41:31.732856 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:4001 | |
| 2020-12-17 20:41:31.732883 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/clients/server.crt | |
| 2020-12-17 20:41:31.732892 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true | |
| 2020-12-17 20:41:31.732901 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/rootfs/mnt/master-vol-0963c543ee9f70a6d/data/etcd-cluster-token-etcd | |
| 2020-12-17 20:41:31.732914 I | pkg/flags: recognized and used environment variable ETCD_ENABLE_V2=false | |
| 2020-12-17 20:41:31.732938 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 20:41:31.732950 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-a=https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380,etcd-b=https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380,etcd-c=https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 20:41:31.732960 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing | |
| 2020-12-17 20:41:31.732968 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-token-etcd | |
| 2020-12-17 20:41:31.732975 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/clients/server.key | |
| 2020-12-17 20:41:31.732988 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:4001 | |
| 2020-12-17 20:41:31.733000 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380 | |
| 2020-12-17 20:41:31.733016 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-a | |
| 2020-12-17 20:41:31.733030 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/peers/me.crt | |
| 2020-12-17 20:41:31.733038 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true | |
| 2020-12-17 20:41:31.733045 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/peers/me.key | |
| 2020-12-17 20:41:31.733052 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/peers/ca.crt | |
| 2020-12-17 20:41:31.733072 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/clients/ca.crt | |
| 2020-12-17 20:41:31.733084 W | pkg/flags: unrecognized environment variable ETCD_LISTEN_METRICS_URLS= | |
| [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead | |
| 2020-12-17 20:41:31.733119 I | etcdmain: etcd Version: 3.4.13 | |
| 2020-12-17 20:41:31.733128 I | etcdmain: Git SHA: ae9734ed2 | |
| 2020-12-17 20:41:31.733132 I | etcdmain: Go Version: go1.12.17 | |
| 2020-12-17 20:41:31.733136 I | etcdmain: Go OS/Arch: linux/amd64 | |
| 2020-12-17 20:41:31.733140 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4 | |
| 2020-12-17 20:41:31.734401 N | etcdmain: the server is already initialized as member before, starting as etcd member... | |
| [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead | |
| 2020-12-17 20:41:31.734434 I | embed: peerTLS: cert = /rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/peers/me.crt, key = /rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/peers/me.key, trusted-ca = /rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/peers/ca.crt, client-cert-auth = true, crl-file = | |
| 2020-12-17 20:41:31.735722 I | embed: name = etcd-a | |
| 2020-12-17 20:41:31.735733 I | embed: data dir = /rootfs/mnt/master-vol-0963c543ee9f70a6d/data/etcd-cluster-token-etcd | |
| 2020-12-17 20:41:31.735738 I | embed: member dir = /rootfs/mnt/master-vol-0963c543ee9f70a6d/data/etcd-cluster-token-etcd/member | |
| 2020-12-17 20:41:31.735742 I | embed: heartbeat = 100ms | |
| 2020-12-17 20:41:31.735745 I | embed: election = 1000ms | |
| 2020-12-17 20:41:31.735749 I | embed: snapshot count = 100000 | |
| 2020-12-17 20:41:31.735759 I | embed: advertise client URLs = https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:4001 | |
| 2020-12-17 20:41:31.735770 I | embed: initial advertise peer URLs = https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 20:41:31.735775 I | embed: initial cluster = | |
| 2020-12-17 20:41:31.735807 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0963c543ee9f70a6d/data/etcd-cluster-token-etcd" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| 2020-12-17 20:41:31.736319 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0963c543ee9f70a6d/data/etcd-cluster-token-etcd/member/snap" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| I1217 20:41:31.822057 4169 volumes.go:86] AWS API Request: ec2/DescribeInstances | |
| I1217 20:41:31.869761 4169 peers.go:117] found new candidate peer from discovery: etcd-b [{10.206.148.211 0} {10.206.148.211 0}] | |
| I1217 20:41:31.870009 4169 peers.go:117] found new candidate peer from discovery: etcd-c [{10.206.149.50 0} {10.206.149.50 0}] | |
| I1217 20:41:31.870026 4169 peers.go:117] found new candidate peer from discovery: etcd-a [{10.206.147.64 0} {10.206.147.64 0}] | |
| I1217 20:41:31.870072 4169 peers.go:297] connecting to peer "etcd-b" with TLS policy, servername="etcd-manager-server-etcd-b" | |
| I1217 20:41:31.870088 4169 peers.go:297] connecting to peer "etcd-c" with TLS policy, servername="etcd-manager-server-etcd-c" | |
| I1217 20:41:31.870160 4169 hosts.go:84] hosts update: primary=map[], fallbacks=map[etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.147.64 10.206.147.64] etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.148.211 10.206.148.211] etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.149.50 10.206.149.50]], final=map[10.206.147.64:[etcd-a.internal.k8s.ctnror0.dev.mmerrill.net etcd-a.internal.k8s.ctnror0.dev.mmerrill.net] 10.206.148.211:[etcd-b.internal.k8s.ctnror0.dev.mmerrill.net etcd-b.internal.k8s.ctnror0.dev.mmerrill.net] 10.206.149.50:[etcd-c.internal.k8s.ctnror0.dev.mmerrill.net etcd-c.internal.k8s.ctnror0.dev.mmerrill.net]] | |
| I1217 20:41:31.870349 4169 peers.go:297] connecting to peer "etcd-a" with TLS policy, servername="etcd-manager-server-etcd-a" | |
| I1217 20:41:33.721145 4169 controller.go:189] starting controller iteration | |
| I1217 20:41:33.723078 4169 leadership.go:37] Got LeaderNotification view:<leader:<id:"etcd-a" endpoints:"10.206.147.64:3996" > leadership_token:"4vvUopq6J_mYzGFk3UV58A" healthy:<id:"etcd-c" endpoints:"10.206.149.50:3996" > healthy:<id:"etcd-a" endpoints:"10.206.147.64:3996" > healthy:<id:"etcd-b" endpoints:"10.206.148.211:3996" > > | |
| I1217 20:41:33.723256 4169 commands.go:41] refreshing commands | |
| I1217 20:41:33.739123 4169 vfs.go:120] listed commands in s3://kubernetes-ctnror0-dev/k8s.ctnror0.dev.mmerrill.net/backups/etcd/main/control: 0 commands | |
| 2020-12-17 20:41:33.806853 I | etcdserver: recovered store from snapshot at index 857134115 | |
| 2020-12-17 20:41:33.809497 I | mvcc: restore compact to 454382797 | |
| 2020-12-17 20:41:34.150963 I | etcdserver: restarting member 85e9a1ec5b303a2 in cluster 72141c0e48dbb6f2 at commit index 857228235 | |
| raft2020/12/17 20:41:34 INFO: 85e9a1ec5b303a2 switched to configuration voters=(603088857070764962 4909666646868542013 13841704517881535899) | |
| raft2020/12/17 20:41:34 INFO: 85e9a1ec5b303a2 became follower at term 49603 | |
| raft2020/12/17 20:41:34 INFO: newRaft 85e9a1ec5b303a2 [peers: [85e9a1ec5b303a2,4422a3cd842ec23d,c0179cec2a3f9d9b], term: 49603, commit: 857228235, applied: 857134115, lastindex: 857228236, lastterm: 49603] | |
| 2020-12-17 20:41:34.159582 I | etcdserver/api: enabled capabilities for version 3.4 | |
| 2020-12-17 20:41:34.159600 I | etcdserver/membership: added member 4422a3cd842ec23d [https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 20:41:34.159615 I | etcdserver/membership: added member 85e9a1ec5b303a2 [https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 20:41:34.159621 I | etcdserver/membership: added member c0179cec2a3f9d9b [https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 20:41:34.159633 I | etcdserver/membership: set the cluster version to 3.4 from store | |
| 2020-12-17 20:41:34.159670 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0963c543ee9f70a6d/data/etcd-cluster-token-etcd/member" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| 2020-12-17 20:41:34.164406 W | auth: simple token is not cryptographically signed | |
| 2020-12-17 20:41:34.165602 I | mvcc: restore compact to 454382797 | |
| 2020-12-17 20:41:34.186264 I | rafthttp: starting peer 4422a3cd842ec23d... | |
| 2020-12-17 20:41:34.186292 I | rafthttp: started HTTP pipelining with peer 4422a3cd842ec23d | |
| 2020-12-17 20:41:34.187172 I | rafthttp: started streaming with peer 4422a3cd842ec23d (writer) | |
| 2020-12-17 20:41:34.187332 I | rafthttp: started streaming with peer 4422a3cd842ec23d (stream MsgApp v2 reader) | |
| 2020-12-17 20:41:34.187811 I | rafthttp: started peer 4422a3cd842ec23d | |
| 2020-12-17 20:41:34.187839 I | rafthttp: added peer 4422a3cd842ec23d | |
| 2020-12-17 20:41:34.187849 I | rafthttp: started streaming with peer 4422a3cd842ec23d (writer) | |
| 2020-12-17 20:41:34.187860 I | rafthttp: starting peer c0179cec2a3f9d9b... | |
| 2020-12-17 20:41:34.187881 I | rafthttp: started HTTP pipelining with peer c0179cec2a3f9d9b | |
| 2020-12-17 20:41:34.187894 I | rafthttp: started streaming with peer 4422a3cd842ec23d (stream Message reader) | |
| 2020-12-17 20:41:34.188233 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (writer) | |
| 2020-12-17 20:41:34.188316 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (writer) | |
| 2020-12-17 20:41:34.188528 I | rafthttp: started peer c0179cec2a3f9d9b | |
| 2020-12-17 20:41:34.188547 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (stream MsgApp v2 reader) | |
| 2020-12-17 20:41:34.188886 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (stream Message reader) | |
| 2020-12-17 20:41:34.189011 I | rafthttp: added peer c0179cec2a3f9d9b | |
| 2020-12-17 20:41:34.189034 I | etcdserver: starting server... [version: 3.4.13, cluster version: 3.4] | |
| 2020-12-17 20:41:34.193589 I | embed: ClientTLS: cert = /rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/clients/server.crt, key = /rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/clients/server.key, trusted-ca = /rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/clients/ca.crt, client-cert-auth = true, crl-file = | |
| 2020-12-17 20:41:34.193716 I | embed: listening for peers on [::]:2380 | |
| raft2020/12/17 20:41:35 INFO: 85e9a1ec5b303a2 is starting a new election at term 49603 | |
| raft2020/12/17 20:41:35 INFO: 85e9a1ec5b303a2 became candidate at term 49604 | |
| raft2020/12/17 20:41:35 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49604 | |
| raft2020/12/17 20:41:35 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49604 | |
| raft2020/12/17 20:41:35 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49604 | |
| raft2020/12/17 20:41:37 INFO: 85e9a1ec5b303a2 is starting a new election at term 49604 | |
| raft2020/12/17 20:41:37 INFO: 85e9a1ec5b303a2 became candidate at term 49605 | |
| raft2020/12/17 20:41:37 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49605 | |
| raft2020/12/17 20:41:37 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49605 | |
| raft2020/12/17 20:41:37 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49605 | |
| raft2020/12/17 20:41:39 INFO: 85e9a1ec5b303a2 is starting a new election at term 49605 | |
| raft2020/12/17 20:41:39 INFO: 85e9a1ec5b303a2 became candidate at term 49606 | |
| raft2020/12/17 20:41:39 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49606 | |
| raft2020/12/17 20:41:39 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49606 | |
| raft2020/12/17 20:41:39 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49606 | |
| 2020-12-17 20:41:39.188254 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| 2020-12-17 20:41:39.188316 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| 2020-12-17 20:41:39.189251 W | rafthttp: health check for peer c0179cec2a3f9d9b could not connect: EOF | |
| 2020-12-17 20:41:39.189280 W | rafthttp: health check for peer c0179cec2a3f9d9b could not connect: EOF | |
| raft2020/12/17 20:41:40 INFO: 85e9a1ec5b303a2 is starting a new election at term 49606 | |
| raft2020/12/17 20:41:40 INFO: 85e9a1ec5b303a2 became candidate at term 49607 | |
| raft2020/12/17 20:41:40 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49607 | |
| raft2020/12/17 20:41:40 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49607 | |
| raft2020/12/17 20:41:40 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49607 | |
| 2020-12-17 20:41:41.189593 E | etcdserver: publish error: etcdserver: request timed out | |
| raft2020/12/17 20:41:42 INFO: 85e9a1ec5b303a2 is starting a new election at term 49607 | |
| raft2020/12/17 20:41:42 INFO: 85e9a1ec5b303a2 became candidate at term 49608 | |
| raft2020/12/17 20:41:42 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49608 | |
| raft2020/12/17 20:41:42 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49608 | |
| raft2020/12/17 20:41:42 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49608 | |
| raft2020/12/17 20:41:43 INFO: 85e9a1ec5b303a2 is starting a new election at term 49608 | |
| raft2020/12/17 20:41:43 INFO: 85e9a1ec5b303a2 became candidate at term 49609 | |
| raft2020/12/17 20:41:43 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49609 | |
| raft2020/12/17 20:41:43 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49609 | |
| raft2020/12/17 20:41:43 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49609 | |
| I1217 20:41:43.774957 4169 controller.go:189] starting controller iteration | |
| I1217 20:41:43.774985 4169 controller.go:285] I am leader with token "4vvUopq6J_mYzGFk3UV58A" | |
| 2020-12-17 20:41:44.188433 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| 2020-12-17 20:41:44.188466 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| 2020-12-17 20:41:44.189413 W | rafthttp: health check for peer c0179cec2a3f9d9b could not connect: EOF | |
| 2020-12-17 20:41:44.189465 W | rafthttp: health check for peer c0179cec2a3f9d9b could not connect: EOF | |
| raft2020/12/17 20:41:44 INFO: 85e9a1ec5b303a2 is starting a new election at term 49609 | |
| raft2020/12/17 20:41:44 INFO: 85e9a1ec5b303a2 became candidate at term 49610 | |
| raft2020/12/17 20:41:44 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49610 | |
| raft2020/12/17 20:41:44 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49610 | |
| raft2020/12/17 20:41:44 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49610 | |
| raft2020/12/17 20:41:46 INFO: 85e9a1ec5b303a2 is starting a new election at term 49610 | |
| raft2020/12/17 20:41:46 INFO: 85e9a1ec5b303a2 became candidate at term 49611 | |
| raft2020/12/17 20:41:46 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49611 | |
| raft2020/12/17 20:41:46 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49611 | |
| raft2020/12/17 20:41:46 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49611 | |
| raft2020/12/17 20:41:48 INFO: 85e9a1ec5b303a2 is starting a new election at term 49611 | |
| raft2020/12/17 20:41:48 INFO: 85e9a1ec5b303a2 became candidate at term 49612 | |
| raft2020/12/17 20:41:48 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 49612 | |
| raft2020/12/17 20:41:48 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to 4422a3cd842ec23d at term 49612 | |
| raft2020/12/17 20:41:48 INFO: 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] sent MsgVote request to c0179cec2a3f9d9b at term 49612 | |
| 2020-12-17 20:41:48.189725 E | etcdserver: publish error: etcdserver: request timed out | |
| 2020-12-17 20:41:49.188545 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| 2020-12-17 20:41:49.188566 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| 2020-12-17 20:41:49.189554 W | rafthttp: health check for peer c0179cec2a3f9d9b could not connect: EOF | |
| 2020-12-17 20:41:49.189573 W | rafthttp: health check for peer c0179cec2a3f9d9b could not connect: EOF | |
| Let's go to etcd-b (peer 4422a3cd842ec23d) and see what's up? | |
| 2020-12-17 20:52:53.028668 I | embed: rejected connection from "10.206.147.64:13060" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 20:52:53.036002 I | embed: rejected connection from "10.206.147.64:13062" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| Ok, problem with the new cert from etcd-A? | |
| /rootfs/mnt/master-vol-0963c543ee9f70a6d/pki/etcd-cluster-token-etcd/peers/me.crt | |
| cert on etcd-a | |
| Certificate: | |
| Data: | |
| Version: 3 (0x2) | |
| Serial Number: 6842219166765251623 (0x5ef472149e697c27) | |
| Signature Algorithm: sha256WithRSAEncryption | |
| Issuer: CN = etcd-peers-ca-main | |
| Validity | |
| Not Before: Apr 13 19:45:40 2019 GMT | |
| Not After : Dec 17 20:41:31 2022 GMT | |
| Subject: CN = etcd-a | |
| Subject Public Key Info: | |
| Public Key Algorithm: rsaEncryption | |
| RSA Public-Key: (2048 bit) | |
| Modulus: | |
| 00:df:c4:8d:29:05:3d:a6:9e:6e:fd:1b:cb:ae:06: | |
| de:79:de:c6:e1:eb:f9:65:4d:83:6f:bd:79:4f:2f: | |
| cc:66:c3:b7:42:f0:fb:a7:fd:d2:41:fe:05:98:6c: | |
| e2:ea:6c:24:8e:d0:67:cc:ed:07:5b:84:18:cb:91: | |
| f0:c2:25:2b:29:e8:e5:f3:2f:22:29:cf:0c:c4:93: | |
| 0c:ce:df:7c:ab:db:1e:fa:63:6a:32:f0:09:7f:41: | |
| e5:ca:c2:a1:ea:3c:a7:9a:d3:88:b6:7c:2b:9f:a4: | |
| de:c8:09:3c:f0:ec:2a:2d:ca:be:4c:5d:07:15:f9: | |
| 98:45:8d:79:7c:e9:8c:e5:07:9c:f7:04:ac:cc:c4: | |
| 6c:f5:d8:d7:3b:aa:12:c8:d9:3c:45:b5:f3:e4:c2: | |
| cc:f8:48:09:f2:4f:46:73:3d:20:63:cb:d8:48:e6: | |
| 5f:ba:81:75:2c:f3:73:61:e5:b0:76:83:81:48:c4: | |
| 05:09:a3:71:c3:e8:9f:46:5c:4c:29:16:7d:e6:66: | |
| 6f:83:cf:56:fc:15:df:38:50:19:a1:9f:88:35:b2: | |
| 7f:cc:17:74:6b:c2:ed:9a:f8:63:23:a0:15:20:f0: | |
| 7f:ff:32:ae:49:29:ff:d9:0d:2f:5b:51:9b:da:4e: | |
| ae:6f:84:dd:9e:a9:c5:90:ca:d8:88:fb:db:b2:38: | |
| 76:f1 | |
| Exponent: 65537 (0x10001) | |
| X509v3 extensions: | |
| X509v3 Key Usage: critical | |
| Digital Signature, Key Encipherment | |
| X509v3 Extended Key Usage: | |
| TLS Web Client Authentication, TLS Web Server Authentication | |
| X509v3 Subject Alternative Name: | |
| DNS:etcd-a, DNS:etcd-a.internal.k8s.ctnror0.dev.mmerrill.net, IP Address:127.0.0.1 | |
| Signature Algorithm: sha256WithRSAEncryption | |
| 4a:22:06:56:fa:70:bc:4d:3a:37:24:fa:6b:69:53:78:49:f4: | |
| a5:65:33:f2:d2:ca:9a:23:f2:9b:30:12:68:f7:fe:13:a1:08: | |
| ad:5d:ef:f2:7a:78:03:8c:50:3a:a5:51:c8:eb:91:47:e5:6c: | |
| c5:fc:da:df:8d:33:70:2e:d8:a5:59:b3:58:7c:33:3e:83:c5: | |
| 20:99:6b:ac:5f:32:69:2c:1d:f1:9a:c5:f8:b5:31:8e:6c:51: | |
| 39:97:f1:a7:28:0b:fb:3c:b5:57:b6:00:b2:a1:df:2c:d0:ac: | |
| 33:f3:38:b4:11:3b:03:62:cd:29:5f:bc:db:1f:90:7c:0a:55: | |
| 23:bf:a9:32:05:b2:96:e7:f8:48:15:d2:fa:e8:73:25:17:7f: | |
| 77:6d:58:1e:44:a0:f0:32:81:71:ef:ce:e6:7c:13:49:c4:ba: | |
| 38:03:6b:3d:85:52:8c:4c:fd:94:f4:0d:e9:1c:7d:f3:a4:9c: | |
| 0e:84:f3:42:b4:81:fa:62:32:1c:5b:95:24:a0:7c:5e:67:dd: | |
| 26:ba:e2:dd:1b:41:e0:38:89:16:9d:f1:05:9e:80:8d:2a:be: | |
| 8c:db:95:67:61:6e:17:e8:8f:45:2b:02:de:9d:8c:68:28:b6: | |
| 3b:e8:dd:5e:39:64:9c:af:52:11:96:b3:60:50:eb:5e:bb:24: | |
| 76:38:cc:73 | |
| cert on etcd-b | |
| Certificate: | |
| Data: | |
| Version: 3 (0x2) | |
| Serial Number: 5764821434558929013 (0x5000c28d28715875) | |
| Signature Algorithm: sha256WithRSAEncryption | |
| Issuer: CN = etcd-peers-ca-main | |
| Validity | |
| Not Before: Apr 13 19:45:40 2019 GMT | |
| Not After : Nov 24 17:07:52 2022 GMT | |
| Subject: CN = etcd-b | |
| Subject Public Key Info: | |
| Public Key Algorithm: rsaEncryption | |
| RSA Public-Key: (2048 bit) | |
| Modulus: | |
| 00:fa:d8:77:9a:03:07:49:9a:49:c5:f7:6b:4a:0d: | |
| 10:9d:ec:da:fe:96:84:27:84:83:b1:f1:b9:96:1d: | |
| 1f:13:80:a0:29:b8:a5:d1:ef:69:fe:b6:e3:80:48: | |
| f0:9e:2b:47:8a:56:84:35:62:f8:2c:9e:9a:c8:41: | |
| 15:ca:32:57:96:a5:4e:9c:d5:60:59:f7:03:79:f3: | |
| ec:fb:17:12:d5:67:73:de:2d:2f:46:57:d7:f0:23: | |
| 93:3b:3c:af:fb:52:c6:4e:54:6f:f7:4a:61:5c:8e: | |
| 01:51:74:71:d0:17:19:aa:e1:e2:3b:46:84:27:7b: | |
| bb:61:8d:62:79:ec:b7:d7:5d:37:6c:15:d0:a6:9f: | |
| a7:1a:b6:e0:5c:d8:3e:6a:37:ce:5b:32:8e:74:9d: | |
| e4:d8:e6:54:4e:f0:2e:34:22:a3:0e:40:d8:dc:b5: | |
| 72:55:dc:43:64:22:80:f0:82:c7:c6:b8:29:2e:63: | |
| ac:fb:2d:ec:12:e3:39:19:73:a5:27:cc:93:4c:96: | |
| 40:e6:96:22:f3:c7:d3:c7:a6:54:d6:0b:07:73:ae: | |
| 8e:bd:56:59:41:9b:7f:93:d9:bd:e4:58:7d:31:a7: | |
| 36:2a:32:47:9c:13:d0:6c:2a:52:29:89:05:3d:99: | |
| ab:cb:b7:88:f3:b0:f0:bd:85:64:3a:eb:9b:96:43: | |
| 4a:0f | |
| Exponent: 65537 (0x10001) | |
| X509v3 extensions: | |
| X509v3 Key Usage: critical | |
| Digital Signature, Key Encipherment | |
| X509v3 Extended Key Usage: | |
| TLS Web Client Authentication, TLS Web Server Authentication | |
| X509v3 Subject Alternative Name: | |
| DNS:etcd-b, DNS:etcd-b.internal.k8s.ctnror0.dev.mmerrill.net, IP Address:127.0.0.1 | |
| Signature Algorithm: sha256WithRSAEncryption | |
| 4e:66:9a:80:01:ff:6d:c4:d3:59:7a:75:de:3f:a8:90:f9:a3: | |
| 49:1e:c7:94:1d:32:94:e4:3f:48:f6:8e:2c:83:37:03:e0:bd: | |
| 9c:db:c8:28:53:81:ce:ba:dc:d8:bc:08:c0:a8:27:28:c0:1f: | |
| 36:91:1d:8a:6e:92:e2:ba:73:cc:68:1f:50:2d:df:b8:06:dd: | |
| 47:f3:39:ee:91:2f:c3:2e:2c:f7:79:78:87:38:97:b8:c2:1b: | |
| d9:a2:de:23:ee:78:34:b7:0d:4d:83:9c:8a:2e:f0:1c:19:cb: | |
| d0:29:c9:2d:23:91:95:98:a1:a3:85:fc:dc:e8:3c:7c:15:4f: | |
| d3:5b:68:5f:98:d0:7d:96:53:e0:de:70:71:67:07:2b:f0:f1: | |
| 25:53:8e:ab:fd:65:d7:2b:a9:17:4e:79:86:dc:63:44:75:7f: | |
| 9a:d7:94:e6:d4:3b:73:08:b9:db:cd:f5:b6:6e:e3:14:32:f9: | |
| 76:62:f9:1c:d7:2c:7a:ab:b0:3e:c3:a9:16:2d:63:f4:4c:31: | |
| 15:fe:aa:df:9f:d0:65:55:6a:ef:82:74:ff:1a:93:0f:a5:db: | |
| 1a:28:b3:a9:92:44:26:e5:d6:15:ac:09:31:62:8a:e7:eb:9b: | |
| 2a:24:83:02:76:0d:a6:55:6d:0f:e7:2a:4a:26:89:9c:bc:95: | |
| bc:fe:bc:03 | |
| Restart docker with sudo docker restart $(sudo docker ps -q -f "label=io.kubernetes.container.name=etcd-manager") on etcd-b | |
| etcd-manager | |
| I1217 21:09:47.292145 26945 volumes.go:86] AWS API Request: ec2metadata/GetToken | |
| I1217 21:09:47.292788 26945 volumes.go:86] AWS API Request: ec2metadata/GetDynamicData | |
| I1217 21:09:47.293300 26945 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 21:09:47.293777 26945 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 21:09:47.294268 26945 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 21:09:47.294838 26945 main.go:279] Mounting available etcd volumes matching tags [k8s.io/etcd/main k8s.io/role/master=1 kubernetes.io/cluster/k8s.ctnror0.dev.mmerrill.net=owned]; nameTag=k8s.io/etcd/main | |
| I1217 21:09:47.296626 26945 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 21:09:47.415450 26945 mounter.go:72] Master volume "vol-0c4d8a578a28c993f" is attached at "/dev/xvdv" | |
| I1217 21:09:47.415471 26945 mounter.go:86] Doing safe-format-and-mount of /dev/xvdv to /mnt/master-vol-0c4d8a578a28c993f | |
| I1217 21:09:47.415483 26945 volumes.go:234] volume vol-0c4d8a578a28c993f not mounted at /rootfs/dev/xvdv | |
| I1217 21:09:47.415516 26945 volumes.go:248] found nvme volume "nvme-Amazon_Elastic_Block_Store_vol0c4d8a578a28c993f" at "/dev/nvme2n1" | |
| I1217 21:09:47.415546 26945 mounter.go:125] Found volume "vol-0c4d8a578a28c993f" mounted at device "/dev/nvme2n1" | |
| I1217 21:09:47.416540 26945 mounter.go:179] Device already mounted on "/mnt/master-vol-0c4d8a578a28c993f", verifying it is our device | |
| I1217 21:09:47.416572 26945 mounter.go:191] Found existing mount of "/dev/nvme2n1" at "/mnt/master-vol-0c4d8a578a28c993f" | |
| I1217 21:09:47.418687 26945 mounter.go:232] matched device "/dev/nvme2n1" and "/dev/nvme2n1" via '\x00' | |
| I1217 21:09:47.418704 26945 mounter.go:94] mounted master volume "vol-0c4d8a578a28c993f" on /mnt/master-vol-0c4d8a578a28c993f | |
| I1217 21:09:47.418713 26945 main.go:294] discovered IP address: 10.206.148.211 | |
| I1217 21:09:47.418719 26945 main.go:299] Setting data dir to /rootfs/mnt/master-vol-0c4d8a578a28c993f | |
| I1217 21:09:47.419124 26945 certs.go:122] existing certificate not valid after 2022-11-24T16:36:39Z; will regenerate | |
| I1217 21:09:47.419134 26945 certs.go:183] generating certificate for "etcd-manager-server-etcd-b" | |
| I1217 21:09:47.422817 26945 certs.go:122] existing certificate not valid after 2022-11-24T16:36:39Z; will regenerate | |
| I1217 21:09:47.422833 26945 certs.go:183] generating certificate for "etcd-manager-client-etcd-b" | |
| I1217 21:09:47.426244 26945 server.go:87] starting GRPC server using TLS, ServerName="etcd-manager-server-etcd-b" | |
| I1217 21:09:47.542038 26945 etcdserver.go:550] starting etcd with state cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-a" peer_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-b" peer_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" tls_enabled:true > nodes:<name:"etcd-c" peer_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > > etcd_version:"3.4.13" | |
| I1217 21:09:47.542215 26945 etcdserver.go:559] starting etcd with datadir /rootfs/mnt/master-vol-0c4d8a578a28c993f/data/etcd-cluster-token-etcd | |
| I1217 21:09:47.542248 26945 etcdserver.go:573] overriding clientURLs with [https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:4001] (state had [https://0.0.0.0:4001]) | |
| I1217 21:09:47.542300 26945 etcdserver.go:577] overriding quarantinedClientURLs with [https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:3994] (state had [https://0.0.0.0:3994]) | |
| I1217 21:09:47.542452 26945 pki.go:58] generating peer keypair for etcd: {CommonName:etcd-b Organization:[] AltNames:{DNSNames:[etcd-b etcd-b.internal.k8s.ctnror0.dev.mmerrill.net] IPs:[127.0.0.1]} Usages:[2 1]} | |
| I1217 21:09:47.542727 26945 certs.go:122] existing certificate not valid after 2022-11-24T17:07:52Z; will regenerate | |
| I1217 21:09:47.542739 26945 certs.go:183] generating certificate for "etcd-b" | |
| I1217 21:09:47.545048 26945 pki.go:101] building client-serving certificate: {CommonName:etcd-b Organization:[] AltNames:{DNSNames:[etcd-b etcd-b.internal.k8s.ctnror0.dev.mmerrill.net etcd-b.internal.k8s.ctnror0.dev.mmerrill.net] IPs:[127.0.0.1 127.0.0.1]} Usages:[1 2]} | |
| I1217 21:09:47.545278 26945 certs.go:122] existing certificate not valid after 2022-11-24T17:07:52Z; will regenerate | |
| I1217 21:09:47.545296 26945 certs.go:183] generating certificate for "etcd-b" | |
| I1217 21:09:47.677575 26945 certs.go:183] generating certificate for "etcd-manager-etcd-b" | |
| I1217 21:09:47.680654 26945 server.go:105] GRPC server listening on "10.206.148.211:3996" | |
| I1217 21:09:47.681306 26945 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 21:09:47.738313 26945 certs.go:183] generating certificate for "etcd-b" | |
| I1217 21:09:47.741618 26945 etcdprocess.go:196] executing command /opt/etcd-v3.4.13-linux-amd64/etcd [/opt/etcd-v3.4.13-linux-amd64/etcd] | |
| I1217 21:09:47.743830 26945 volumes.go:86] AWS API Request: ec2/DescribeInstances | |
| 2020-12-17 21:09:47.751041 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:4001 | |
| 2020-12-17 21:09:47.751081 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/clients/server.crt | |
| 2020-12-17 21:09:47.751089 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true | |
| 2020-12-17 21:09:47.751101 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/rootfs/mnt/master-vol-0c4d8a578a28c993f/data/etcd-cluster-token-etcd | |
| 2020-12-17 21:09:47.751112 I | pkg/flags: recognized and used environment variable ETCD_ENABLE_V2=false | |
| 2020-12-17 21:09:47.751134 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 21:09:47.751141 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-a=https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380,etcd-b=https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380,etcd-c=https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 21:09:47.751147 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing | |
| 2020-12-17 21:09:47.751154 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-token-etcd | |
| 2020-12-17 21:09:47.751159 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/clients/server.key | |
| 2020-12-17 21:09:47.751167 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:4001 | |
| 2020-12-17 21:09:47.751175 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380 | |
| 2020-12-17 21:09:47.751189 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-b | |
| 2020-12-17 21:09:47.751200 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/peers/me.crt | |
| 2020-12-17 21:09:47.751205 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true | |
| 2020-12-17 21:09:47.751212 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/peers/me.key | |
| 2020-12-17 21:09:47.751218 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/peers/ca.crt | |
| 2020-12-17 21:09:47.751234 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/clients/ca.crt | |
| 2020-12-17 21:09:47.751246 W | pkg/flags: unrecognized environment variable ETCD_LISTEN_METRICS_URLS= | |
| [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead | |
| 2020-12-17 21:09:47.751278 I | etcdmain: etcd Version: 3.4.13 | |
| 2020-12-17 21:09:47.751283 I | etcdmain: Git SHA: ae9734ed2 | |
| 2020-12-17 21:09:47.751286 I | etcdmain: Go Version: go1.12.17 | |
| 2020-12-17 21:09:47.751290 I | etcdmain: Go OS/Arch: linux/amd64 | |
| 2020-12-17 21:09:47.751294 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4 | |
| 2020-12-17 21:09:47.751352 N | etcdmain: the server is already initialized as member before, starting as etcd member... | |
| [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead | |
| 2020-12-17 21:09:47.751375 I | embed: peerTLS: cert = /rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/peers/me.crt, key = /rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/peers/me.key, trusted-ca = /rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/peers/ca.crt, client-cert-auth = true, crl-file = | |
| 2020-12-17 21:09:47.751954 I | embed: name = etcd-b | |
| 2020-12-17 21:09:47.751964 I | embed: data dir = /rootfs/mnt/master-vol-0c4d8a578a28c993f/data/etcd-cluster-token-etcd | |
| 2020-12-17 21:09:47.751969 I | embed: member dir = /rootfs/mnt/master-vol-0c4d8a578a28c993f/data/etcd-cluster-token-etcd/member | |
| 2020-12-17 21:09:47.751973 I | embed: heartbeat = 100ms | |
| 2020-12-17 21:09:47.751976 I | embed: election = 1000ms | |
| 2020-12-17 21:09:47.751980 I | embed: snapshot count = 100000 | |
| 2020-12-17 21:09:47.751988 I | embed: advertise client URLs = https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:4001 | |
| 2020-12-17 21:09:47.751993 I | embed: initial advertise peer URLs = https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 21:09:47.751999 I | embed: initial cluster = | |
| 2020-12-17 21:09:47.752027 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0c4d8a578a28c993f/data/etcd-cluster-token-etcd" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| 2020-12-17 21:09:47.752137 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0c4d8a578a28c993f/data/etcd-cluster-token-etcd/member/snap" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| I1217 21:09:47.799270 26945 peers.go:117] found new candidate peer from discovery: etcd-b [{10.206.148.211 0} {10.206.148.211 0}] | |
| I1217 21:09:47.799308 26945 peers.go:117] found new candidate peer from discovery: etcd-c [{10.206.149.50 0} {10.206.149.50 0}] | |
| I1217 21:09:47.799316 26945 peers.go:117] found new candidate peer from discovery: etcd-a [{10.206.147.64 0} {10.206.147.64 0}] | |
| I1217 21:09:47.799334 26945 hosts.go:84] hosts update: primary=map[], fallbacks=map[etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.147.64 10.206.147.64] etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.148.211 10.206.148.211] etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.149.50 10.206.149.50]], final=map[10.206.147.64:[etcd-a.internal.k8s.ctnror0.dev.mmerrill.net etcd-a.internal.k8s.ctnror0.dev.mmerrill.net] 10.206.148.211:[etcd-b.internal.k8s.ctnror0.dev.mmerrill.net etcd-b.internal.k8s.ctnror0.dev.mmerrill.net] 10.206.149.50:[etcd-c.internal.k8s.ctnror0.dev.mmerrill.net etcd-c.internal.k8s.ctnror0.dev.mmerrill.net]] | |
| I1217 21:09:47.799356 26945 peers.go:297] connecting to peer "etcd-b" with TLS policy, servername="etcd-manager-server-etcd-b" | |
| I1217 21:09:47.799411 26945 peers.go:297] connecting to peer "etcd-c" with TLS policy, servername="etcd-manager-server-etcd-c" | |
| I1217 21:09:47.799554 26945 peers.go:297] connecting to peer "etcd-a" with TLS policy, servername="etcd-manager-server-etcd-a" | |
| 2020-12-17 21:09:47.834668 I | embed: rejected connection from "10.206.147.64:13194" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| ____________________ | |
| SAME MESSAGE REPEATS | |
| ____________________ | |
| 2020-12-17 21:09:48.539383 I | embed: rejected connection from "10.206.147.64:13288" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 21:09:48.570265 I | etcdserver: recovered store from snapshot at index 857153621 | |
| 2020-12-17 21:09:48.570977 I | mvcc: restore compact to 454393447 | |
| 2020-12-17 21:09:48.639696 I | embed: rejected connection from "10.206.147.64:13298" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| ____________________ | |
| SAME MESSAGE REPEATS | |
| ____________________ | |
| 2020-12-17 21:09:48.842758 I | embed: rejected connection from "10.206.147.64:13340" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 21:09:48.904196 I | etcdserver: restarting member c0179cec2a3f9d9b in cluster 72141c0e48dbb6f2 at commit index 857239459 | |
| raft2020/12/17 21:09:48 INFO: c0179cec2a3f9d9b switched to configuration voters=(603088857070764962 4909666646868542013 13841704517881535899) | |
| raft2020/12/17 21:09:48 INFO: c0179cec2a3f9d9b became follower at term 49604 | |
| raft2020/12/17 21:09:48 INFO: newRaft c0179cec2a3f9d9b [peers: [85e9a1ec5b303a2,4422a3cd842ec23d,c0179cec2a3f9d9b], term: 49604, commit: 857239459, applied: 857153621, lastindex: 857239460, lastterm: 49604] | |
| 2020-12-17 21:09:48.913859 I | etcdserver/api: enabled capabilities for version 3.4 | |
| 2020-12-17 21:09:48.913882 I | etcdserver/membership: added member 4422a3cd842ec23d [https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 21:09:48.913889 I | etcdserver/membership: added member 85e9a1ec5b303a2 [https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 21:09:48.913907 I | etcdserver/membership: added member c0179cec2a3f9d9b [https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 21:09:48.913915 I | etcdserver/membership: set the cluster version to 3.4 from store | |
| 2020-12-17 21:09:48.913958 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0c4d8a578a28c993f/data/etcd-cluster-token-etcd/member" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| 2020-12-17 21:09:48.915899 W | auth: simple token is not cryptographically signed | |
| 2020-12-17 21:09:48.916980 I | mvcc: restore compact to 454393447 | |
| 2020-12-17 21:09:48.931437 I | embed: rejected connection from "10.206.147.64:13352" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 21:09:48.936880 I | rafthttp: starting peer 85e9a1ec5b303a2... | |
| 2020-12-17 21:09:48.936921 I | rafthttp: started HTTP pipelining with peer 85e9a1ec5b303a2 | |
| 2020-12-17 21:09:48.937146 I | embed: rejected connection from "10.206.147.64:13354" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 21:09:48.937213 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (writer) | |
| 2020-12-17 21:09:48.937934 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (writer) | |
| 2020-12-17 21:09:48.938273 I | rafthttp: started peer 85e9a1ec5b303a2 | |
| 2020-12-17 21:09:48.938297 I | rafthttp: added peer 85e9a1ec5b303a2 | |
| 2020-12-17 21:09:48.938302 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (stream Message reader) | |
| 2020-12-17 21:09:48.938310 I | rafthttp: starting peer 4422a3cd842ec23d... | |
| 2020-12-17 21:09:48.938382 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (stream MsgApp v2 reader) | |
| 2020-12-17 21:09:48.938768 I | rafthttp: started HTTP pipelining with peer 4422a3cd842ec23d | |
| 2020-12-17 21:09:48.939201 I | rafthttp: started streaming with peer 4422a3cd842ec23d (writer) | |
| 2020-12-17 21:09:48.939301 I | rafthttp: started streaming with peer 4422a3cd842ec23d (writer) | |
| 2020-12-17 21:09:48.939804 I | rafthttp: started peer 4422a3cd842ec23d | |
| 2020-12-17 21:09:48.939813 I | rafthttp: started streaming with peer 4422a3cd842ec23d (stream MsgApp v2 reader) | |
| 2020-12-17 21:09:48.939823 I | rafthttp: added peer 4422a3cd842ec23d | |
| 2020-12-17 21:09:48.939869 I | etcdserver: starting server... [version: 3.4.13, cluster version: 3.4] | |
| 2020-12-17 21:09:48.940016 I | rafthttp: started streaming with peer 4422a3cd842ec23d (stream Message reader) | |
| 2020-12-17 21:09:48.942533 I | embed: ClientTLS: cert = /rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/clients/server.crt, key = /rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/clients/server.key, trusted-ca = /rootfs/mnt/master-vol-0c4d8a578a28c993f/pki/etcd-cluster-token-etcd/clients/ca.crt, client-cert-auth = true, crl-file = | |
| 2020-12-17 21:09:48.942633 I | embed: listening for peers on [::]:2380 | |
| 2020-12-17 21:09:48.942935 I | rafthttp: peer 4422a3cd842ec23d became active | |
| 2020-12-17 21:09:48.942956 I | rafthttp: established a TCP streaming connection with peer 4422a3cd842ec23d (stream MsgApp v2 writer) | |
| raft2020/12/17 21:09:48 INFO: raft.node: c0179cec2a3f9d9b elected leader 4422a3cd842ec23d at term 49604 | |
| 2020-12-17 21:09:48.943855 I | rafthttp: established a TCP streaming connection with peer 4422a3cd842ec23d (stream Message writer) | |
| 2020-12-17 21:09:48.951876 I | rafthttp: established a TCP streaming connection with peer 4422a3cd842ec23d (stream Message reader) | |
| 2020-12-17 21:09:48.953085 I | rafthttp: established a TCP streaming connection with peer 4422a3cd842ec23d (stream MsgApp v2 reader) | |
| 2020-12-17 21:09:49.034058 I | embed: rejected connection from "10.206.147.64:13364" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| ____________________ | |
| SAME MESSAGE REPEATS | |
| ____________________ | |
| 2020-12-17 21:09:49.635858 I | embed: rejected connection from "10.206.147.64:13444" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| I1217 21:09:49.681165 26945 controller.go:189] starting controller iteration | |
| I1217 21:09:49.681215 26945 controller.go:214] we are not leader | |
| 2020-12-17 21:09:49.728880 I | embed: rejected connection from "10.206.147.64:13460" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| ____________________ | |
| SAME MESSAGE REPEATS | |
| ____________________ | |
| 2020-12-17 21:09:49.928805 I | embed: rejected connection from "10.206.147.64:13486" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| raft2020/12/17 21:09:49 INFO: c0179cec2a3f9d9b [term: 49604] received a MsgVote message with higher term from 4422a3cd842ec23d [term: 49605] | |
| raft2020/12/17 21:09:49 INFO: c0179cec2a3f9d9b became follower at term 49605 | |
| raft2020/12/17 21:09:49 INFO: c0179cec2a3f9d9b [logterm: 49604, index: 857239460, vote: 0] cast MsgVote for 4422a3cd842ec23d [logterm: 49604, index: 857239460] at term 49605 | |
| raft2020/12/17 21:09:49 INFO: raft.node: c0179cec2a3f9d9b lost leader 4422a3cd842ec23d at term 49605 | |
| raft2020/12/17 21:09:49 INFO: raft.node: c0179cec2a3f9d9b elected leader 4422a3cd842ec23d at term 49605 | |
| 2020-12-17 21:09:49.937199 I | embed: rejected connection from "10.206.147.64:15688" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 21:09:49.940654 I | etcdserver: published {Name:etcd-b ClientURLs:[https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:4001]} to cluster 72141c0e48dbb6f2 | |
| 2020-12-17 21:09:49.940672 I | embed: ready to serve client requests | |
| 2020-12-17 21:09:49.941800 I | embed: serving client requests on [::]:4001 | |
| 2020-12-17 21:09:50.071812 I | embed: rejected connection from "10.206.147.64:15700" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| ____________________ | |
| SAME MESSAGE REPEATS | |
| ____________________ | |
| 2020-12-17 21:09:52.763769 I | embed: rejected connection from "10.206.147.64:16066" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-b.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 21:09:52.836120 I | rafthttp: peer 85e9a1ec5b303a2 became active | |
| 2020-12-17 21:09:52.836144 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream Message writer) | |
| 2020-12-17 21:09:52.851893 I | etcdserver: c0179cec2a3f9d9b initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s) | |
| 2020-12-17 21:09:52.855654 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream MsgApp v2 writer) | |
| 2020-12-17 21:09:52.938846 E | rafthttp: failed to dial 85e9a1ec5b303a2 on stream Message (dial tcp 10.206.147.234:2380: i/o timeout) | |
| 2020-12-17 21:09:52.938924 I | rafthttp: peer 85e9a1ec5b303a2 became inactive (message send to peer failed) | |
| 2020-12-17 21:09:52.960335 I | rafthttp: peer 85e9a1ec5b303a2 became active | |
| 2020-12-17 21:09:52.960467 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream MsgApp v2 reader) | |
| 2020-12-17 21:09:52.961162 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream Message reader) | |
| raft2020/12/17 21:09:53 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239490, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:09:55 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239499, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:09:56 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239509, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:09:57 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239513, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:09:59 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239524, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| I1217 21:09:59.682422 26945 controller.go:189] starting controller iteration | |
| I1217 21:09:59.682461 26945 controller.go:214] we are not leader | |
| raft2020/12/17 21:10:01 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239536, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:02 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239543, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:03 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239551, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:04 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239561, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:06 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239575, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:08 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239578, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| I1217 21:10:09.683644 26945 controller.go:189] starting controller iteration | |
| I1217 21:10:09.683681 26945 controller.go:214] we are not leader | |
| raft2020/12/17 21:10:09 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239586, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:10 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239597, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:12 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239606, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:14 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239621, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:15 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239637, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:16 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239645, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:17 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239645, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:19 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239657, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| I1217 21:10:19.684881 26945 controller.go:189] starting controller iteration | |
| I1217 21:10:19.684922 26945 controller.go:214] we are not leader | |
| raft2020/12/17 21:10:21 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239669, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:22 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239676, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:23 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239685, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:24 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239694, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| raft2020/12/17 21:10:26 INFO: c0179cec2a3f9d9b [logterm: 49605, index: 857239701, vote: 4422a3cd842ec23d] ignored MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 49605: lease is not expired (remaining ticks: 10) | |
| Back on etcd-a: | |
| 2020-12-17 21:09:52.836498 I | rafthttp: peer c0179cec2a3f9d9b became active | |
| 2020-12-17 21:09:52.836520 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream Message reader) | |
| 2020-12-17 21:09:52.855937 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream MsgApp v2 reader) | |
| 2020-12-17 21:09:52.960023 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream MsgApp v2 writer) | |
| 2020-12-17 21:09:52.960635 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream Message writer) | |
| raft2020/12/17 21:09:53 INFO: 85e9a1ec5b303a2 is starting a new election at term 50773 | |
| raft2020/12/17 21:09:53 INFO: 85e9a1ec5b303a2 became candidate at term 50774 | |
| raft2020/12/17 21:09:53 INFO: 85e9a1ec5b303a2 received MsgVoteResp from 85e9a1ec5b303a2 at term 50774 | |
| Now, back on etcd-c (10.206.219.50), need to restart that one too, because etcd-a still can't heartbeat with it | |
| from etcd-a: | |
| 2020-12-17 21:13:04.237330 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| 2020-12-17 21:13:04.237508 W | rafthttp: health check for peer 4422a3cd842ec23d could not connect: EOF | |
| from etcd-c: | |
| 2020-12-17 21:17:15.040840 I | embed: rejected connection from "10.206.147.64:45856" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-c.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| 2020-12-17 21:17:15.076905 I | embed: rejected connection from "10.206.147.64:45862" (error "tls: \"10.206.147.64\" does not match any of DNSNames [\"etcd-a\" \"etcd-a.internal.k8s.ctnror0.dev.mmerrill.net\"] (lookup etcd-a on 10.206.144.2:53: no such host)", ServerName "etcd-c.internal.k8s.ctnror0.dev.mmerrill.net", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a" "etcd-a.internal.k8s.ctnror0.dev.mmerrill.net"]) | |
| Restart docker on etcd-c | |
| Restart docker with sudo docker restart $(sudo docker ps -q -f "label=io.kubernetes.container.name=etcd-manager") on etcd-c | |
| From etcd-c | |
| etcd-manager | |
| I1217 21:18:10.952342 26074 volumes.go:86] AWS API Request: ec2metadata/GetToken | |
| I1217 21:18:10.953077 26074 volumes.go:86] AWS API Request: ec2metadata/GetDynamicData | |
| I1217 21:18:10.953650 26074 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 21:18:10.954114 26074 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 21:18:10.954577 26074 volumes.go:86] AWS API Request: ec2metadata/GetMetadata | |
| I1217 21:18:10.955054 26074 main.go:279] Mounting available etcd volumes matching tags [k8s.io/etcd/main k8s.io/role/master=1 kubernetes.io/cluster/k8s.ctnror0.dev.mmerrill.net=owned]; nameTag=k8s.io/etcd/main | |
| I1217 21:18:10.956491 26074 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 21:18:11.136665 26074 mounter.go:72] Master volume "vol-0cbe3e24576916464" is attached at "/dev/xvdu" | |
| I1217 21:18:11.136722 26074 mounter.go:86] Doing safe-format-and-mount of /dev/xvdu to /mnt/master-vol-0cbe3e24576916464 | |
| I1217 21:18:11.136760 26074 volumes.go:234] volume vol-0cbe3e24576916464 not mounted at /rootfs/dev/xvdu | |
| I1217 21:18:11.136804 26074 volumes.go:248] found nvme volume "nvme-Amazon_Elastic_Block_Store_vol0cbe3e24576916464" at "/dev/nvme1n1" | |
| I1217 21:18:11.136834 26074 mounter.go:125] Found volume "vol-0cbe3e24576916464" mounted at device "/dev/nvme1n1" | |
| I1217 21:18:11.137662 26074 mounter.go:179] Device already mounted on "/mnt/master-vol-0cbe3e24576916464", verifying it is our device | |
| I1217 21:18:11.137692 26074 mounter.go:191] Found existing mount of "/dev/nvme1n1" at "/mnt/master-vol-0cbe3e24576916464" | |
| I1217 21:18:11.138698 26074 mounter.go:232] matched device "/dev/nvme1n1" and "/dev/nvme1n1" via '\x00' | |
| I1217 21:18:11.138712 26074 mounter.go:94] mounted master volume "vol-0cbe3e24576916464" on /mnt/master-vol-0cbe3e24576916464 | |
| I1217 21:18:11.138723 26074 main.go:294] discovered IP address: 10.206.149.50 | |
| I1217 21:18:11.138728 26074 main.go:299] Setting data dir to /rootfs/mnt/master-vol-0cbe3e24576916464 | |
| I1217 21:18:11.139275 26074 certs.go:122] existing certificate not valid after 2022-11-24T16:08:05Z; will regenerate | |
| I1217 21:18:11.139291 26074 certs.go:183] generating certificate for "etcd-manager-server-etcd-c" | |
| I1217 21:18:11.142861 26074 certs.go:122] existing certificate not valid after 2022-11-24T16:08:06Z; will regenerate | |
| I1217 21:18:11.142877 26074 certs.go:183] generating certificate for "etcd-manager-client-etcd-c" | |
| I1217 21:18:11.145901 26074 server.go:87] starting GRPC server using TLS, ServerName="etcd-manager-server-etcd-c" | |
| I1217 21:18:11.218212 26074 etcdserver.go:550] starting etcd with state cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-a" peer_urls:"https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-b" peer_urls:"https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-c" peer_urls:"https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" tls_enabled:true > > etcd_version:"3.4.13" | |
| I1217 21:18:11.218682 26074 etcdserver.go:559] starting etcd with datadir /rootfs/mnt/master-vol-0cbe3e24576916464/data/etcd-cluster-token-etcd | |
| I1217 21:18:11.218976 26074 etcdserver.go:573] overriding clientURLs with [https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:4001] (state had [https://0.0.0.0:4001]) | |
| I1217 21:18:11.219006 26074 etcdserver.go:577] overriding quarantinedClientURLs with [https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:3994] (state had [https://0.0.0.0:3994]) | |
| I1217 21:18:11.219169 26074 pki.go:58] generating peer keypair for etcd: {CommonName:etcd-c Organization:[] AltNames:{DNSNames:[etcd-c etcd-c.internal.k8s.ctnror0.dev.mmerrill.net] IPs:[127.0.0.1]} Usages:[2 1]} | |
| I1217 21:18:11.219581 26074 certs.go:122] existing certificate not valid after 2022-11-24T17:07:48Z; will regenerate | |
| I1217 21:18:11.219610 26074 certs.go:183] generating certificate for "etcd-c" | |
| I1217 21:18:11.223063 26074 pki.go:101] building client-serving certificate: {CommonName:etcd-c Organization:[] AltNames:{DNSNames:[etcd-c etcd-c.internal.k8s.ctnror0.dev.mmerrill.net etcd-c.internal.k8s.ctnror0.dev.mmerrill.net] IPs:[127.0.0.1 127.0.0.1]} Usages:[1 2]} | |
| I1217 21:18:11.223273 26074 certs.go:122] existing certificate not valid after 2022-11-24T17:07:48Z; will regenerate | |
| I1217 21:18:11.223285 26074 certs.go:183] generating certificate for "etcd-c" | |
| I1217 21:18:11.380381 26074 certs.go:183] generating certificate for "etcd-manager-etcd-c" | |
| I1217 21:18:11.390201 26074 server.go:105] GRPC server listening on "10.206.149.50:3996" | |
| I1217 21:18:11.390537 26074 volumes.go:86] AWS API Request: ec2/DescribeVolumes | |
| I1217 21:18:11.454425 26074 volumes.go:86] AWS API Request: ec2/DescribeInstances | |
| I1217 21:18:11.517304 26074 certs.go:183] generating certificate for "etcd-c" | |
| I1217 21:18:11.520283 26074 etcdprocess.go:196] executing command /opt/etcd-v3.4.13-linux-amd64/etcd [/opt/etcd-v3.4.13-linux-amd64/etcd] | |
| I1217 21:18:11.521071 26074 peers.go:117] found new candidate peer from discovery: etcd-b [{10.206.148.211 0} {10.206.148.211 0}] | |
| I1217 21:18:11.521099 26074 peers.go:117] found new candidate peer from discovery: etcd-c [{10.206.149.50 0} {10.206.149.50 0}] | |
| I1217 21:18:11.521106 26074 peers.go:117] found new candidate peer from discovery: etcd-a [{10.206.147.64 0} {10.206.147.64 0}] | |
| I1217 21:18:11.521124 26074 hosts.go:84] hosts update: primary=map[], fallbacks=map[etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.147.64 10.206.147.64] etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.148.211 10.206.148.211] etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:[10.206.149.50 10.206.149.50]], final=map[10.206.147.64:[etcd-a.internal.k8s.ctnror0.dev.mmerrill.net etcd-a.internal.k8s.ctnror0.dev.mmerrill.net] 10.206.148.211:[etcd-b.internal.k8s.ctnror0.dev.mmerrill.net etcd-b.internal.k8s.ctnror0.dev.mmerrill.net] 10.206.149.50:[etcd-c.internal.k8s.ctnror0.dev.mmerrill.net etcd-c.internal.k8s.ctnror0.dev.mmerrill.net]] | |
| I1217 21:18:11.521287 26074 peers.go:297] connecting to peer "etcd-a" with TLS policy, servername="etcd-manager-server-etcd-a" | |
| I1217 21:18:11.521542 26074 peers.go:297] connecting to peer "etcd-b" with TLS policy, servername="etcd-manager-server-etcd-b" | |
| I1217 21:18:11.521639 26074 peers.go:297] connecting to peer "etcd-c" with TLS policy, servername="etcd-manager-server-etcd-c" | |
| 2020-12-17 21:18:11.532588 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:4001 | |
| 2020-12-17 21:18:11.532649 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/clients/server.crt | |
| 2020-12-17 21:18:11.532667 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true | |
| 2020-12-17 21:18:11.532686 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/rootfs/mnt/master-vol-0cbe3e24576916464/data/etcd-cluster-token-etcd | |
| 2020-12-17 21:18:11.532716 I | pkg/flags: recognized and used environment variable ETCD_ENABLE_V2=false | |
| 2020-12-17 21:18:11.532757 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 21:18:11.532772 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-a=https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380,etcd-b=https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380,etcd-c=https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 21:18:11.532786 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing | |
| 2020-12-17 21:18:11.532810 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-token-etcd | |
| 2020-12-17 21:18:11.532830 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/clients/server.key | |
| 2020-12-17 21:18:11.532847 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:4001 | |
| 2020-12-17 21:18:11.532863 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380 | |
| 2020-12-17 21:18:11.532885 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-c | |
| 2020-12-17 21:18:11.533288 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/peers/me.crt | |
| 2020-12-17 21:18:11.533316 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true | |
| 2020-12-17 21:18:11.533340 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/peers/me.key | |
| 2020-12-17 21:18:11.533361 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/peers/ca.crt | |
| 2020-12-17 21:18:11.533386 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/clients/ca.crt | |
| 2020-12-17 21:18:11.533405 W | pkg/flags: unrecognized environment variable ETCD_LISTEN_METRICS_URLS= | |
| [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead | |
| 2020-12-17 21:18:11.533471 I | etcdmain: etcd Version: 3.4.13 | |
| 2020-12-17 21:18:11.533484 I | etcdmain: Git SHA: ae9734ed2 | |
| 2020-12-17 21:18:11.533496 I | etcdmain: Go Version: go1.12.17 | |
| 2020-12-17 21:18:11.533508 I | etcdmain: Go OS/Arch: linux/amd64 | |
| 2020-12-17 21:18:11.533520 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4 | |
| 2020-12-17 21:18:11.533596 N | etcdmain: the server is already initialized as member before, starting as etcd member... | |
| [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead | |
| 2020-12-17 21:18:11.533695 I | embed: peerTLS: cert = /rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/peers/me.crt, key = /rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/peers/me.key, trusted-ca = /rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/peers/ca.crt, client-cert-auth = true, crl-file = | |
| 2020-12-17 21:18:11.534264 I | embed: name = etcd-c | |
| 2020-12-17 21:18:11.534303 I | embed: data dir = /rootfs/mnt/master-vol-0cbe3e24576916464/data/etcd-cluster-token-etcd | |
| 2020-12-17 21:18:11.534319 I | embed: member dir = /rootfs/mnt/master-vol-0cbe3e24576916464/data/etcd-cluster-token-etcd/member | |
| 2020-12-17 21:18:11.534332 I | embed: heartbeat = 100ms | |
| 2020-12-17 21:18:11.534344 I | embed: election = 1000ms | |
| 2020-12-17 21:18:11.534356 I | embed: snapshot count = 100000 | |
| 2020-12-17 21:18:11.534386 I | embed: advertise client URLs = https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:4001 | |
| 2020-12-17 21:18:11.534400 I | embed: initial advertise peer URLs = https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380 | |
| 2020-12-17 21:18:11.534414 I | embed: initial cluster = | |
| 2020-12-17 21:18:11.534462 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0cbe3e24576916464/data/etcd-cluster-token-etcd" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| 2020-12-17 21:18:11.534615 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0cbe3e24576916464/data/etcd-cluster-token-etcd/member/snap" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| 2020-12-17 21:18:12.323813 I | etcdserver: recovered store from snapshot at index 857191015 | |
| 2020-12-17 21:18:12.324467 I | mvcc: restore compact to 454395198 | |
| 2020-12-17 21:18:12.605919 I | etcdserver: restarting member 4422a3cd842ec23d in cluster 72141c0e48dbb6f2 at commit index 857242635 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d switched to configuration voters=(603088857070764962 4909666646868542013 13841704517881535899) | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d became follower at term 49605 | |
| raft2020/12/17 21:18:12 INFO: newRaft 4422a3cd842ec23d [peers: [85e9a1ec5b303a2,4422a3cd842ec23d,c0179cec2a3f9d9b], term: 49605, commit: 857242635, applied: 857191015, lastindex: 857242636, lastterm: 49605] | |
| 2020-12-17 21:18:12.612162 I | etcdserver/api: enabled capabilities for version 3.4 | |
| 2020-12-17 21:18:12.612175 I | etcdserver/membership: added member 4422a3cd842ec23d [https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 21:18:12.612180 I | etcdserver/membership: added member 85e9a1ec5b303a2 [https://etcd-a.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 21:18:12.612189 I | etcdserver/membership: added member c0179cec2a3f9d9b [https://etcd-b.internal.k8s.ctnror0.dev.mmerrill.net:2380] to cluster 72141c0e48dbb6f2 from store | |
| 2020-12-17 21:18:12.612194 I | etcdserver/membership: set the cluster version to 3.4 from store | |
| 2020-12-17 21:18:12.612226 W | pkg/fileutil: check file permission: directory "/rootfs/mnt/master-vol-0cbe3e24576916464/data/etcd-cluster-token-etcd/member" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data. | |
| 2020-12-17 21:18:12.615185 W | auth: simple token is not cryptographically signed | |
| 2020-12-17 21:18:12.616557 I | mvcc: restore compact to 454395198 | |
| 2020-12-17 21:18:12.634126 I | rafthttp: starting peer 85e9a1ec5b303a2... | |
| 2020-12-17 21:18:12.634158 I | rafthttp: started HTTP pipelining with peer 85e9a1ec5b303a2 | |
| 2020-12-17 21:18:12.634675 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (writer) | |
| 2020-12-17 21:18:12.634774 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (writer) | |
| 2020-12-17 21:18:12.636048 I | rafthttp: started peer 85e9a1ec5b303a2 | |
| 2020-12-17 21:18:12.640218 I | rafthttp: added peer 85e9a1ec5b303a2 | |
| 2020-12-17 21:18:12.640277 I | rafthttp: starting peer c0179cec2a3f9d9b... | |
| 2020-12-17 21:18:12.640337 I | rafthttp: started HTTP pipelining with peer c0179cec2a3f9d9b | |
| 2020-12-17 21:18:12.640897 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (writer) | |
| 2020-12-17 21:18:12.640911 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (stream MsgApp v2 reader) | |
| 2020-12-17 21:18:12.641038 I | rafthttp: started streaming with peer 85e9a1ec5b303a2 (stream Message reader) | |
| 2020-12-17 21:18:12.641892 I | rafthttp: started peer c0179cec2a3f9d9b | |
| 2020-12-17 21:18:12.641916 I | rafthttp: added peer c0179cec2a3f9d9b | |
| 2020-12-17 21:18:12.642077 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (stream MsgApp v2 reader) | |
| 2020-12-17 21:18:12.642317 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (writer) | |
| 2020-12-17 21:18:12.642365 I | rafthttp: started streaming with peer c0179cec2a3f9d9b (stream Message reader) | |
| 2020-12-17 21:18:12.642437 I | etcdserver: starting server... [version: 3.4.13, cluster version: 3.4] | |
| 2020-12-17 21:18:12.647593 I | embed: ClientTLS: cert = /rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/clients/server.crt, key = /rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/clients/server.key, trusted-ca = /rootfs/mnt/master-vol-0cbe3e24576916464/pki/etcd-cluster-token-etcd/clients/ca.crt, client-cert-auth = true, crl-file = | |
| 2020-12-17 21:18:12.648041 I | embed: listening for peers on [::]:2380 | |
| 2020-12-17 21:18:12.648525 I | rafthttp: peer 85e9a1ec5b303a2 became active | |
| 2020-12-17 21:18:12.648578 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream Message writer) | |
| 2020-12-17 21:18:12.649240 I | rafthttp: peer c0179cec2a3f9d9b became active | |
| 2020-12-17 21:18:12.649360 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream Message writer) | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d [term: 49605] received a MsgVote message with higher term from 85e9a1ec5b303a2 [term: 51119] | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d became follower at term 51119 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d [logterm: 49605, index: 857242636, vote: 0] rejected MsgVote from 85e9a1ec5b303a2 [logterm: 49603, index: 857228236] at term 51119 | |
| 2020-12-17 21:18:12.649830 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream MsgApp v2 writer) | |
| 2020-12-17 21:18:12.650067 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream MsgApp v2 writer) | |
| 2020-12-17 21:18:12.652370 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream Message reader) | |
| 2020-12-17 21:18:12.656861 I | rafthttp: established a TCP streaming connection with peer 85e9a1ec5b303a2 (stream MsgApp v2 reader) | |
| 2020-12-17 21:18:12.660164 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream MsgApp v2 reader) | |
| 2020-12-17 21:18:12.662438 I | rafthttp: established a TCP streaming connection with peer c0179cec2a3f9d9b (stream Message reader) | |
| 2020-12-17 21:18:12.695635 I | etcdserver: 4422a3cd842ec23d initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s) | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d is starting a new election at term 51119 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d became candidate at term 51120 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d received MsgVoteResp from 4422a3cd842ec23d at term 51120 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d [logterm: 49605, index: 857242636] sent MsgVote request to 85e9a1ec5b303a2 at term 51120 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d [logterm: 49605, index: 857242636] sent MsgVote request to c0179cec2a3f9d9b at term 51120 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d received MsgVoteResp from 85e9a1ec5b303a2 at term 51120 | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d has received 2 MsgVoteResp votes and 0 vote rejections | |
| raft2020/12/17 21:18:12 INFO: 4422a3cd842ec23d became leader at term 51120 | |
| raft2020/12/17 21:18:12 INFO: raft.node: 4422a3cd842ec23d elected leader 4422a3cd842ec23d at term 51120 | |
| 2020-12-17 21:18:12.920372 I | etcdserver: published {Name:etcd-c ClientURLs:[https://etcd-c.internal.k8s.ctnror0.dev.mmerrill.net:4001]} to cluster 72141c0e48dbb6f2 | |
| 2020-12-17 21:18:12.920422 I | embed: ready to serve client requests | |
| 2020-12-17 21:18:12.921606 I | embed: serving client requests on [::]:4001 | |
| 2020-12-17 21:18:13.162097 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 12.37078ms, to c0179cec2a3f9d9b) | |
| 2020-12-17 21:18:13.162117 W | etcdserver: server is likely overloaded | |
| 2020-12-17 21:18:13.162124 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 12.405754ms, to 85e9a1ec5b303a2) | |
| 2020-12-17 21:18:13.162127 W | etcdserver: server is likely overloaded | |
| I1217 21:18:13.390450 26074 controller.go:189] starting controller iteration | |
| I1217 21:18:13.390508 26074 controller.go:214] we are not leader | |
| I1217 21:18:23.391726 26074 controller.go:189] starting controller iteration | |
| I1217 21:18:23.391777 26074 controller.go:214] we are not leader | |
| after a couple minutes, API server comes back, and kubelet registers etcd-a again, and we have three members of cluster.. | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment