Created
April 23, 2022 03:01
-
-
Save kmahyyg/e87506038fdf85fb05731cf79948ba0c to your computer and use it in GitHub Desktop.
k8s-kubeadm-bug-log
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@k8shostcp1 ~]# crictl logs 3fe6d0e31fd66 | |
| I0423 02:57:35.900998 1 server.go:565] external host was not specified, using 192.168.56.10 | |
| I0423 02:57:35.901535 1 server.go:172] Version: v1.23.6 | |
| I0423 02:57:36.239754 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer | |
| I0423 02:57:36.240076 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. | |
| I0423 02:57:36.240103 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. | |
| I0423 02:57:36.240893 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. | |
| I0423 02:57:36.240917 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. | |
| W0423 02:57:36.243476 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:37.241971 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:37.245310 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:38.242705 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:38.695670 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:40.125488 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:41.423115 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:42.482327 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:45.135239 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:46.156245 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:52.535294 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| W0423 02:57:53.675466 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| E0423 02:57:56.244364 1 run.go:74] "command failed" err="context deadline exceeded" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 127.0.0.1 localhost | |
| ::1 localhost | |
| 127.0.1.1 k8shostcp1.localdomain k8shostcp1 | |
| 192.168.56.10 k8shostcp |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@k8shostcp1 ~]# crictl ps -a | |
| CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID | |
| 3fe6d0e31fd66 8fa62c12256df About a minute ago Exited kube-apiserver 17 2c75621d438db | |
| 354a677266860 df7b72818ad2e 2 minutes ago Exited kube-controller-manager 20 0143f5526c298 | |
| 229296794bcda 25f8c7f3da61c 4 minutes ago Exited etcd 41 cf1534393031f | |
| 932285475f4f7 595f327f224a4 4 minutes ago Exited kube-scheduler 20 e9d693c10064f | |
| 109d9d0637f9a 4c03754524064 6 minutes ago Exited kube-proxy 19 03edbf86309f0 | |
| [root@k8shostcp1 ~]# crictl logs 229296794bcda | |
| {"level":"info","ts":"2022-04-23T02:54:42.724Z","caller":"etcdmain/etcd.go:72","msg":"Running: ","args":["etcd","--advertise-client-urls=https://192.168.56.10:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--initial-advertise-peer-urls=https://192.168.56.10:2380","--initial-cluster=k8shostcp1=https://192.168.56.10:2380","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://192.168.56.10:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://192.168.56.10:2380","--name=k8shostcp1","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]} | |
| {"level":"info","ts":"2022-04-23T02:54:42.724Z","caller":"etcdmain/etcd.go:115","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.724Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://192.168.56.10:2380"]} | |
| {"level":"info","ts":"2022-04-23T02:54:42.724Z","caller":"embed/etcd.go:478","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]} | |
| {"level":"info","ts":"2022-04-23T02:54:42.725Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://192.168.56.10:2379"]} | |
| {"level":"info","ts":"2022-04-23T02:54:42.725Z","caller":"embed/etcd.go:307","msg":"starting an etcd server","etcd-version":"3.5.1","git-sha":"e8732fb5f","go-version":"go1.16.3","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"k8shostcp1","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.56.10:2380"],"listen-peer-urls":["https://192.168.56.10:2380"],"advertise-client-urls":["https://192.168.56.10:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.56.10:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} | |
| {"level":"warn","ts":1650682482.7261639,"caller":"fileutil/fileutil.go:57","msg":"check file permission","error":"directory \"/var/lib/etcd\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.728Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"2.443875ms"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.735Z","caller":"etcdserver/server.go:526","msg":"No snapshot found. Recovering WAL from scratch!"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.744Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"1d4a08974b065330","local-member-id":"f0e4577804b51494","commit-index":2109} | |
| {"level":"info","ts":"2022-04-23T02:54:42.745Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 switched to configuration voters=()"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.745Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 became follower at term 22"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.745Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft f0e4577804b51494 [peers: [], term: 22, commit: 2109, applied: 0, lastindex: 2109, lastterm: 22]"} | |
| {"level":"warn","ts":"2022-04-23T02:54:42.745Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.750Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1742} | |
| {"level":"info","ts":"2022-04-23T02:54:42.761Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.763Z","caller":"etcdserver/server.go:843","msg":"starting etcd server","local-member-id":"f0e4577804b51494","local-server-version":"3.5.1","cluster-version":"to_be_decided"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.764Z","caller":"etcdserver/server.go:744","msg":"starting initial election tick advance","election-ticks":10} | |
| {"level":"info","ts":"2022-04-23T02:54:42.764Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 switched to configuration voters=(17358095036779402388)"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.764Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"1d4a08974b065330","local-member-id":"f0e4577804b51494","added-peer-id":"f0e4577804b51494","added-peer-peer-urls":["https://192.168.56.10:2380"]} | |
| {"level":"info","ts":"2022-04-23T02:54:42.764Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"1d4a08974b065330","local-member-id":"f0e4577804b51494","cluster-version":"3.5"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.764Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.766Z","caller":"embed/etcd.go:687","msg":"starting with client TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]} | |
| {"level":"info","ts":"2022-04-23T02:54:42.766Z","caller":"embed/etcd.go:580","msg":"serving peer traffic","address":"192.168.56.10:2380"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.766Z","caller":"embed/etcd.go:552","msg":"cmux::serve","address":"192.168.56.10:2380"} | |
| {"level":"info","ts":"2022-04-23T02:54:42.766Z","caller":"embed/etcd.go:276","msg":"now serving peer/client/metrics","local-member-id":"f0e4577804b51494","initial-advertise-peer-urls":["https://192.168.56.10:2380"],"listen-peer-urls":["https://192.168.56.10:2380"],"advertise-client-urls":["https://192.168.56.10:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.56.10:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"]} | |
| {"level":"info","ts":"2022-04-23T02:54:42.766Z","caller":"embed/etcd.go:762","msg":"serving metrics","address":"http://127.0.0.1:2381"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.446Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 is starting a new election at term 22"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.446Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 became pre-candidate at term 22"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.446Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 received MsgPreVoteResp from f0e4577804b51494 at term 22"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.446Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 became candidate at term 23"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.446Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 received MsgVoteResp from f0e4577804b51494 at term 23"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.446Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f0e4577804b51494 became leader at term 23"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.446Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: f0e4577804b51494 elected leader f0e4577804b51494 at term 23"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.448Z","caller":"etcdserver/server.go:2027","msg":"published local member to cluster through raft","local-member-id":"f0e4577804b51494","local-member-attributes":"{Name:k8shostcp1 ClientURLs:[https://192.168.56.10:2379]}","request-path":"/0/members/f0e4577804b51494/attributes","cluster-id":"1d4a08974b065330","publish-timeout":"7s"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.448Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.448Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.448Z","caller":"etcdmain/main.go:47","msg":"notifying init daemon"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.448Z","caller":"etcdmain/main.go:53","msg":"successfully notified init daemon"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.449Z","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"127.0.0.1:2379"} | |
| {"level":"info","ts":"2022-04-23T02:54:44.449Z","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"192.168.56.10:2379"} | |
| {"level":"info","ts":"2022-04-23T02:54:50.996Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"} | |
| {"level":"info","ts":"2022-04-23T02:54:50.996Z","caller":"embed/etcd.go:367","msg":"closing etcd server","name":"k8shostcp1","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.56.10:2380"],"advertise-client-urls":["https://192.168.56.10:2379"]} | |
| WARNING: 2022/04/23 02:54:50 [core] grpc: addrConn.createTransport failed to connect to {192.168.56.10:2379 192.168.56.10:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.56.10:2379: connect: connection refused". Reconnecting... | |
| WARNING: 2022/04/23 02:54:50 [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... | |
| {"level":"info","ts":"2022-04-23T02:54:51.000Z","caller":"etcdserver/server.go:1438","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"f0e4577804b51494","current-leader-member-id":"f0e4577804b51494"} | |
| {"level":"info","ts":"2022-04-23T02:54:51.002Z","caller":"embed/etcd.go:562","msg":"stopping serving peer traffic","address":"192.168.56.10:2380"} | |
| {"level":"info","ts":"2022-04-23T02:54:51.002Z","caller":"embed/etcd.go:567","msg":"stopped serving peer traffic","address":"192.168.56.10:2380"} | |
| {"level":"info","ts":"2022-04-23T02:54:51.002Z","caller":"embed/etcd.go:369","msg":"closed etcd server","name":"k8shostcp1","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.56.10:2380"],"advertise-client-urls":["https://192.168.56.10:2379"]} | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment