Skip to content

Instantly share code, notes, and snippets.

@mjudeikis
Last active July 18, 2016 20:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mjudeikis/4857bf6502f935795b46ba2ff4ba9a17 to your computer and use it in GitHub Desktop.
Save mjudeikis/4857bf6502f935795b46ba2ff4ba9a17 to your computer and use it in GitHub Desktop.
[root@k8s-master ~]# journalctl -u k8s-master -f
-- Logs begin at Sat 2016-06-25 21:58:01 EEST. --
Jul 18 23:00:37 k8s-master sh[1610]: E0718 20:00:37.436151 1619 machine.go:176] failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.188247 1619 manager.go:169] Machine: {NumCores:4 CpuFrequency:1200000 MemoryCapacity:967053312 MachineID:f21c916157f34d369cfa9e81943edba7 SystemUUID:f21c916157f34d369cfa9e81943edba7 BootID:a408bc6b-b32d-4316-b5be-3d0b96abae65 Filesystems:[{Device:/dev/root Capacity:62623985664 Type:vfs Inodes:3891776}] DiskMap:map[9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none} 179:0:{Name:mmcblk0 Major:179 Minor:0 Size:63864569856 Scheduler:noop}] NetworkDevices:[{Name:eth0 MacAddress:b8:27:eb:15:c6:af Speed:100 Mtu:1500} {Name:wlan0 MacAddress:b8:27:eb:40:93:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:0 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]} {Id:2 Threads:[2] Caches:[]} {Id:3 Threads:[3] Caches:[]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.192566 1619 manager.go:175] Version: {KernelVersion:4.4.15-1-ARCH ContainerOsVersion:Raspbian GNU/Linux 8 (jessie) DockerVersion:1.10.0 CadvisorVersion: CadvisorRevision:}
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.206908 1619 server.go:319] Using root directory: /var/lib/kubelet
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.208688 1619 server.go:673] Adding manifest file: /etc/kubernetes/manifests-multi
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.208973 1619 file.go:47] Watching path "/etc/kubernetes/manifests-multi"
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.209089 1619 server.go:683] Watching apiserver
Jul 18 23:00:40 k8s-master sh[1610]: W0718 20:00:40.251225 1619 kubelet.go:508] Hairpin mode set to "promiscuous-bridge" but configureCBR0 is false, falling back to "hairpin-veth"
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.251486 1619 kubelet.go:276] Hairpin mode set to "hairpin-veth"
Jul 18 23:00:40 k8s-master sh[1610]: I0718 20:00:40.303146 1619 manager.go:244] Setting dockerRoot to /var/lib/docker
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.331298 1619 plugins.go:56] Registering credential provider: .dockercfg
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.334194 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/aws-ebs"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.334451 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/empty-dir"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.334633 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/gce-pd"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.334818 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/git-repo"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.334997 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/host-path"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.335166 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/nfs"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.335344 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/secret"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.335519 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/iscsi"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.335728 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/glusterfs"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.335937 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/persistent-claim"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.336096 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/rbd"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.336280 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/cinder"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.336516 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/cephfs"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.336735 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/downward-api"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.336901 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/fc"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.337079 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/flocker"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.337266 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/azure-file"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.337454 1619 plugins.go:291] Loaded volume plugin "kubernetes.io/configmap"
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.337837 1619 server.go:632] Started kubelet
Jul 18 23:00:57 k8s-master sh[1610]: E0718 20:00:57.338602 1619 kubelet.go:956] Image garbage collection failed: unable to find data for container /
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.338939 1619 server.go:109] Starting to listen on 0.0.0.0:10250
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.343641 1619 server.go:126] Starting to listen read-only on 0.0.0.0:10255
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.389368 1619 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.389743 1619 manager.go:123] Starting to sync pod status with apiserver
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.390015 1619 kubelet.go:2356] Starting kubelet main sync loop.
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.390195 1619 kubelet.go:2365] skipping pod synchronization - [container runtime is down]
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.571677 1619 factory.go:230] Registering Docker factory
Jul 18 23:00:57 k8s-master sh[1610]: E0718 20:00:57.590752 1619 manager.go:2139] Unable to inspect container "7809586d5b5f887fb14dc465a13ef125efd5a36e59e89b5662585668721c5512": No such container: 7809586d5b5f887fb14dc465a13ef125efd5a36e59e89b5662585668721c5512
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.606084 1619 factory.go:97] Registering Raw factory
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.656875 1619 kubelet.go:2750] Recording NodeHasSufficientDisk event message for node 192.168.1.105
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.657029 1619 kubelet.go:1134] Attempting to register node 192.168.1.105
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.690531 1619 kubelet.go:1150] Node 192.168.1.105 was previously registered
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.734063 1619 kubelet.go:2750] Recording NodeHasSufficientDisk event message for node 192.168.1.105
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.734231 1619 kubelet.go:2750] Recording NodeNotReady event message for node 192.168.1.105
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.797951 1619 manager.go:1003] Started watching for new ooms in manager
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.800615 1619 oomparser.go:182] oomparser using systemd
Jul 18 23:00:57 k8s-master sh[1610]: I0718 20:00:57.803260 1619 manager.go:256] Starting recovery of all containers
Jul 18 23:00:58 k8s-master sh[1610]: I0718 20:00:58.075289 1619 manager.go:261] Recovery completed
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.391311 1619 kubelet.go:2388] SyncLoop (ADD, "file"): "k8s-master-192.168.1.105_default(80a366492c94fafeacf2fc27c3de60c5)"
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.392022 1619 kubelet.go:2451] SyncLoop (PLEG): "k8s-master-192.168.1.105_default(80a366492c94fafeacf2fc27c3de60c5)", event: &pleg.PodLifecycleEvent{ID:"80a366492c94fafeacf2fc27c3de60c5", Type:"ContainerStarted", Data:"8753578f1670657269c2f74b88311663fa8e1d168f6f5976408b0b64b8ba9258"}
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.406785 1619 kubelet.go:2451] SyncLoop (PLEG): "k8s-master-192.168.1.105_default(80a366492c94fafeacf2fc27c3de60c5)", event: &pleg.PodLifecycleEvent{ID:"80a366492c94fafeacf2fc27c3de60c5", Type:"ContainerDied", Data:"7809586d5b5f887fb14dc465a13ef125efd5a36e59e89b5662585668721c5512"}
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.420888 1619 kubelet.go:2451] SyncLoop (PLEG): "k8s-master-192.168.1.105_default(80a366492c94fafeacf2fc27c3de60c5)", event: &pleg.PodLifecycleEvent{ID:"80a366492c94fafeacf2fc27c3de60c5", Type:"ContainerStarted", Data:"ef1a30217d3ed57a80c898a95bb4247a02f83f7fd192b41a891fbfb5125ee670"}
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.434678 1619 kubelet.go:2391] SyncLoop (UPDATE, "api"): "k8s-master-192.168.1.105_default(083f4926-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.434965 1619 kubelet.go:2451] SyncLoop (PLEG): "k8s-master-192.168.1.105_default(80a366492c94fafeacf2fc27c3de60c5)", event: &pleg.PodLifecycleEvent{ID:"80a366492c94fafeacf2fc27c3de60c5", Type:"ContainerStarted", Data:"c5532d91de156b7f0477a21bcbe4f330bce6c91db776eb59d33e72c0ede1185b"}
Jul 18 23:01:02 k8s-master sh[1610]: E0718 20:01:02.442907 1619 kubelet.go:1781] Failed creating a mirror pod for "k8s-master-192.168.1.105_default(80a366492c94fafeacf2fc27c3de60c5)": pods "k8s-master-192.168.1.105" already exists
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.450550 1619 kubelet.go:2391] SyncLoop (UPDATE, "api"): "k8s-master-192.168.1.105_default(083f4926-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:02 k8s-master sh[1610]: E0718 20:01:02.755279 1619 kubelet.go:1781] Deleting mirror pod "k8s-master-192.168.1.105_default(083f4926-4d22-11e6-b426-b827eb15c6af)" because it is outdated
Jul 18 23:01:02 k8s-master sh[1610]: W0718 20:01:02.804606 1619 manager.go:397] Failed to update status for pod "_()": pods "k8s-master-192.168.1.105" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.827179 1619 kubelet.go:2391] SyncLoop (UPDATE, "api"): "k8s-master-192.168.1.105_default(083f4926-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.835815 1619 kubelet.go:2394] SyncLoop (REMOVE, "api"): "k8s-master-192.168.1.105_default(083f4926-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:02 k8s-master sh[1610]: I0718 20:01:02.904945 1619 kubelet.go:2388] SyncLoop (ADD, "api"): "k8s-master-192.168.1.105_default(5ac54c3d-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:03 k8s-master sh[1610]: I0718 20:01:03.488221 1619 kubelet.go:2391] SyncLoop (UPDATE, "api"): "k8s-master-192.168.1.105_default(5ac54c3d-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:03 k8s-master sh[1610]: E0718 20:01:03.769340 1619 kubelet.go:1781] Deleting mirror pod "k8s-master-192.168.1.105_default(083f4926-4d22-11e6-b426-b827eb15c6af)" because it is outdated
Jul 18 23:01:03 k8s-master sh[1610]: I0718 20:01:03.823582 1619 kubelet.go:2391] SyncLoop (UPDATE, "api"): "k8s-master-192.168.1.105_default(5ac54c3d-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:03 k8s-master sh[1610]: I0718 20:01:03.849117 1619 kubelet.go:2394] SyncLoop (REMOVE, "api"): "k8s-master-192.168.1.105_default(5ac54c3d-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:03 k8s-master sh[1610]: I0718 20:01:03.861931 1619 kubelet.go:2388] SyncLoop (ADD, "api"): "k8s-master-192.168.1.105_default(5b59b908-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:04 k8s-master sh[1610]: E0718 20:01:04.782671 1619 kubelet.go:1781] Deleting mirror pod "k8s-master-192.168.1.105_default(5ac54c3d-4d22-11e6-b426-b827eb15c6af)" because it is outdated
Jul 18 23:01:04 k8s-master sh[1610]: I0718 20:01:04.837558 1619 kubelet.go:2391] SyncLoop (UPDATE, "api"): "k8s-master-192.168.1.105_default(5b59b908-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:04 k8s-master sh[1610]: I0718 20:01:04.837843 1619 kubelet.go:2394] SyncLoop (REMOVE, "api"): "k8s-master-192.168.1.105_default(5b59b908-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:04 k8s-master sh[1610]: I0718 20:01:04.868339 1619 kubelet.go:2388] SyncLoop (ADD, "api"): "k8s-master-192.168.1.105_default(5bf4979b-4d22-11e6-b426-b827eb15c6af)"
Jul 18 23:01:07 k8s-master sh[1610]: I0718 20:01:07.838853 1619 kubelet.go:2750] Recording NodeReady event message for node 192.168.1.105
f145e7a1621956d52e3c with output stdout: 177108 /var/lib/docker/overlay/37c0c138654a079d04f0bd13b4e1ff6cfc8d759ca365f145e7a1621956d52e3c
Jul 18 23:02:00 k8s-master sh[1610]: , stderr: du: cannot access ‘/var/lib/docker/overlay/37c0c138654a079d04f0bd13b4e1ff6cfc8d759ca365f145e7a1621956d52e3c/merged/proc/1704/task/1704/fd/3’: No such file or directory
Jul 18 23:02:00 k8s-master sh[1610]: du: cannot access ‘/var/lib/docker/overlay/37c0c138654a079d04f0bd13b4e1ff6cfc8d759ca365f145e7a1621956d52e3c/merged/proc/1704/task/1704/fdinfo/3’: No such file or directory
Jul 18 23:02:00 k8s-master sh[1610]: du: cannot access ‘/var/lib/docker/overlay/37c0c138654a079d04f0bd13b4e1ff6cfc8d759ca365f145e7a1621956d52e3c/merged/proc/1704/fd/4’: No such file or directory
Jul 18 23:02:00 k8s-master sh[1610]: du: cannot access ‘/var/lib/docker/overlay/37c0c138654a079d04f0bd13b4e1ff6cfc8d759ca365f145e7a1621956d52e3c/merged/proc/1704/fdinfo/4’: No such file or directory
Jul 18 23:02:00 k8s-master sh[1610]: - exit status 1
Jul 18 23:02:08 k8s-master sh[1610]: E0718 20:02:08.587433 1619 kubelet.go:2741] Error updating node status, will retry: nodes "192.168.1.105" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
Jul 18 23:02:08 k8s-master sh[1610]: I0718 20:02:08.703277 1619 kubelet.go:2750] Recording NodeHasSufficientDisk event message for node 192.168.1.105
Jul 18 23:02:08 k8s-master sh[1610]: I0718 20:02:08.703502 1619 kubelet.go:2750] Recording NodeReady event message for node 192.168.1.105
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment