Skip to content

Instantly share code, notes, and snippets.

@nkumar15
Created June 24, 2018 14:09
Show Gist options
  • Save nkumar15/3f20c944a502362959f8352556ffabfc to your computer and use it in GitHub Desktop.
Save nkumar15/3f20c944a502362959f8352556ffabfc to your computer and use it in GitHub Desktop.
Name: gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-2
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=default-poo/l
failure-domain.beta.kubernetes.io/region=us-central1
failure-domain.beta.kubernetes.io/zone=us-central1-a
kubernetes.io/hostname=gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Sun, 24 Jun 2018 21:30:27 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
KernelDeadlock False Sun, 24 Jun 2018 22:00:24 +0800 Sun, 24 Jun 2018 21:29:27 +0800 KernelHasNoDeadlock kernel has no deadlock
NetworkUnavailable False Sun, 24 Jun 2018 21:30:41 +0800 Sun, 24 Jun 2018 21:30:41 +0800 RouteCreated RouteController created a route
OutOfDisk False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:47 +0800 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.128.0.2
ExternalIP: intentionaly deleted during copying of ouptut by me
Hostname: gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l
Capacity:
cpu: 2
ephemeral-storage: 98868448Ki
hugepages-2Mi: 0
memory: 7658196Ki
pods: 110
Allocatable:
cpu: 1930m
ephemeral-storage: 47093746742
hugepages-2Mi: 0
memory: 5778132Ki
pods: 110
System Info:
Machine ID: 10fddac93e8dd41c99fdae7cd5ffbb24
System UUID: 10FDDAC9-3E8D-D41C-99FD-AE7CD5FFBB24
Boot ID: 2a6d643b-b0a2-4b53-9909-f17757c958d2
Kernel Version: 4.14.22+
OS Image: Container-Optimized OS from Google
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://17.3.2
Kubelet Version: v1.10.4-gke.2
Kube-Proxy Version: v1.10.4-gke.2
PodCIDR: 10.60.0.0/24
ExternalID: 5502524089681715826
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system event-exporter-v0.2.1-5f5b89fcc8-gtrv7 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system fluentd-gcp-scaler-7c5db745fc-dwxtw 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system fluentd-gcp-v3.0.0-z88r6 100m (5%) 0 (0%) 200Mi (3%) 300Mi (5%)
kube-system kube-dns-788979dc8f-rhgld 260m (13%) 0 (0%) 110Mi (1%) 170Mi (3%)
kube-system kube-proxy-gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l 100m (5%) 0 (0%) 0 (0%) 0 (0%)
kube-system metrics-server-v0.2.1-7486f5bd67-wznx7 53m (2%) 148m (7%) 154Mi (2%) 404Mi (7%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
513m (26%) 148m (7%) 464Mi (8%) 874Mi (15%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Starting kubelet.
Normal NodeHasSufficientDisk 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Updated Node Allocatable limit across pods
Normal Starting 30m kube-proxy, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Starting kube-proxy.
Normal NodeReady 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-dv3l status is now: NodeReady
Name: gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-2
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=default-pool
failure-domain.beta.kubernetes.io/region=us-central1
failure-domain.beta.kubernetes.io/zone=us-central1-a
kubernetes.io/hostname=gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Sun, 24 Jun 2018 21:30:27 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
KernelDeadlock False Sun, 24 Jun 2018 22:00:20 +0800 Sun, 24 Jun 2018 21:29:26 +0800 KernelHasNoDeadlock kernel has no deadlock
NetworkUnavailable False Sun, 24 Jun 2018 21:30:39 +0800 Sun, 24 Jun 2018 21:30:39 +0800 RouteCreated RouteController created a route
OutOfDisk False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:27 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:30:47 +0800 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.128.0.3
ExternalIP: 35.224.252.102
Hostname: gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9
Capacity:
cpu: 2
ephemeral-storage: 98868448Ki
hugepages-2Mi: 0
memory: 7658188Ki
pods: 110
Allocatable:
cpu: 1930m
ephemeral-storage: 47093746742
hugepages-2Mi: 0
memory: 5778124Ki
pods: 110
System Info:
Machine ID: f738e9a75dae0a2c20388e74938b6230
System UUID: F738E9A7-5DAE-0A2C-2038-8E74938B6230
Boot ID: d4fd32e8-d5fc-4bf6-97fc-84dee23995b1
Kernel Version: 4.14.22+
OS Image: Container-Optimized OS from Google
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://17.3.2
Kubelet Version: v1.10.4-gke.2
Kube-Proxy Version: v1.10.4-gke.2
PodCIDR: 10.60.1.0/24
ExternalID: 1637632046786552434
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system fluentd-gcp-v3.0.0-hwpzl 100m (5%) 0 (0%) 200Mi (3%) 300Mi (5%)
kube-system heapster-v1.5.3-698d4dc8d5-p5bgp 138m (7%) 138m (7%) 301656Ki (5%) 301656Ki (5%)
kube-system kube-dns-788979dc8f-8pglq 260m (13%) 0 (0%) 110Mi (1%) 170Mi (3%)
kube-system kube-dns-autoscaler-79b4b844b9-bfzc5 20m (1%) 0 (0%) 10Mi (0%) 0 (0%)
kube-system kube-proxy-gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 100m (5%) 0 (0%) 0 (0%) 0 (0%)
kube-system l7-default-backend-5d5b9874d5-rmc2b 10m (0%) 10m (0%) 20Mi (0%) 20Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
628m (32%) 148m (7%) 649816Ki (11%) 803416Ki (13%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Starting kubelet.
Normal NodeHasSufficientDisk 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Updated Node Allocatable limit across pods
Normal Starting 30m kube-proxy, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Starting kube-proxy.
Normal NodeReady 30m kubelet, gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 Node gke-k8s-gpu-cluster-default-pool-8c2e8cc0-f4w9 status is now: NodeReady
Name: gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-2
beta.kubernetes.io/os=linux
cloud.google.com/gke-accelerator=nvidia-tesla-k80
cloud.google.com/gke-nodepool=pool-1
failure-domain.beta.kubernetes.io/region=us-central1
failure-domain.beta.kubernetes.io/zone=us-central1-a
kubernetes.io/hostname=gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: nvidia.com/gpu=present:NoSchedule
CreationTimestamp: Sun, 24 Jun 2018 21:33:28 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
KernelDeadlock False Sun, 24 Jun 2018 22:00:21 +0800 Sun, 24 Jun 2018 21:33:27 +0800 KernelHasNoDeadlock kernel has no deadlock
NetworkUnavailable False Sun, 24 Jun 2018 21:33:39 +0800 Sun, 24 Jun 2018 21:33:39 +0800 RouteCreated RouteController created a route
OutOfDisk False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:33:28 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:33:28 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:33:28 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:33:28 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 24 Jun 2018 22:01:09 +0800 Sun, 24 Jun 2018 21:33:48 +0800 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.128.0.4
ExternalIP: 104.198.173.217
Hostname: gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs
Capacity:
cpu: 2
ephemeral-storage: 98868448Ki
hugepages-2Mi: 0
memory: 7658196Ki
nvidia.com/gpu: 2
pods: 110
Allocatable:
cpu: 1930m
ephemeral-storage: 47093746742
hugepages-2Mi: 0
memory: 5778132Ki
nvidia.com/gpu: 2
pods: 110
System Info:
Machine ID: e9fae90311ad51b9f349659521608845
System UUID: E9FAE903-11AD-51B9-F349-659521608845
Boot ID: cc6b5cb9-0599-47de-8f23-5d0182d3a323
Kernel Version: 4.14.22+
OS Image: Container-Optimized OS from Google
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://17.3.2
Kubelet Version: v1.10.4-gke.2
Kube-Proxy Version: v1.10.4-gke.2
PodCIDR: 10.60.2.0/24
ExternalID: 2161016959045608333
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
default mnist-5dbb7ff74d-d2gw2 100m (5%) 0 (0%) 0 (0%) 0 (0%)
default mnist-5dbb7ff74d-fl5qm 100m (5%) 0 (0%) 0 (0%) 0 (0%)
kube-system fluentd-gcp-v3.0.0-xjr9n 100m (5%) 0 (0%) 200Mi (3%) 300Mi (5%)
kube-system kube-proxy-gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs 100m (5%) 0 (0%) 0 (0%) 0 (0%)
kube-system nvidia-driver-installer-mbpz6 150m (7%) 0 (0%) 0 (0%) 0 (0%)
kube-system nvidia-gpu-device-plugin-zwph7 50m (2%) 50m (2%) 10Mi (0%) 10Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
600m (31%) 50m (2%) 210Mi (3%) 310Mi (5%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 27m kubelet, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Starting kubelet.
Normal NodeHasSufficientDisk 27m kubelet, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Node gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 27m kubelet, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Node gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 27m kubelet, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Node gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 27m kubelet, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Node gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 27m kubelet, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Updated Node Allocatable limit across pods
Normal Starting 27m kube-proxy, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Starting kube-proxy.
Normal NodeReady 27m kubelet, gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs Node gke-k8s-gpu-cluster-pool-1-794b5c7f-40fs status is now: NodeReady
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment