Skip to content

Instantly share code, notes, and snippets.

@iogbole

iogbole/container_logs_portworx.out Secret

Last active February 6, 2021 11:24
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save iogbole/6857fbf846dad6823de2bdb586ed1aa6 to your computer and use it in GitHub Desktop.
Save iogbole/6857fbf846dad6823de2bdb586ed1aa6 to your computer and use it in GitHub Desktop.
Download ZIP
portworx-errors
Raw
container_logs_portworx.out
kubectl logs -c portworx -n kube-system -l name=portworx --tail=99999
time="2021-02-06T11:07:32Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s --oem esse"
time="2021-02-06T11:07:32Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s"
time="2021-02-06T11:07:32Z" level=info msg="OCI-Monitor computed version v2.6.3-ge63b7d82-dirty"
time="2021-02-06T11:07:32Z" level=info msg="REAPER: Starting ..."
time="2021-02-06T11:07:32Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc0004e62e0}"
time="2021-02-06T11:07:32Z" level=info msg="Activating REST server"
time="2021-02-06T11:07:32Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"
time="2021-02-06T11:07:32Z" level=info msg="Locating my container handler"
time="2021-02-06T11:07:32Z" level=info msg="> Attempt to use Docker as container handler failed" error="/var/run/docker.sock not a socket-file"
time="2021-02-06T11:07:32Z" level=info msg="> Attempt to use ContainerD as container handler failed" error="stat /run/containerd/containerd.sock: no such file or directory"
time="2021-02-06T11:07:32Z" level=info msg="> Attempt to use k8s-CRI as container handler failed" error="CRI-Socket path not specified"
time="2021-02-06T11:07:32Z" level=error msg="Could not instantiate container client" error="Could not initialize container handler"
time="2021-02-06T11:07:32Z" level=error msg="Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted."
Usage: /px-oci-mon [options]
options:
--endpoint <ip:port> Start REST service at specific endpoint
--dev Deploy PX-Developer rather than PX-Enterprise build
--sync Will issue sync operation before stopping/restarting the PX-OCI service
--drain-all Will drain ALL PX-dependent pods before upgrade (dfl. only managed nodes get drained)
--log <file> Will use logfile instead of Docker-log
--ignore-preexec-fail Will ignore failed Pre/Post -exec scriplets
--disable-service-log Disables following portworx-service log
--max-log-lines <#> Specify number of log-lines when flushing portworx service logs
--cnthandler <typ> Force a container handler (<typ> is docker or containerd)
--svcandler <typ> Force a service handler (<typ> is dbus, dbusp, nssysd or nsups)
--debug Increase logs-verbosity to debug-level
* Any additional options will be passed on to px-runc
NOTE that any options not explicitly listed above, will be passed directly to px-runc.
For details please see http://docs.portworx.com/runc
time="2021-02-06T11:07:34Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s --oem esse"
time="2021-02-06T11:07:34Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s"
time="2021-02-06T11:07:34Z" level=info msg="OCI-Monitor computed version v2.6.3-ge63b7d82-dirty"
time="2021-02-06T11:07:34Z" level=info msg="REAPER: Starting ..."
time="2021-02-06T11:07:34Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc000153ca0}"
time="2021-02-06T11:07:34Z" level=info msg="Activating REST server"
time="2021-02-06T11:07:34Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"
time="2021-02-06T11:07:34Z" level=info msg="Locating my container handler"
time="2021-02-06T11:07:34Z" level=info msg="> Attempt to use Docker as container handler failed" error="/var/run/docker.sock not a socket-file"
time="2021-02-06T11:07:34Z" level=info msg="> Attempt to use ContainerD as container handler failed" error="stat /run/containerd/containerd.sock: no such file or directory"
time="2021-02-06T11:07:34Z" level=info msg="> Attempt to use k8s-CRI as container handler failed" error="CRI-Socket path not specified"
time="2021-02-06T11:07:34Z" level=error msg="Could not instantiate container client" error="Could not initialize container handler"
time="2021-02-06T11:07:34Z" level=error msg="Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted."
Usage: /px-oci-mon [options]
options:
--endpoint <ip:port> Start REST service at specific endpoint
--dev Deploy PX-Developer rather than PX-Enterprise build
--sync Will issue sync operation before stopping/restarting the PX-OCI service
--drain-all Will drain ALL PX-dependent pods before upgrade (dfl. only managed nodes get drained)
--log <file> Will use logfile instead of Docker-log
--ignore-preexec-fail Will ignore failed Pre/Post -exec scriplets
--disable-service-log Disables following portworx-service log
--max-log-lines <#> Specify number of log-lines when flushing portworx service logs
--cnthandler <typ> Force a container handler (<typ> is docker or containerd)
--svcandler <typ> Force a service handler (<typ> is dbus, dbusp, nssysd or nsups)
--debug Increase logs-verbosity to debug-level
* Any additional options will be passed on to px-runc
NOTE that any options not explicitly listed above, will be passed directly to px-runc.
For details please see http://docs.portworx.com/runc
time="2021-02-06T11:07:28Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s --oem esse"
time="2021-02-06T11:07:28Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s"
time="2021-02-06T11:07:28Z" level=info msg="OCI-Monitor computed version v2.6.3-ge63b7d82-dirty"
time="2021-02-06T11:07:28Z" level=info msg="REAPER: Starting ..."
time="2021-02-06T11:07:28Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc0002574c0}"
time="2021-02-06T11:07:28Z" level=info msg="Activating REST server"
time="2021-02-06T11:07:28Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"
time="2021-02-06T11:07:28Z" level=info msg="Locating my container handler"
time="2021-02-06T11:07:28Z" level=info msg="> Attempt to use Docker as container handler failed" error="/var/run/docker.sock not a socket-file"
time="2021-02-06T11:07:28Z" level=info msg="> Attempt to use ContainerD as container handler failed" error="stat /run/containerd/containerd.sock: no such file or directory"
time="2021-02-06T11:07:28Z" level=info msg="> Attempt to use k8s-CRI as container handler failed" error="CRI-Socket path not specified"
time="2021-02-06T11:07:28Z" level=error msg="Could not instantiate container client" error="Could not initialize container handler"
time="2021-02-06T11:07:28Z" level=error msg="Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted."
Usage: /px-oci-mon [options]
options:
--endpoint <ip:port> Start REST service at specific endpoint
--dev Deploy PX-Developer rather than PX-Enterprise build
--sync Will issue sync operation before stopping/restarting the PX-OCI service
--drain-all Will drain ALL PX-dependent pods before upgrade (dfl. only managed nodes get drained)
--log <file> Will use logfile instead of Docker-log
--ignore-preexec-fail Will ignore failed Pre/Post -exec scriplets
--disable-service-log Disables following portworx-service log
--max-log-lines <#> Specify number of log-lines when flushing portworx service logs
--cnthandler <typ> Force a container handler (<typ> is docker or containerd)
--svcandler <typ> Force a service handler (<typ> is dbus, dbusp, nssysd or nsups)
--debug Increase logs-verbosity to debug-level
* Any additional options will be passed on to px-runc
NOTE that any options not explicitly listed above, will be passed directly to px-runc.
For details please see http://docs.portworx.com/runc
Raw
describe-po.out
Name: px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5
Namespace: kube-system
Priority: 0
Node: ip-10-97-7-111/10.97.7.111
Start Time: Sat, 06 Feb 2021 10:50:48 +0000
Labels: controller-revision-hash=7499b67c55
name=portworx
operator.libopenstorage.org/driver=portworx
operator.libopenstorage.org/name=px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
operator.libopenstorage.org/node-labels:
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"k3s","beta.kubernetes.io/os":"linux","k3s.io/hostname":"ip-10-97-13...
Status: Running
IP: 10.97.7.111
IPs:
IP: 10.97.7.111
Controlled By: StorageCluster/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Containers:
portworx:
Container ID: containerd://31d92f7b81066413facf81522de814ec8b991f1732736d7c47d1bb65233eacd8
Image: portworx/oci-monitor:2.6.3
Image ID: docker.io/portworx/oci-monitor@sha256:1fac588c0702e7076ed897b9758d80df896da1eb1e864674b9bc54cd00ff7e5a
Port: <none>
Host Port: <none>
Args:
-c
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
-x
kubernetes
-b
-A
-f
-j
auto
-secret_type
k8s
--oem
esse
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 06 Feb 2021 11:17:42 +0000
Finished: Sat, 06 Feb 2021 11:17:42 +0000
Ready: False
Restart Count: 10
Liveness: http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
PX_NAMESPACE: kube-system
PX_SECRETS_NAMESPACE: kube-system
PX_TEMPLATE_VERSION: v4
Mounts:
/etc/crictl.yaml from crioconf (rw)
/etc/pwx from etcpwx (rw)
/etc/systemd/system from sysdmount (rw)
/host_proc from procmount (rw)
/opt/pwx from optpwx (rw)
/run/containerd from containerdsock (rw)
/var/cores from diagsdump (rw)
/var/log from journalmount2 (ro)
/var/run/crio from criosock (rw)
/var/run/dbus from dbusmount (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/log from journalmount1 (ro)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
csi-node-driver-registrar:
Container ID: containerd://761e1ff64a5e1053e43032214b8b27e090deb97169b7932a325f3040fd565448
Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
Image ID: sha256:d6f8cc587ff3fcad821ddd1e69db2d6fc5be84f4616a7b8e4f3f54889596137e
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=/var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
State: Running
Started: Sat, 06 Feb 2021 10:51:15 +0000
Ready: True
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from csi-driver-path (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
diagsdump:
Type: HostPath (bare host directory volume)
Path: /var/cores
HostPathType:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
containerdsock:
Type: HostPath (bare host directory volume)
Path: /run/containerd
HostPathType:
criosock:
Type: HostPath (bare host directory volume)
Path: /var/run/crio
HostPathType:
crioconf:
Type: HostPath (bare host directory volume)
Path: /etc/crictl.yaml
HostPathType: FileOrCreate
etcpwx:
Type: HostPath (bare host directory volume)
Path: /etc/pwx
HostPathType:
optpwx:
Type: HostPath (bare host directory volume)
Path: /opt/pwx
HostPathType:
procmount:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sysdmount:
Type: HostPath (bare host directory volume)
Path: /etc/systemd/system
HostPathType:
journalmount1:
Type: HostPath (bare host directory volume)
Path: /var/run/log
HostPathType:
journalmount2:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
dbusmount:
Type: HostPath (bare host directory volume)
Path: /var/run/dbus
HostPathType:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: DirectoryOrCreate
csi-driver-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/pxd.portworx.com
HostPathType: DirectoryOrCreate
portworx-token-4m29r:
Type: Secret (a volume populated by a Secret)
SecretName: portworx-token-4m29r
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 28m kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 22.116425118s
Normal Pulling 28m kubelet Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0"
Normal Pulled 28m kubelet Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" in 3.367794502s
Normal Created 28m kubelet Created container csi-node-driver-registrar
Normal Started 28m kubelet Started container csi-node-driver-registrar
Normal Pulled 28m kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 719.432453ms
Normal Pulled 28m kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 735.658136ms
Normal Pulling 28m (x4 over 29m) kubelet Pulling image "portworx/oci-monitor:2.6.3"
Normal Pulled 28m kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 715.865971ms
Normal Created 28m (x4 over 28m) kubelet Created container portworx
Normal Started 28m (x4 over 28m) kubelet Started container portworx
Warning BackOff 4m1s (x120 over 28m) kubelet Back-off restarting failed container
Raw
describe_pod.out
Name: px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6
Namespace: kube-system
Priority: 0
Node: ip-10-97-3-85/10.97.3.85
Start Time: Sat, 06 Feb 2021 10:50:48 +0000
Labels: controller-revision-hash=7499b67c55
name=portworx
operator.libopenstorage.org/driver=portworx
operator.libopenstorage.org/name=px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
operator.libopenstorage.org/node-labels:
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"k3s","beta.kubernetes.io/os":"linux","k3s.io/hostname":"ip-10-97-13...
Status: Running
IP: 10.97.3.85
IPs:
IP: 10.97.3.85
Controlled By: StorageCluster/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Containers:
portworx:
Container ID: containerd://1f7c3fc97cd84a46ff28eb092bb93f87f92f42d79fa520917810a796e12068ad
Image: portworx/oci-monitor:2.6.3
Image ID: docker.io/portworx/oci-monitor@sha256:1fac588c0702e7076ed897b9758d80df896da1eb1e864674b9bc54cd00ff7e5a
Port: <none>
Host Port: <none>
Args:
-c
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
-x
kubernetes
-b
-A
-f
-j
auto
-secret_type
k8s
--oem
esse
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 06 Feb 2021 10:54:04 +0000
Finished: Sat, 06 Feb 2021 10:54:04 +0000
Ready: False
Restart Count: 5
Liveness: http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
PX_NAMESPACE: kube-system
PX_SECRETS_NAMESPACE: kube-system
PX_TEMPLATE_VERSION: v4
Mounts:
/etc/crictl.yaml from crioconf (rw)
/etc/pwx from etcpwx (rw)
/etc/systemd/system from sysdmount (rw)
/host_proc from procmount (rw)
/opt/pwx from optpwx (rw)
/run/containerd from containerdsock (rw)
/var/cores from diagsdump (rw)
/var/log from journalmount2 (ro)
/var/run/crio from criosock (rw)
/var/run/dbus from dbusmount (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/log from journalmount1 (ro)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
csi-node-driver-registrar:
Container ID: containerd://f22cd6e5960733c8a96af038501bf43f3af7d74c8c58b4373c4815c6227612be
Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
Image ID: sha256:d6f8cc587ff3fcad821ddd1e69db2d6fc5be84f4616a7b8e4f3f54889596137e
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=/var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
State: Running
Started: Sat, 06 Feb 2021 10:51:15 +0000
Ready: True
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from csi-driver-path (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
diagsdump:
Type: HostPath (bare host directory volume)
Path: /var/cores
HostPathType:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
containerdsock:
Type: HostPath (bare host directory volume)
Path: /run/containerd
HostPathType:
criosock:
Type: HostPath (bare host directory volume)
Path: /var/run/crio
HostPathType:
crioconf:
Type: HostPath (bare host directory volume)
Path: /etc/crictl.yaml
HostPathType: FileOrCreate
etcpwx:
Type: HostPath (bare host directory volume)
Path: /etc/pwx
HostPathType:
optpwx:
Type: HostPath (bare host directory volume)
Path: /opt/pwx
HostPathType:
procmount:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sysdmount:
Type: HostPath (bare host directory volume)
Path: /etc/systemd/system
HostPathType:
journalmount1:
Type: HostPath (bare host directory volume)
Path: /var/run/log
HostPathType:
journalmount2:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
dbusmount:
Type: HostPath (bare host directory volume)
Path: /var/run/dbus
HostPathType:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: DirectoryOrCreate
csi-driver-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/pxd.portworx.com
HostPathType: DirectoryOrCreate
portworx-token-4m29r:
Type: Secret (a volume populated by a Secret)
SecretName: portworx-token-4m29r
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 3m21s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 20.902840475s
Normal Pulling 3m19s kubelet Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0"
Normal Pulled 3m15s kubelet Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" in 3.297283109s
Normal Created 3m15s kubelet Created container csi-node-driver-registrar
Normal Started 3m15s kubelet Started container csi-node-driver-registrar
Normal Pulled 3m14s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 723.620167ms
Normal Pulled 2m59s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 726.811327ms
Warning BackOff 2m43s (x5 over 3m13s) kubelet Back-off restarting failed container
Normal Pulling 2m31s (x4 over 3m42s) kubelet Pulling image "portworx/oci-monitor:2.6.3"
Normal Pulled 2m30s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 749.971917ms
Normal Created 2m30s (x4 over 3m19s) kubelet Created container portworx
Normal Started 2m30s (x4 over 3m19s) kubelet Started container portworx
Name: px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-q94sx
Namespace: kube-system
Priority: 0
Node: ip-10-97-13-113/10.97.13.113
Start Time: Sat, 06 Feb 2021 10:50:47 +0000
Labels: controller-revision-hash=7499b67c55
name=portworx
operator.libopenstorage.org/driver=portworx
operator.libopenstorage.org/name=px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
operator.libopenstorage.org/node-labels:
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"k3s","beta.kubernetes.io/os":"linux","k3s.io/hostname":"ip-10-97-13...
Status: Running
IP: 10.97.13.113
IPs:
IP: 10.97.13.113
Controlled By: StorageCluster/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Containers:
portworx:
Container ID: containerd://bd71d4e00082627d465eadf0b83bf69a1eb875ff7e5fcda5993056295681df1e
Image: portworx/oci-monitor:2.6.3
Image ID: docker.io/portworx/oci-monitor@sha256:1fac588c0702e7076ed897b9758d80df896da1eb1e864674b9bc54cd00ff7e5a
Port: <none>
Host Port: <none>
Args:
-c
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
-x
kubernetes
-b
-A
-f
-j
auto
-secret_type
k8s
--oem
esse
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 06 Feb 2021 10:54:18 +0000
Finished: Sat, 06 Feb 2021 10:54:18 +0000
Ready: False
Restart Count: 5
Liveness: http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
PX_NAMESPACE: kube-system
PX_SECRETS_NAMESPACE: kube-system
PX_TEMPLATE_VERSION: v4
Mounts:
/etc/crictl.yaml from crioconf (rw)
/etc/pwx from etcpwx (rw)
/etc/systemd/system from sysdmount (rw)
/host_proc from procmount (rw)
/opt/pwx from optpwx (rw)
/run/containerd from containerdsock (rw)
/var/cores from diagsdump (rw)
/var/log from journalmount2 (ro)
/var/run/crio from criosock (rw)
/var/run/dbus from dbusmount (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/log from journalmount1 (ro)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
csi-node-driver-registrar:
Container ID: containerd://ccf961a58540fce9d789efb734f9fd3d0b59b4ae021f8b804762edf1524e722a
Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
Image ID: sha256:d6f8cc587ff3fcad821ddd1e69db2d6fc5be84f4616a7b8e4f3f54889596137e
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=/var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
State: Running
Started: Sat, 06 Feb 2021 10:51:15 +0000
Ready: True
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from csi-driver-path (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
diagsdump:
Type: HostPath (bare host directory volume)
Path: /var/cores
HostPathType:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
containerdsock:
Type: HostPath (bare host directory volume)
Path: /run/containerd
HostPathType:
criosock:
Type: HostPath (bare host directory volume)
Path: /var/run/crio
HostPathType:
crioconf:
Type: HostPath (bare host directory volume)
Path: /etc/crictl.yaml
HostPathType: FileOrCreate
etcpwx:
Type: HostPath (bare host directory volume)
Path: /etc/pwx
HostPathType:
optpwx:
Type: HostPath (bare host directory volume)
Path: /opt/pwx
HostPathType:
procmount:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sysdmount:
Type: HostPath (bare host directory volume)
Path: /etc/systemd/system
HostPathType:
journalmount1:
Type: HostPath (bare host directory volume)
Path: /var/run/log
HostPathType:
journalmount2:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
dbusmount:
Type: HostPath (bare host directory volume)
Path: /var/run/dbus
HostPathType:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: DirectoryOrCreate
csi-driver-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/pxd.portworx.com
HostPathType: DirectoryOrCreate
portworx-token-4m29r:
Type: Secret (a volume populated by a Secret)
SecretName: portworx-token-4m29r
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 3m20s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 21.171477471s
Normal Pulling 3m19s kubelet Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0"
Normal Pulled 3m16s kubelet Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" in 3.045674984s
Normal Created 3m16s kubelet Created container csi-node-driver-registrar
Normal Started 3m15s kubelet Started container csi-node-driver-registrar
Normal Pulled 3m14s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 671.404706ms
Normal Pulled 3m kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 672.17062ms
Normal Pulling 2m31s (x4 over 3m42s) kubelet Pulling image "portworx/oci-monitor:2.6.3"
Normal Pulled 2m30s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 789.796744ms
Normal Created 2m30s (x4 over 3m19s) kubelet Created container portworx
Normal Started 2m30s (x4 over 3m19s) kubelet Started container portworx
Warning BackOff 2m29s (x5 over 3m13s) kubelet Back-off restarting failed container
Name: px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5
Namespace: kube-system
Priority: 0
Node: ip-10-97-7-111/10.97.7.111
Start Time: Sat, 06 Feb 2021 10:50:48 +0000
Labels: controller-revision-hash=7499b67c55
name=portworx
operator.libopenstorage.org/driver=portworx
operator.libopenstorage.org/name=px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
operator.libopenstorage.org/node-labels:
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"k3s","beta.kubernetes.io/os":"linux","k3s.io/hostname":"ip-10-97-13...
Status: Running
IP: 10.97.7.111
IPs:
IP: 10.97.7.111
Controlled By: StorageCluster/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Containers:
portworx:
Container ID: containerd://996684222f9b3dfaec139971b45f5bf3639e38a632a4b9739079bef69ecc11f7
Image: portworx/oci-monitor:2.6.3
Image ID: docker.io/portworx/oci-monitor@sha256:1fac588c0702e7076ed897b9758d80df896da1eb1e864674b9bc54cd00ff7e5a
Port: <none>
Host Port: <none>
Args:
-c
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
-x
kubernetes
-b
-A
-f
-j
auto
-secret_type
k8s
--oem
esse
State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 06 Feb 2021 10:54:18 +0000
Finished: Sat, 06 Feb 2021 10:54:18 +0000
Ready: False
Restart Count: 5
Liveness: http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
PX_NAMESPACE: kube-system
PX_SECRETS_NAMESPACE: kube-system
PX_TEMPLATE_VERSION: v4
Mounts:
/etc/crictl.yaml from crioconf (rw)
/etc/pwx from etcpwx (rw)
/etc/systemd/system from sysdmount (rw)
/host_proc from procmount (rw)
/opt/pwx from optpwx (rw)
/run/containerd from containerdsock (rw)
/var/cores from diagsdump (rw)
/var/log from journalmount2 (ro)
/var/run/crio from criosock (rw)
/var/run/dbus from dbusmount (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/log from journalmount1 (ro)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
csi-node-driver-registrar:
Container ID: containerd://761e1ff64a5e1053e43032214b8b27e090deb97169b7932a325f3040fd565448
Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
Image ID: sha256:d6f8cc587ff3fcad821ddd1e69db2d6fc5be84f4616a7b8e4f3f54889596137e
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=/var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
State: Running
Started: Sat, 06 Feb 2021 10:51:15 +0000
Ready: True
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from csi-driver-path (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from portworx-token-4m29r (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
diagsdump:
Type: HostPath (bare host directory volume)
Path: /var/cores
HostPathType:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
containerdsock:
Type: HostPath (bare host directory volume)
Path: /run/containerd
HostPathType:
criosock:
Type: HostPath (bare host directory volume)
Path: /var/run/crio
HostPathType:
crioconf:
Type: HostPath (bare host directory volume)
Path: /etc/crictl.yaml
HostPathType: FileOrCreate
etcpwx:
Type: HostPath (bare host directory volume)
Path: /etc/pwx
HostPathType:
optpwx:
Type: HostPath (bare host directory volume)
Path: /opt/pwx
HostPathType:
procmount:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sysdmount:
Type: HostPath (bare host directory volume)
Path: /etc/systemd/system
HostPathType:
journalmount1:
Type: HostPath (bare host directory volume)
Path: /var/run/log
HostPathType:
journalmount2:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
dbusmount:
Type: HostPath (bare host directory volume)
Path: /var/run/dbus
HostPathType:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: DirectoryOrCreate
csi-driver-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/pxd.portworx.com
HostPathType: DirectoryOrCreate
portworx-token-4m29r:
Type: Secret (a volume populated by a Secret)
SecretName: portworx-token-4m29r
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 3m19s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 22.116425118s
Normal Pulling 3m19s kubelet Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0"
Normal Pulled 3m16s kubelet Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" in 3.367794502s
Normal Created 3m15s kubelet Created container csi-node-driver-registrar
Normal Started 3m15s kubelet Started container csi-node-driver-registrar
Normal Pulled 3m14s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 719.432453ms
Normal Pulled 3m kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 735.658136ms
Warning BackOff 2m44s (x5 over 3m13s) kubelet Back-off restarting failed container
Normal Pulling 2m29s (x4 over 3m42s) kubelet Pulling image "portworx/oci-monitor:2.6.3"
Normal Pulled 2m29s kubelet Successfully pulled image "portworx/oci-monitor:2.6.3" in 715.865971ms
Normal Created 2m29s (x4 over 3m19s) kubelet Created container portworx
Normal Started 2m28s (x4 over 3m19s) kubelet Started container portworx
Raw
get-events.out
24m Normal Pulled pod/px-lighthouse-7dc48b77c8-4slwc Successfully pulled image "portworx/lh-config-sync:2.0.7" in 684.067966ms
24m Normal Started pod/px-lighthouse-7dc48b77c8-4slwc Started container config-init
21m Normal Pulling pod/px-lighthouse-7dc48b77c8-4slwc Pulling image "portworx/lh-config-sync:2.0.7"
21m Normal Pulled pod/px-lighthouse-7dc48b77c8-4slwc Successfully pulled image "portworx/lh-config-sync:2.0.7" in 728.851936ms
21m Normal Created pod/px-lighthouse-7dc48b77c8-4slwc Created container config-init
19m Warning PortworxMonitorInstallFailed storagenode/ip-10-97-3-85 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
19m Warning PortworxMonitorInstallFailed storagenode/ip-10-97-7-111 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
19m Warning PortworxMonitorInstallFailed storagenode/ip-10-97-13-113 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
14m Warning PortworxMonitorInstallFailed storagenode/ip-10-97-3-85 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
14m Warning PortworxMonitorInstallFailed storagenode/ip-10-97-13-113 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
9m26s Warning PortworxMonitorInstallFailed storagenode/ip-10-97-3-85 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
9m24s Warning PortworxMonitorInstallFailed storagenode/ip-10-97-7-111 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
9m19s Warning PortworxMonitorInstallFailed storagenode/ip-10-97-13-113 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
4m18s Warning PortworxMonitorInstallFailed storagenode/ip-10-97-7-111 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
4m17s Warning PortworxMonitorInstallFailed storagenode/ip-10-97-13-113 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
4m16s Warning PortworxMonitorInstallFailed storagenode/ip-10-97-3-85 Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted.
73s Warning Unhealthy pod/portworx-api-g6hxf Readiness probe failed: Get "http://127.0.0.1:9001/status": dial tcp 127.0.0.1:9001: connect: connection refused
71s Warning BackOff pod/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5 Back-off restarting failed container
71s Warning Unhealthy pod/portworx-api-8zvxk Readiness probe failed: Get "http://127.0.0.1:9001/status": dial tcp 127.0.0.1:9001: connect: connection refused
70s Warning Unhealthy pod/portworx-api-jc7xl Readiness probe failed: Get "http://127.0.0.1:9001/status": dial tcp 127.0.0.1:9001: connect: connection refused
67s Warning BackOff pod/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-q94sx Back-off restarting failed container
65s Warning BackOff pod/px-lighthouse-7dc48b77c8-4slwc Back-off restarting failed container
65s Warning BackOff pod/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6 Back-off restarting failed container
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment