Skip to content

Instantly share code, notes, and snippets.

@phyxavier
Created December 13, 2019 21:43
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save phyxavier/37000dfaf32b58326fad314f2ae4d64c to your computer and use it in GitHub Desktop.
Save phyxavier/37000dfaf32b58326fad314f2ae4d64c to your computer and use it in GitHub Desktop.
time="2019-12-13T21:39:39Z" level=info msg="Input arguments: /px-oci-mon -s type=pd-ssd,size=500 -b -c portworx-cluster -secret_type k8s -x kubernetes -metadata type=pd-ssd,size=64 -max_storage_nodes_per_zone 3"
time="2019-12-13T21:39:39Z" level=info msg="Updated arguments: /px-oci-mon -s type=pd-ssd,size=500 -b -c portworx-cluster -secret_type k8s -x kubernetes -metadata type=pd-ssd,size=64 -max_storage_nodes_per_zone 3"
time="2019-12-13T21:39:39Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc420183300}"
time="2019-12-13T21:39:39Z" level=info msg="Activating REST server"
time="2019-12-13T21:39:39Z" level=info msg="Negotiated Docker API version: 1.32"
time="2019-12-13T21:39:39Z" level=info msg="Using Docker as container handler"
time="2019-12-13T21:39:39Z" level=info msg="Detected NetworkMode container:d1a67f3e0746c660db2aad94c1d8b8ff38ce272f7b41fa636f6d1bfc671c0225 -> polling d1a67f3e0746 for network settings"
time="2019-12-13T21:39:39Z" level=info msg="Detected HostNetwork setting - will track portworx status via REST"
time="2019-12-13T21:39:39Z" level=info msg="Requested PX-OCI from portworx/px-enterprise:2.0.3.3 via env. variable"
time="2019-12-13T21:39:39Z" level=info msg="Preparing to download Portworx image..."
time="2019-12-13T21:39:39Z" level=info msg="REST: Changing install-state: ST_UNKNOWN -> ST_INSTALL"
time="2019-12-13T21:39:39Z" level=info msg="Attempting to retrieve latest portworx/px-enterprise:2.0.3.3 image (pullPolicy Always)"
time="2019-12-13T21:39:39Z" level=info msg="Using anonymous Docker credentials"
time="2019-12-13T21:39:39Z" level=info msg="Could not determine remote image digest for portworx/px-enterprise:2.0.3.3 (will use local image instead)" error="Error response from daemon: manifest unknown: Failed to fetch \"2.0.3.3\" from request \"/v2/portworx/px-enterprise/manifests/2.0.3.3\"."
{"status":"Pulling from portworx/px-enterprise","id":"2.0.3.3"}
{"status":"Digest: sha256:10b00962744d0801b02d3ed0444636df62f2e6524cb7daee8d3bbc588496bd2a"}
{"status":"Status: Image is up to date for portworx/px-enterprise:2.0.3.3"}
time="2019-12-13T21:39:40Z" level=info msg="Found Portworx image portworx/px-enterprise:2.0.3.3 with digest sha256:10b00962744d0801b02d3ed0444636df62f2e6524cb7daee8d3bbc588496bd2a"
time="2019-12-13T21:39:40Z" level=info msg="Installed image digest sha256:10b00962744d0801b02d3ed0444636df62f2e6524cb7daee8d3bbc588496bd2a same as remote/pulled image's (no need to reinstall)"
time="2019-12-13T21:39:40Z" level=info msg="Installing Portworx OCI service to /opt/pwx..."
time="2019-12-13T21:39:40Z" level=info msg="Prepending computed kubelet directory mount /var/lib/kubelet:/var/lib/kubelet:shared"
time="2019-12-13T21:39:40Z" level=info msg="Adding computed k8s-secrets-vol as volatile mount regex"
time="2019-12-13T21:39:40Z" level=info msg="Modified mount /run/containerd:/run/containerd -> /run/containerd:/run/containerd:bind"
time="2019-12-13T21:39:40Z" level=info msg="RSYNC: Found volatile mount '/var/lib/kubelet/pods/8b98bf03-1d79-11ea-ab24-42010a800270/volumes/kubernetes.io~secret/portworx-enterprise-1-portworxserviceaccount-9a0e-token-nwrbr:/var/run/secrets/kubernetes.io/serviceaccount:ro'"
time="2019-12-13T21:39:40Z" level=info msg="> run-local: /usr/bin/rsync -aL --delete /var/run/secrets/kubernetes.io/serviceaccount/ /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount/"
time="2019-12-13T21:39:40Z" level=info msg="> Changed mount /var/lib/kubelet/pods/8b98bf03-1d79-11ea-ab24-42010a800270/volumes/kubernetes.io~secret/portworx-enterprise-1-portworxserviceaccount-9a0e-token-nwrbr:/var/run/secrets/kubernetes.io/serviceaccount:ro to /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount:/var/run/secrets/kubernetes.io/serviceaccount"
time="2019-12-13T21:39:40Z" level=info msg="RSYNC: Found volatile mount '/var/lib/kubelet/pods/8b98bf03-1d79-11ea-ab24-42010a800270/etc-hosts:/etc/hosts'"
time="2019-12-13T21:39:40Z" level=info msg="> run-local: /usr/bin/rsync -aL --delete /etc/hosts /opt/pwx/oci/mounts/etc/hosts"
time="2019-12-13T21:39:40Z" level=info msg="> Changed mount /var/lib/kubelet/pods/8b98bf03-1d79-11ea-ab24-42010a800270/etc-hosts:/etc/hosts to /opt/pwx/oci/mounts/etc/hosts:/etc/hosts"
time="2019-12-13T21:39:40Z" level=info msg="RSYNC: Found volatile mount '/var/lib/kubelet/pods/8b98bf03-1d79-11ea-ab24-42010a800270/containers/portworx/6e9ad330:/tmp/px-termination-log'"
time="2019-12-13T21:39:40Z" level=info msg="> run-local: /usr/bin/rsync -aL --delete /tmp/px-termination-log /opt/pwx/oci/mounts/tmp/px-termination-log"
time="2019-12-13T21:39:40Z" level=info msg="> Changed mount /var/lib/kubelet/pods/8b98bf03-1d79-11ea-ab24-42010a800270/containers/portworx/6e9ad330:/tmp/px-termination-log to /opt/pwx/oci/mounts/tmp/px-termination-log:/tmp/px-termination-log"
time="2019-12-13T21:39:40Z" level=info msg="RSYNC: Found volatile mount '/var/lib/docker/containers/d1a67f3e0746c660db2aad94c1d8b8ff38ce272f7b41fa636f6d1bfc671c0225/resolv.conf:/etc/resolv.conf:ro'"
time="2019-12-13T21:39:40Z" level=info msg="> run-local: /usr/bin/rsync -aL --delete /etc/resolv.conf /opt/pwx/oci/mounts/etc/resolv.conf"
time="2019-12-13T21:39:40Z" level=info msg="> Changed mount /var/lib/docker/containers/d1a67f3e0746c660db2aad94c1d8b8ff38ce272f7b41fa636f6d1bfc671c0225/resolv.conf:/etc/resolv.conf:ro to /opt/pwx/oci/mounts/etc/resolv.conf:/etc/resolv.conf"
time="2019-12-13T21:39:40Z" level=info msg="Ignoring non-default `--cpu-shares=102`"
time="2019-12-13T21:39:40Z" level=info msg="> run-host: /opt/pwx/bin/px-runc install -oci /opt/pwx/oci -s type=pd-ssd,size=500 -b -c portworx-cluster -secret_type k8s -x kubernetes -metadata type=pd-ssd,size=64 -max_storage_nodes_per_zone 3 -v /var/lib/kubelet:/var/lib/kubelet:shared -v /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount:/var/run/secrets/kubernetes.io/serviceaccount -v /run/containerd:/run/containerd:bind -v /var/run/dbus:/var/run/dbus -v /proc:/host_proc -v /var/cores:/var/cores -v /opt/pwx/oci/mounts/etc/hosts:/etc/hosts -v /opt/pwx/oci/mounts/tmp/px-termination-log:/tmp/px-termination-log -v /opt/pwx/oci/mounts/etc/resolv.conf:/etc/resolv.conf -e PX_TEMPLATE_VERSION=v4 -e PX_IMAGE=portworx/px-enterprise:2.0.3.3 -e REPORTING_SECRET=portworx-enterprise-1-reporting-secret -e REPORTING_SECRET_NAMESPACE=default -e PX_SECRETS_NAMESPACE=default -e STORK_SERVICE_PORT_8099_TCP_PORT=8099 -e PX_LIGHTHOUSE_SERVICE_PORT_HTTPS=443 -e PX_LIGHTHOUSE_PORT=tcp://10.4.29.164:443 -e PX_LIGHTHOUSE_PORT_80_TCP_PROTO=tcp -e PX_LIGHTHOUSE_PORT_80_TCP_ADDR=10.4.29.164 -e PORTWORX_SERVICE_PORT_9021_TCP_PORT=9021 -e STORK_SERVICE_PORT_8099_TCP_PROTO=tcp -e PORTWORX_SERVICE_SERVICE_PORT_PX_API=9001 -e PORTWORX_SERVICE_PORT_9001_TCP_ADDR=10.4.25.189 -e PORTWORX_SERVICE_PORT_9019_TCP=tcp://10.4.25.189:9019 -e PORTWORX_SERVICE_PORT_9019_TCP_ADDR=10.4.25.189 -e PORTWORX_SERVICE_PORT_9021_TCP=tcp://10.4.25.189:9021 -e KUBERNETES_PORT=tcp://10.4.16.1:443 -e KUBERNETES_PORT_443_TCP_PROTO=tcp -e STORK_SERVICE_SERVICE_PORT=8099 -e STORK_SERVICE_PORT=tcp://10.4.17.81:8099 -e STORK_SERVICE_PORT_8099_TCP=tcp://10.4.17.81:8099 -e PX_LIGHTHOUSE_PORT_443_TCP_PROTO=tcp -e PORTWORX_SERVICE_SERVICE_PORT_PX_KVDB=9019 -e PORTWORX_SERVICE_PORT=tcp://10.4.25.189:9001 -e PORTWORX_SERVICE_PORT_9021_TCP_PROTO=tcp -e KUBERNETES_PORT_443_TCP_PORT=443 -e PX_LIGHTHOUSE_SERVICE_HOST=10.4.29.164 -e PORTWORX_SERVICE_PORT_9001_TCP_PROTO=tcp -e PORTWORX_SERVICE_SERVICE_PORT_PX_SDK=9021 -e PORTWORX_SERVICE_PORT_9021_TCP_ADDR=10.4.25.189 -e KUBERNETES_SERVICE_HOST=10.4.16.1 -e KUBERNETES_SERVICE_PORT=443 -e KUBERNETES_SERVICE_PORT_HTTPS=443 -e KUBERNETES_PORT_443_TCP=tcp://10.4.16.1:443 -e KUBERNETES_PORT_443_TCP_ADDR=10.4.16.1 -e PX_LIGHTHOUSE_SERVICE_PORT=443 -e PX_LIGHTHOUSE_SERVICE_PORT_HTTP=80 -e PX_LIGHTHOUSE_PORT_443_TCP_PORT=443 -e PX_LIGHTHOUSE_PORT_80_TCP_PORT=80 -e PX_LIGHTHOUSE_PORT_443_TCP=tcp://10.4.29.164:443 -e PX_LIGHTHOUSE_PORT_80_TCP=tcp://10.4.29.164:80 -e PORTWORX_SERVICE_SERVICE_HOST=10.4.25.189 -e PORTWORX_SERVICE_SERVICE_PORT=9001 -e PORTWORX_SERVICE_PORT_9001_TCP_PORT=9001 -e STORK_SERVICE_SERVICE_HOST=10.4.17.81 -e STORK_SERVICE_PORT_8099_TCP_ADDR=10.4.17.81 -e PX_LIGHTHOUSE_PORT_443_TCP_ADDR=10.4.29.164 -e PORTWORX_SERVICE_PORT_9001_TCP=tcp://10.4.25.189:9001 -e PORTWORX_SERVICE_PORT_9019_TCP_PROTO=tcp -e PORTWORX_SERVICE_PORT_9019_TCP_PORT=9019 -e container=oci -e CONTAINER_RUNTIME=docker -e PX_IMAGE_DIGEST=sha256:10b00962744d0801b02d3ed0444636df62f2e6524cb7daee8d3bbc588496bd2a"
time="2019-12-13T21:39:40Z" level=info msg="Rootfs found at /opt/pwx/oci/rootfs"
time="2019-12-13T21:39:40Z" level=info msg="PX binaries found at /opt/pwx/bin/px-runc"
time="2019-12-13T21:39:40Z" level=info msg="Initializing as version 2.0.3.3-e3059a9 (OCI)"
time="2019-12-13T21:39:40Z" level=info msg="Fixing docker.sock mount:"
time="2019-12-13T21:39:40Z" level=info msg="> Removing mount for /var/run/docker.sock:/var/run/docker.sock:[rbind rprivate]"
time="2019-12-13T21:39:40Z" level=info msg="> Adding mount for /run:/var/host_run:[bind rprivate]"
time="2019-12-13T21:39:40Z" level=info msg="> Soft-link /opt/pwx/oci/rootfs/run/docker.sock -> /var/host_run/docker.sock already exists"
time="2019-12-13T21:39:40Z" level=info msg="Checking mountpoints for following shared directories: [/var/lib/kubelet /var/lib/osd]"
time="2019-12-13T21:39:40Z" level=info msg="Found following mountpoints for shared dirs: map[/var/lib/osd:{isMP=f,Opts=shared:1,Parent=/} /var/lib/kubelet:{isMP=T,Opts=shared:1,Parent=/} /:{isMP=T,Opts=shared:1}]"
time="2019-12-13T21:39:40Z" level=info msg="SPEC UPDATED [9997e8f2deb828ae3e8c4f3c187f755a /opt/pwx/oci/config.json]"
time="2019-12-13T21:39:40Z" level=info msg="PX-RunC arguments: -b -c portworx-cluster -max_storage_nodes_per_zone 3 -metadata type=pd-ssd,size=64 -s type=pd-ssd,size=500 -secret_type k8s -x kubernetes"
time="2019-12-13T21:39:40Z" level=info msg="PX-RunC mounts: /dev:/dev /opt/pwx/oci/mounts/etc/hosts:/etc/hosts /etc/pwx:/etc/pwx /opt/pwx/oci/mounts/etc/resolv.conf:/etc/resolv.conf /opt/pwx/bin:/export_bin /proc:/host_proc /lib/modules:/lib/modules proc:/proc:nosuid,noexec,nodev /run/containerd:/run/containerd:bind /run/docker:/run/docker /run/lvm:/run/lvm sysfs:/sys:nosuid,noexec,nodev /sys/firmware/efi/efivars:/sys/firmware/efi/efivars:ro cgroup:/sys/fs/cgroup:nosuid,noexec,nodev /opt/pwx/oci/mounts/tmp/px-termination-log:/tmp/px-termination-log /usr/src:/usr/src /var/cores:/var/cores /run:/var/host_run:bind /var/lib/kubelet:/var/lib/kubelet:shared /var/lib/osd:/var/lib/osd:shared /var/log/journal:/var/log/journal:ro /var/run/dbus:/var/run/dbus /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount:/var/run/secrets/kubernetes.io/serviceaccount"
time="2019-12-13T21:39:40Z" level=info msg="PX-RunC env: BTRFS_SOURCE=/home/px_btrfs CONTAINER_RUNTIME=docker GOMAXPROCS=64 GOTRACEBACK=crash KUBERNETES_PORT=tcp://10.4.16.1:443 KUBERNETES_PORT_443_TCP=tcp://10.4.16.1:443 KUBERNETES_PORT_443_TCP_ADDR=10.4.16.1 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_HOST=10.4.16.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 PATH=/export_bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PORTWORX_SERVICE_PORT=tcp://10.4.25.189:9001 PORTWORX_SERVICE_PORT_9001_TCP=tcp://10.4.25.189:9001 PORTWORX_SERVICE_PORT_9001_TCP_ADDR=10.4.25.189 PORTWORX_SERVICE_PORT_9001_TCP_PORT=9001 PORTWORX_SERVICE_PORT_9001_TCP_PROTO=tcp PORTWORX_SERVICE_PORT_9019_TCP=tcp://10.4.25.189:9019 PORTWORX_SERVICE_PORT_9019_TCP_ADDR=10.4.25.189 PORTWORX_SERVICE_PORT_9019_TCP_PORT=9019 PORTWORX_SERVICE_PORT_9019_TCP_PROTO=tcp PORTWORX_SERVICE_PORT_9021_TCP=tcp://10.4.25.189:9021 PORTWORX_SERVICE_PORT_9021_TCP_ADDR=10.4.25.189 PORTWORX_SERVICE_PORT_9021_TCP_PORT=9021 PORTWORX_SERVICE_PORT_9021_TCP_PROTO=tcp PORTWORX_SERVICE_SERVICE_HOST=10.4.25.189 PORTWORX_SERVICE_SERVICE_PORT=9001 PORTWORX_SERVICE_SERVICE_PORT_PX_API=9001 PORTWORX_SERVICE_SERVICE_PORT_PX_KVDB=9019 PORTWORX_SERVICE_SERVICE_PORT_PX_SDK=9021 PXMOD_SOURCE=/home/px-fuse PXMOD_VERSION=5 PX_IMAGE=portworx/px-enterprise:2.0.3.3 PX_IMAGE_DIGEST=sha256:10b00962744d0801b02d3ed0444636df62f2e6524cb7daee8d3bbc588496bd2a PX_LIGHTHOUSE_PORT=tcp://10.4.29.164:443 PX_LIGHTHOUSE_PORT_443_TCP=tcp://10.4.29.164:443 PX_LIGHTHOUSE_PORT_443_TCP_ADDR=10.4.29.164 PX_LIGHTHOUSE_PORT_443_TCP_PORT=443 PX_LIGHTHOUSE_PORT_443_TCP_PROTO=tcp PX_LIGHTHOUSE_PORT_80_TCP=tcp://10.4.29.164:80 PX_LIGHTHOUSE_PORT_80_TCP_ADDR=10.4.29.164 PX_LIGHTHOUSE_PORT_80_TCP_PORT=80 PX_LIGHTHOUSE_PORT_80_TCP_PROTO=tcp PX_LIGHTHOUSE_SERVICE_HOST=10.4.29.164 PX_LIGHTHOUSE_SERVICE_PORT=443 PX_LIGHTHOUSE_SERVICE_PORT_HTTP=80 PX_LIGHTHOUSE_SERVICE_PORT_HTTPS=443 PX_RUNC=true PX_SECRETS_NAMESPACE=default PX_SHARED=/var/lib/kubelet:shared:1;/var/lib/osd:shared:1 PX_TEMPLATE_VERSION=v4 PX_VERSION=2.0.3.3-e3059a9 REPORTING_SECRET=portworx-enterprise-1-reporting-secret REPORTING_SECRET_NAMESPACE=default STORK_SERVICE_PORT=tcp://10.4.17.81:8099 STORK_SERVICE_PORT_8099_TCP=tcp://10.4.17.81:8099 STORK_SERVICE_PORT_8099_TCP_ADDR=10.4.17.81 STORK_SERVICE_PORT_8099_TCP_PORT=8099 STORK_SERVICE_PORT_8099_TCP_PROTO=tcp STORK_SERVICE_SERVICE_HOST=10.4.17.81 STORK_SERVICE_SERVICE_PORT=8099 TERM=xterm container=oci"
time="2019-12-13T21:39:40Z" level=info msg="portworx-reboot.service content unchanged [1dc97b965f3c6ad99aa3a92a02b2e8b1 /etc/systemd/system/portworx-reboot.service]"
time="2019-12-13T21:39:40Z" level=info msg="/etc/systemd/system/portworx.socket.service content unchanged [e80e04204b7a7d113db36c53f420d635 /etc/systemd/system/portworx.socket]"
time="2019-12-13T21:39:40Z" level=info msg="/etc/systemd/system/portworx-output.service.service content unchanged [6c3173fef9cb380b3b1334ebcf02ea19 /etc/systemd/system/portworx-output.service]"
time="2019-12-13T21:39:40Z" level=info msg="/etc/systemd/system/portworx.service.service content unchanged [047a6faad5865ec489b6db81422396fd /etc/systemd/system/portworx.service]"
time="2019-12-13T21:39:40Z" level=info msg="Portworx service restart not required."
time="2019-12-13T21:39:40Z" level=info msg="Activating node-watcher"
time="2019-12-13T21:39:40Z" level=info msg="Portworx service is ACTIVE"
time="2019-12-13T21:39:40Z" level=info msg="REST: Changing install-state: ST_INSTALL -> ST_FINISH"
time="2019-12-13T21:39:40Z" level=info msg="Start tailing portworx.service logs"
time="2019-12-13T21:39:41Z" level=info msg="> Starting local log-tailer"
time="2019-12-13T21:39:41Z" level=info msg="> run-local: /px-log-tail --follow -P @ -tf -u portworx.service -u portworx-output.service -u init.scope -n 20000 -p 16242"
time="2019-12-13T21:39:41Z" level=info msg="Install done - MAIN exiting"
time="2019-12-13T21:39:41Z" level=info msg="-- Flushing logs for PID 16242 [20000 lines] --"
time="2019-12-13T21:39:41Z" level=info msg="-- Start tailing the logs for portworx.service, portworx-output.service, init.scope --"
@gke-werkstorage-default-pool-2c8f45a9-j98r portworx[16226]: time="2019-12-13T21:39:39Z" level=info msg="Generated a new NodeIndex: 768"
@gke-werkstorage-default-pool-2c8f45a9-j98r portworx[16226]: time="2019-12-13T21:39:56Z" level=warning msg="Locked for 15 seconds" Error="ConfigMap is locked." Function=Lock Module=ConfigMap Name=px-cloud-drive-portworxcluster Owner=4689020673482198168
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment