Created
September 30, 2019 14:15
-
-
Save BFMeenink/d63c4e399f818ce372ef0b1c1bb8a391 to your computer and use it in GitHub Desktop.
RexRay volume unmounting during container exit
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1290]: time="2019-09-30T13:28:35Z" level=info msg="shim reaped" id=0dab2f101ac8472dbe893ee4ed56ddd7db86f817e6d9e6ba6c176ebceee2b084 | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35.817204775Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd-networkd[1087]: vethfb38202: Lost carrier | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 kernel: docker0: port 9(vethfb38202) entered disabled state | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 kernel: veth01fae9c: renamed from eth0 | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd-udevd[4854]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd-udevd[4854]: Using default interface naming scheme 'v240'. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 kernel: docker0: port 9(vethfb38202) entered disabled state | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 kernel: device vethfb38202 left promiscuous mode | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 kernel: docker0: port 9(vethfb38202) entered disabled state | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd[30765]: run-docker-netns-452df20c8462.mount: Succeeded. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd[1]: run-docker-netns-452df20c8462.mount: Succeeded. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-containers-0dab2f101ac8472dbe893ee4ed56ddd7db86f817e6d9e6ba6c176ebceee2b084-mounts-shm.mount: Succeeded. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-containers-0dab2f101ac8472dbe893ee4ed56ddd7db86f817e6d9e6ba6c176ebceee2b084-mounts-shm.mount: Succeeded. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-overlay2-45d55a405fa61a019642e0165737e9dc91ac1c01e36bb6407ca8f2fb159207ba-merged.mount: Succeeded. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-overlay2-45d55a405fa61a019642e0165737e9dc91ac1c01e36bb6407ca8f2fb159207ba-merged.mount: Succeeded. | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850115927 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=info msg=\"Entering go-plugins-helpers getPath\" time=1569850115927 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850115928 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=info msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850115928 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850115935 txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850115936 txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850115939 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850115939 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850115940 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850115940 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850115940 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850115940 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850115940 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850115941 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850115941 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850115941 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850115941 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850115941 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850115942 tls=false value=[0] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850115942 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=bc98de2e-60f1-4d00-6c58-a13197c57f16, txCR=1569850115\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850115942 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850115942 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850115942 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850115943 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850115943 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850115943 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850115943 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850115943 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850115944 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850115944 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850115944 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850115945 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e7cd762363001eb838b41a532a9b931183f7414186f54fd1d7fbdf6a7a20dc85\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132835Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:35 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:35Z" level=error msg="time=\"2019-09-30T13:28:35Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850115947 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:28:36 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: LBi7zQhRG4NX+YgzqlFqjP7g7rOhOzM8GJHhVpwHsgbqLwksx999jDUPA9T+/qRELgbCJbQLXcM=\\r\\nX-Amz-Request-Id: E1B0240572BA909C\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116080 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116080 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850116083 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850116083 tls=false txCR=1569850115 txID=bc98de2e-60f1-4d00-6c58-a13197c57f16 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"volume inspected\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850116084 volume=&{remote-s3-bucket1 map[service:s3fs server:s3fs name:remote-s3-bucket1 size:0 iops:0 type: availabilityZone: fields:map[]]} volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850116085 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"Entering go-plugins-helpers unmountPath\" time=1569850116085 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"unmounting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850116085 volumeID= volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"count reset\" count=1 host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850116090 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"init count\" count=0 host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850116090 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"unmounting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850116091 volumeID= volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850116098 txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850116099 txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850116099 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false value=[15] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=59624d22-004e-4e84-7107-7d2053361b51, txCR=1569850116\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850116100 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850116100 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116100 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116100 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116100 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116101 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116101 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116102 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116102 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116102 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=059de8cce9a4ed358d6763d4953fa1760510b9873089112e7811266b23f45abf\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132836Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:28:37 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: hBjCLAl2aP4aDSIIwH05LCDO/e6hJtoL0neSzfnpsrqIfpURkHXdTyWVuIIxc+oqzTrqmY6wyjs=\\r\\nX-Amz-Request-Id: CBB3403E95D53ED1\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"manually calculating attachment state\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116121 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"including volume\" attachmentState=attached host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116124 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116124 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116124 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850116125 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=serviceInspect server=malachite-sight-ad time=1569850116125 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=serviceInspect server=malachite-sight-ad time=1569850116125 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=serviceInspect server=malachite-sight-ad time=1569850116125 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=serviceInspect server=malachite-sight-ad time=1569850116126 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=serviceInspect server=malachite-sight-ad time=1569850116126 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=serviceInspect server=malachite-sight-ad time=1569850116126 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=serviceInspect server=malachite-sight-ad time=1569850116126 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=serviceInspect server=malachite-sight-ad time=1569850116126 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=serviceInspect server=malachite-sight-ad time=1569850116126 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=serviceInspect server=malachite-sight-ad time=1569850116126 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=instance len(value)=1 route=serviceInspect server=malachite-sight-ad time=1569850116127 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=0cc5a5b2-9896-4df2-5210-a4c2ecb89dee, txCR=1569850116\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850116127 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850116127 tls=false txCR=1569850116 txID=0cc5a5b2-9896-4df2-5210-a4c2ecb89dee " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850116127 tls=false txCR=1569850116 txID=0cc5a5b2-9896-4df2-5210-a4c2ecb89dee " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[] host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850116128 tls=false txCR=1569850116 txID=0cc5a5b2-9896-4df2-5210-a4c2ecb89dee " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad service=s3fs time=1569850116128 tls=false txCR=1569850116 txID=0cc5a5b2-9896-4df2-5210-a4c2ecb89dee " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=serviceInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116128 tls=false txCR=1569850116 txID=0cc5a5b2-9896-4df2-5210-a4c2ecb89dee " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850116129 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850116129 tls=false txCR=1569850116 txID=59624d22-004e-4e84-7107-7d2053361b51 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850116137 txCR=1569850116 txID=defefa8e-106a-41a5-757d-63879e761c5a " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"xli mounts success\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850116138 txCR=1569850116 txID=defefa8e-106a-41a5-757d-63879e761c5a " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850116147 txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850116148 txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeDetach server=malachite-sight-ad time=1569850116148 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=post-args-handler route=volumeDetach server=malachite-sight-ad time=1569850116148 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeDetach server=malachite-sight-ad time=1569850116149 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeDetach server=malachite-sight-ad time=1569850116149 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeDetach server=malachite-sight-ad time=1569850116149 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeDetach server=malachite-sight-ad time=1569850116149 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeDetach server=malachite-sight-ad time=1569850116150 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeDetach server=malachite-sight-ad time=1569850116150 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeDetach server=malachite-sight-ad time=1569850116150 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeDetach server=malachite-sight-ad time=1569850116150 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeDetach server=malachite-sight-ad time=1569850116150 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeDetach server=malachite-sight-ad time=1569850116150 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeDetach server=malachite-sight-ad time=1569850116150 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=detach len(value)=1 route=volumeDetach server=malachite-sight-ad time=1569850116151 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=5989e651-fdbe-4da6-4622-f7e43e86bc67, txCR=1569850116\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeDetach server=malachite-sight-ad time=1569850116151 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeDetach server=malachite-sight-ad time=1569850116151 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeDetach server=malachite-sight-ad time=1569850116151 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeDetach server=malachite-sight-ad time=1569850116151 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeDetach server=malachite-sight-ad service=s3fs time=1569850116151 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116151 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116152 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"validating schema\" body=\"{}\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach schema=\"{\\n \\\"$schema\\\": \\\"http://json-schema.org/draft-04/schema#\\\",\\n \\\"$ref\\\": \\\"https://github.com/rexray/libstorage#/definitions/volumeDetachRequest\\\"\\n}\" server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116152 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116158 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116159 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116159 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116159 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"DEBUG: Request s3/HeadBucket Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nHEAD /remote-s3-bucket1 HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7a0cedd16d57475ea0cc30c446008a25758af85e80b9e6aeb45e602d0e43fb6e\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132836Z\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850116160 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"DEBUG: Response s3/HeadBucket Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:28:37 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Bucket-Region: eu-west-1\\r\\nX-Amz-Id-2: TRseGLuKbOKDP200QVuiZwbf8/6SGiEnryOc3ay7tJZA9s8A2hpF/n3jEQbrsoqb/V7fWWnRGmY=\\r\\nX-Amz-Request-Id: 88A797D0752741F2\\r\\n\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"skipping response schema validation; schema == nil\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116172 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850116172 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=info msg=\"unmounted and detached volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850116173 vol=&{[0xc420124460] available false 0 remote-s3-bucket1 0 remote-s3-bucket1 map[]} " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850116175 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f | |
| Sep 30 13:28:36 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:28:36Z" level=error msg="time=\"2019-09-30T13:28:36Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeDetach server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850116175 tls=false txCR=1569850116 txID=5989e651-fdbe-4da6-4622-f7e43e86bc67 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment