Skip to content

Instantly share code, notes, and snippets.

@BFMeenink

BFMeenink/container-startup-logs.txt

Created September 30, 2019 14:14
Show Gist options
  • Save BFMeenink/4ee8ebaf08fb0d7231bd8605db83fe15 to your computer and use it in GitHub Desktop.
Save BFMeenink/4ee8ebaf08fb0d7231bd8605db83fe15 to your computer and use it in GitHub Desktop.
Download ZIP
RexRay logging during container startup with RexRay S3 volume mounted
Raw
container-startup-logs.txt
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-overlay2-45d55a405fa61a019642e0165737e9dc91ac1c01e36bb6407ca8f2fb159207ba\x2dinit-merged.mount: Succeeded.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-overlay2-45d55a405fa61a019642e0165737e9dc91ac1c01e36bb6407ca8f2fb159207ba\x2dinit-merged.mount: Succeeded.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850061261 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"Entering go-plugins-helpers getPath\" time=1569850061263 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850061263 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850061263 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061270 txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061271 txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850061272 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false value=[0] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=f876114b-4f0d-4215-4f54-1724758c4797, txCR=1569850061\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061273 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850061273 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061273 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061273 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061274 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061274 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061274 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061275 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061275 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3e03cd22c0ab1f2cfeabfc849b2e5fbc2d36887b993367d4a3e2726cad61b038\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132741Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061276 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:27:42 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: oyLqGDN2pmchMD0UKmnE+uemikuzuYs1cMmksmQBIlgiCvN7tyU+lgLAO5KruIGWMKfi0Jpz1RU=\\r\\nX-Amz-Request-Id: 3908BB0723DDCA03\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061379 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061379 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850061381 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850061381 tls=false txCR=1569850061 txID=f876114b-4f0d-4215-4f54-1724758c4797 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"volume inspected\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850061381 volume=&{remote-s3-bucket1 map[type: availabilityZone: fields:map[] service:s3fs server:s3fs name:remote-s3-bucket1 size:0 iops:0]} volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-overlay2-45d55a405fa61a019642e0165737e9dc91ac1c01e36bb6407ca8f2fb159207ba-merged.mount: Succeeded.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-overlay2-45d55a405fa61a019642e0165737e9dc91ac1c01e36bb6407ca8f2fb159207ba-merged.mount: Succeeded.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd-udevd[4635]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd-udevd[4635]: Using default interface naming scheme 'v240'.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd-udevd[4635]: Could not generate persistent MAC address for vethfb38202: No such file or directory
Sep 30 13:27:41 prod-1-worker-10.0.21.50 kernel: docker0: port 9(vethfb38202) entered blocking state
Sep 30 13:27:41 prod-1-worker-10.0.21.50 kernel: docker0: port 9(vethfb38202) entered disabled state
Sep 30 13:27:41 prod-1-worker-10.0.21.50 kernel: device vethfb38202 entered promiscuous mode
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd-udevd[4632]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd-udevd[4632]: Using default interface naming scheme 'v240'.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd-udevd[4632]: Could not generate persistent MAC address for veth01fae9c: No such file or directory
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850061476 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"Entering go-plugins-helpers getPath\" time=1569850061477 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850061477 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850061477 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061495 txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061496 txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061498 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850061498 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850061498 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850061499 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850061499 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850061499 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850061499 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850061499 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850061499 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850061500 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850061500 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850061500 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850061500 tls=false value=[0] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850061500 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6, txCR=1569850061\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061501 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061501 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061501 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061501 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850061501 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061502 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061502 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061502 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061503 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061503 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061503 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061503 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3e03cd22c0ab1f2cfeabfc849b2e5fbc2d36887b993367d4a3e2726cad61b038\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132741Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061505 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:27:42 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: X8xu2A4lxWcMft7HfOgpxNdDOVGzUCQk3ctQshBrJQz9zj0hgGcQ5r4yJB+X+n0GtffPgBMgRv8=\\r\\nX-Amz-Request-Id: 40C614BE42C5F729\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061516 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061516 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"volume inspected\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850061517 volume=&{remote-s3-bucket1 map[name:remote-s3-bucket1 size:0 iops:0 type: availabilityZone: fields:map[] service:s3fs server:s3fs]} volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850061517 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850061517 tls=false txCR=1569850061 txID=f4a39c99-203a-4db7-6c21-e8cf020be7b6 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850061518 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"Entering go-plugins-helpers mountPath\" time=1569850061518 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"mounting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=&{false false <nil>} osDriver=linux service=s3fs storageDriver=libstorage time=1569850061522 volumeID= volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"mounting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=&{false false <nil>} osDriver=linux service=s3fs storageDriver=libstorage time=1569850061522 volumeID= volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061529 txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061529 txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061530 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850061530 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850061530 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850061530 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false value=[31] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e, txCR=1569850061\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850061531 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850061531 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061531 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061531 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061532 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061533 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061533 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061533 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061533 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061533 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3e03cd22c0ab1f2cfeabfc849b2e5fbc2d36887b993367d4a3e2726cad61b038\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132741Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:27:42 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: I/Vcma7APZnu+BPbv8mUUTGjpYG8j6wodC6cm8rzQVhZSq2BROfaNrw67dDJ/VppyhBJUB3a4TY=\\r\\nX-Amz-Request-Id: 8667253DF891A759\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"manually calculating attachment state\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061546 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"including volume\" attachmentState=attached host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061546 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061546 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850061546 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850061548 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850061548 tls=false txCR=1569850061 txID=4d39f24e-c2bc-4bea-7371-8d43d83ff30e " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850061548 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=serviceInspect server=malachite-sight-ad time=1569850061548 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=serviceInspect server=malachite-sight-ad time=1569850061548 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=serviceInspect server=malachite-sight-ad time=1569850061549 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=instance len(value)=1 route=serviceInspect server=malachite-sight-ad time=1569850061550 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=2c3410e3-7e48-4019-4e3c-df32c92ec0fc, txCR=1569850061\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850061550 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850061550 tls=false txCR=1569850061 txID=2c3410e3-7e48-4019-4e3c-df32c92ec0fc " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850061550 tls=false txCR=1569850061 txID=2c3410e3-7e48-4019-4e3c-df32c92ec0fc " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[] host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850061550 tls=false txCR=1569850061 txID=2c3410e3-7e48-4019-4e3c-df32c92ec0fc " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad service=s3fs time=1569850061552 tls=false txCR=1569850061 txID=2c3410e3-7e48-4019-4e3c-df32c92ec0fc " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=serviceInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850061552 tls=false txCR=1569850061 txID=2c3410e3-7e48-4019-4e3c-df32c92ec0fc " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061564 txCR=1569850061 txID=782a015d-7d69-40bb-72dc-59a595c0c0d9 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"xli mounts success\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061565 txCR=1569850061 txID=782a015d-7d69-40bb-72dc-59a595c0c0d9 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061576 txCR=1569850061 txID=436edfe6-65b2-4ac8-4029-1017e3112837 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"attempting s3fs mount\" args=[remote-s3-bucket1 /var/lib/rexray/volumes/remote-s3-bucket1 -oallow_other,iam_role=auto,umask=000] bucket=remote-s3-bucket1 cmd=s3fs host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux isAWSAuthEnvVars=true mountPoint=\"/var/lib/rexray/volumes/remote-s3-bucket1\" osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061585 txCR=1569850061 txID=436edfe6-65b2-4ac8-4029-1017e3112837 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"xli mount success\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850061599 txCR=1569850061 txID=436edfe6-65b2-4ac8-4029-1017e3112837 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-plugins-47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f-propagated\x2dmount-volumes-<team name>\x2dxrrlqmpasjo\x2dapp\x2dresticrepo\x2dprod.mount: Succeeded.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-plugins-47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f-propagated\x2dmount-volumes-<team name>\x2dxrrlqmpasjo\x2dapp\x2dresticrepo\x2dprod.mount: Succeeded.
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=info msg=\"volume mounted\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux mntPath=\"/var/lib/rexray/volumes/remote-s3-bucket1/data\" osDriver=linux service=s3fs storageDriver=libstorage time=1569850061737 vol=&{[0xc42006d220] attached false 0 remote-s3-bucket1 0 remote-s3-bucket1 map[]} " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:27:41Z" level=error msg="time=\"2019-09-30T13:27:41Z\" level=debug msg=\"set count\" count=1 host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850061737 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1290]: time="2019-09-30T13:27:41Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/0dab2f101ac8472dbe893ee4ed56ddd7db86f817e6d9e6ba6c176ebceee2b084/shim.sock" debug=true pid=4649
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1290]: time="2019-09-30T13:27:41Z" level=debug msg="registering ttrpc server"
Sep 30 13:27:41 prod-1-worker-10.0.21.50 env[1290]: time="2019-09-30T13:27:41Z" level=debug msg="serving api on unix socket" socket="[inherited from parent]"
Sep 30 13:27:41 prod-1-worker-10.0.21.50 kernel: eth0: renamed from veth01fae9c
Sep 30 13:27:41 prod-1-worker-10.0.21.50 kernel: docker0: port 9(vethfb38202) entered blocking state
Sep 30 13:27:41 prod-1-worker-10.0.21.50 kernel: docker0: port 9(vethfb38202) entered forwarding state
Sep 30 13:27:41 prod-1-worker-10.0.21.50 systemd-networkd[1087]: vethfb38202: Gained carrier
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment