Skip to content

Instantly share code, notes, and snippets.

@BFMeenink

BFMeenink/remounting-attempt.txt

Created September 30, 2019 14:16
Show Gist options
  • Save BFMeenink/be62e216568bf5b26bf8ca2ab73eb919 to your computer and use it in GitHub Desktop.
Save BFMeenink/be62e216568bf5b26bf8ca2ab73eb919 to your computer and use it in GitHub Desktop.
Download ZIP
RexRay S3 volume remounting attempt after breakage
Raw
remounting-attempt.txt
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-overlay2-f8f0331f1fc2ac1f488447502286420be6939772f4af0ff6ad6045aa65338d04\x2dinit-merged.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-overlay2-f8f0331f1fc2ac1f488447502286420be6939772f4af0ff6ad6045aa65338d04\x2dinit-merged.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850144356 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"Entering go-plugins-helpers getPath\" time=1569850144357 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850144358 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850144358 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144365 txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144366 txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144366 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850144367 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850144367 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850144367 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850144367 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850144367 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850144367 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850144367 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false value=[0] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=47ef1b4d-2463-47db-6b26-f3f94b64e663, txCR=1569850144\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144368 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850144368 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144369 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144369 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144369 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144369 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144369 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144370 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144370 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d728c0f4bdaa6f8f6976f4f3e859e89cc3a62951801fa5212dfff09958d7e4ff\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132904Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144370 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:29:05 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: 8oTu5RwgT9yyXWYmZ1UuoOyP9VuP719Rpa9OQ5dYfa6siFPCkL2YqoxZSSGJA5U49r68u2OvLJg=\\r\\nX-Amz-Request-Id: 06905C0A34955722\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144481 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144481 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850144483 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850144483 tls=false txCR=1569850144 txID=47ef1b4d-2463-47db-6b26-f3f94b64e663 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"volume inspected\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850144483 volume=&{remote-s3-bucket1 map[fields:map[] service:s3fs server:s3fs name:remote-s3-bucket1 size:0 iops:0 type: availabilityZone:]} volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-overlay2-f8f0331f1fc2ac1f488447502286420be6939772f4af0ff6ad6045aa65338d04-merged.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-overlay2-f8f0331f1fc2ac1f488447502286420be6939772f4af0ff6ad6045aa65338d04-merged.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 kernel: docker0: port 9(veth710b4fe) entered blocking state
Sep 30 13:29:04 prod-1-worker-10.0.21.50 kernel: docker0: port 9(veth710b4fe) entered disabled state
Sep 30 13:29:04 prod-1-worker-10.0.21.50 kernel: device veth710b4fe entered promiscuous mode
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd-udevd[4898]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd-udevd[4898]: Using default interface naming scheme 'v240'.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd-udevd[4898]: Could not generate persistent MAC address for veth710b4fe: No such file or directory
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd-udevd[4897]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd-udevd[4897]: Using default interface naming scheme 'v240'.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd-udevd[4897]: Could not generate persistent MAC address for vethf1e03c1: No such file or directory
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850144592 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"Entering go-plugins-helpers getPath\" time=1569850144593 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850144593 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"inspecting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=map[] osDriver=linux service=s3fs storageDriver=libstorage time=1569850144593 volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144607 txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144609 txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850144610 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false value=[0] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=a0b7474d-049d-455d-5250-418eb75ead13, txCR=1569850144\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144611 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850144611 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144612 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144612 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144612 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144613 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144613 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144613 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144613 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144614 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d728c0f4bdaa6f8f6976f4f3e859e89cc3a62951801fa5212dfff09958d7e4ff\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132904Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:29:05 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: KFwsWTLa0QfZ5/fgANvvmvvqazTBWYIuzmrm6m+CjVSPZNHyVqDswX2+Zp45HZBYoam9PO+hZ4E=\\r\\nX-Amz-Request-Id: 97581E63430A25A0\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144630 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144630 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"volume inspected\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850144631 volume=&{remote-s3-bucket1 map[fields:map[] service:s3fs server:s3fs name:remote-s3-bucket1 size:0 iops:0 type: availabilityZone:]} volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1569850144631 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"Entering go-plugins-helpers mountPath\" time=1569850144632 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850144634 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850144634 tls=false txCR=1569850144 txID=a0b7474d-049d-455d-5250-418eb75ead13 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"mounting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=&{false false <nil>} osDriver=linux service=s3fs storageDriver=libstorage time=1569850144638 volumeID= volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"mounting volume\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=&{false false <nil>} osDriver=linux service=s3fs storageDriver=libstorage time=1569850144638 volumeID= volumeName=remote-s3-bucket1 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144646 txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144647 txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144647 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=volumeInspect server=malachite-sight-ad time=1569850144648 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=storage-session-handler route=volumeInspect server=malachite-sight-ad time=1569850144648 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=volumeInspect server=malachite-sight-ad time=1569850144648 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=volumeInspect server=malachite-sight-ad time=1569850144648 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850144648 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=volumeInspect server=malachite-sight-ad time=1569850144649 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=volumeInspect server=malachite-sight-ad time=1569850144649 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=volumeInspect server=malachite-sight-ad time=1569850144649 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=volumeInspect server=malachite-sight-ad time=1569850144650 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=volumeInspect server=malachite-sight-ad time=1569850144650 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=volumeInspect server=malachite-sight-ad time=1569850144650 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=attachments len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850144650 tls=false value=[31] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=byName len(value)=1 route=volumeInspect server=malachite-sight-ad time=1569850144650 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94, txCR=1569850144\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144651 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144651 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144651 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[s3fs=] host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad time=1569850144651 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=volumeInspect server=malachite-sight-ad service=s3fs time=1569850144652 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144652 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"driver is not StorageDriverWithLogin\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144652 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144652 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144654 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144654 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144655 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"got existing s3 service connection\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" region=eu-west-1 route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144655 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"DEBUG: Request s3/ListBuckets Details:\\n---[ REQUEST POST-SIGN ]-----------------------------\\nGET / HTTP/1.1\\r\\nHost: s3.eu-west-1.amazonaws.com\\r\\nUser-Agent: aws-sdk-go/1.12.46 (go1.9.1; linux; amd64)\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=<access key>/20190930/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d728c0f4bdaa6f8f6976f4f3e859e89cc3a62951801fa5212dfff09958d7e4ff\\r\\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\r\\nX-Amz-Date: 20190930T132904Z\\r\\nAccept-Encoding: gzip\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144657 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"DEBUG: Response s3/ListBuckets Details:\\n---[ RESPONSE ]--------------------------------------\\nHTTP/1.1 200 OK\\r\\nTransfer-Encoding: chunked\\r\\nContent-Type: application/xml\\r\\nDate: Mon, 30 Sep 2019 13:29:05 GMT\\r\\nServer: AmazonS3\\r\\nX-Amz-Id-2: fymTS2hjQCnZVcjF+l4FUXYB90PemK9SEyXRWz/ztGe0lR8Of0fAOaB7Lr5WZWSZ4GvaEDWqZMQ=\\r\\nX-Amz-Request-Id: F1708D6F7F7EE6E8\\r\\n\\r\\n\\n-----------------------------------------------------\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<ListAllMyBucketsResult xmlns=\\\"http://s3.amazonaws.com/doc/2006-03-01/\\\"><Owner><ID>98c8041cb8ad8a1d20c5a06c7980acf116e8308453101f4f0ca54be4011c599a</ID><DisplayName><team name>+aws-<client></DisplayName></Owner><Buckets><Bucket><Name>remote-s3-bucket2</Name><CreationDate>2019-05-21T10:53:24.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket3</Name><CreationDate>2019-05-21T10:28:05.000Z</CreationDate></Bucket><Bucket><Name>remote-s3-bucket1</Name><CreationDate>2019-05-22T08:43:27.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>\" " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"manually calculating attachment state\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144670 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"including volume\" attachmentState=attached host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144670 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144670 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 time=1569850144670 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" removedAfter=0s route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=1 time=1569850144672 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=volumeInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs task=0 tasksLen=0 time=1569850144672 tls=false txCR=1569850144 txID=37fe548d-95ff-4b11-6daf-e6946d0c8f94 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850144673 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=schema-validator route=serviceInspect server=malachite-sight-ad time=1569850144673 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-svc-handler route=serviceInspect server=malachite-sight-ad time=1569850144674 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=service-validator route=serviceInspect server=malachite-sight-ad time=1569850144674 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=serviceInspect server=malachite-sight-ad time=1569850144674 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=local-devices-handler route=serviceInspect server=malachite-sight-ad time=1569850144674 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=instanceIDs-handler route=serviceInspect server=malachite-sight-ad time=1569850144674 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=auth-global-handler route=serviceInspect server=malachite-sight-ad time=1569850144675 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=error-handler route=serviceInspect server=malachite-sight-ad time=1569850144675 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=transaction-handler route=serviceInspect server=malachite-sight-ad time=1569850144675 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/508982324.sock\" middleware=query-params-handler route=serviceInspect server=malachite-sight-ad time=1569850144675 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/508982324.sock\" key=instance len(value)=1 route=serviceInspect server=malachite-sight-ad time=1569850144675 tls=false value=[] " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=5ebd4fe0-8bb7-4e07-49b3-92ae2f780262, txCR=1569850144\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850144676 tls=false " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850144677 tls=false txCR=1569850144 txID=5ebd4fe0-8bb7-4e07-49b3-92ae2f780262 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[s3fs=prod-1-worker-10.0.21.50] host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850144677 tls=false txCR=1569850144 txID=5ebd4fe0-8bb7-4e07-49b3-92ae2f780262 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[] host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad time=1569850144677 tls=false txCR=1569850144 txID=5ebd4fe0-8bb7-4e07-49b3-92ae2f780262 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/508982324.sock\" route=serviceInspect server=malachite-sight-ad service=s3fs time=1569850144677 tls=false txCR=1569850144 txID=5ebd4fe0-8bb7-4e07-49b3-92ae2f780262 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/508982324.sock\" instanceID=\"s3fs=prod-1-worker-10.0.21.50\" route=serviceInspect server=malachite-sight-ad service=s3fs storageDriver=s3fs time=1569850144677 tls=false txCR=1569850144 txID=5ebd4fe0-8bb7-4e07-49b3-92ae2f780262 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144694 txCR=1569850144 txID=8b63141c-515b-47f6-743e-690e9763b0ea " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"xli mounts success\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144695 txCR=1569850144 txID=8b63141c-515b-47f6-743e-690e9763b0ea " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"storage executor initialized\" cmd=s3fs driver=s3fs host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux opts=[allow_other,iam_role=auto,umask=000] osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144706 txCR=1569850144 txID=bb9c5de0-8e36-43a2-52bf-e180adc7cc72 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=debug msg=\"attempting s3fs mount\" args=[remote-s3-bucket1 /var/lib/rexray/volumes/remote-s3-bucket1 -oallow_other,iam_role=auto,umask=000] bucket=remote-s3-bucket1 cmd=s3fs host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux isAWSAuthEnvVars=true mountPoint=\"/var/lib/rexray/volumes/remote-s3-bucket1\" osDriver=linux server=malachite-sight-ad service=s3fs storageDriver=libstorage time=1569850144716 txCR=1569850144 txID=bb9c5de0-8e36-43a2-52bf-e180adc7cc72 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04Z" level=error msg="time=\"2019-09-30T13:29:04Z\" level=error msg=\"docker-legacy: Mount: remote-s3-bucket1: failed: error mounting s3fs bucket\" host=\"unix:///var/run/rexray/508982324.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1569850144725 " plugin=47e7a1618608ff2423824400345618f2f04da6e279bc181c6af62b84233fa75f
Sep 30 13:29:04 prod-1-worker-10.0.21.50 kernel: docker0: port 9(veth710b4fe) entered disabled state
Sep 30 13:29:04 prod-1-worker-10.0.21.50 kernel: device veth710b4fe left promiscuous mode
Sep 30 13:29:04 prod-1-worker-10.0.21.50 kernel: docker0: port 9(veth710b4fe) entered disabled state
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-containers-8ac64e58c1143547c5f00df59a0dc597d318818926891911f000f2be39663a44-mounts-shm.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-containers-8ac64e58c1143547c5f00df59a0dc597d318818926891911f000f2be39663a44-mounts-shm.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[1]: var-lib-docker-overlay2-f8f0331f1fc2ac1f488447502286420be6939772f4af0ff6ad6045aa65338d04-merged.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 systemd[30765]: var-lib-docker-overlay2-f8f0331f1fc2ac1f488447502286420be6939772f4af0ff6ad6045aa65338d04-merged.mount: Succeeded.
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04.815163537Z" level=error msg="8ac64e58c1143547c5f00df59a0dc597d318818926891911f000f2be39663a44 cleanup: failed to delete container from containerd: no such container"
Sep 30 13:29:04 prod-1-worker-10.0.21.50 env[1463]: time="2019-09-30T13:29:04.823315113Z" level=error msg="Handler for POST /v1.38/containers/8ac64e58c1143547c5f00df59a0dc597d318818926891911f000f2be39663a44/start returned error: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: remote-s3-bucket1: failed: error mounting s3fs bucket"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment