Skip to content

Instantly share code, notes, and snippets.

@yebyen
Created July 20, 2016 16:52
Show Gist options
  • Save yebyen/802a2122d6a02134b0001b8723afa1aa to your computer and use it in GitHub Desktop.
Save yebyen/802a2122d6a02134b0001b8723afa1aa to your computer and use it in GitHub Desktop.
2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:14:44UTC deis-database[1]: data=data)
2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:14:44UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:14:44UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:14:44UTC deis-database[1]: raise ex
2016-07-20T15:14:44UTC deis-database[1]: timeout: timed out
2016-07-20T15:14:44UTC deis-database[1]:
2016-07-20T15:14:44UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:14:44UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:14:44.891200-00 pid=3939
2016-07-20T15:15:47UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:15:47UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:15:47UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:15:47UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:15:47UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:15:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:15:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:15:47UTC deis-database[1]: query_args=query_args
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:15:47UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:15:47UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:15:47UTC deis-database[1]: raise ex
2016-07-20T15:15:47UTC deis-database[1]: timeout: timed out
2016-07-20T15:15:47UTC deis-database[1]: <Greenlet at 0x7f17de5c7af0: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:15:47UTC deis-database[1]:
2016-07-20T15:15:48UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:15:48UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:15:48UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:15:48UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:15:48UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:15:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:15:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:15:48UTC deis-database[1]: query_args=query_args
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:15:48UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:15:48UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:15:48UTC deis-database[1]: raise ex
2016-07-20T15:15:48UTC deis-database[1]: timeout: timed out
2016-07-20T15:15:48UTC deis-database[1]: <Greenlet at 0x7f17de35f190: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:15:48UTC deis-database[1]:
2016-07-20T15:15:55UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:15:55UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:15:55UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:15:55UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:15:55UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:15:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:15:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:15:55UTC deis-database[1]: query_args=query_args
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:15:55UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:15:55UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:15:55UTC deis-database[1]: raise ex
2016-07-20T15:15:55UTC deis-database[1]: timeout: timed out
2016-07-20T15:15:55UTC deis-database[1]: <Greenlet at 0x7f17de35f050: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:15:55UTC deis-database[1]:
2016-07-20T15:16:04UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:16:04UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:16:04UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:16:04UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:16:04UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:16:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:16:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:16:04UTC deis-database[1]: query_args=query_args
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:16:04UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:16:04UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:16:04UTC deis-database[1]: raise ex
2016-07-20T15:16:04UTC deis-database[1]: timeout: timed out
2016-07-20T15:16:04UTC deis-database[1]: <Greenlet at 0x7f17de35f0f0: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:16:04UTC deis-database[1]:
2016-07-20T15:16:15UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:16:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:16:15UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:16:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:16:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:16:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:16:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:16:15UTC deis-database[1]: query_args=query_args
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:16:15UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:16:15UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:16:15UTC deis-database[1]: raise ex
2016-07-20T15:16:15UTC deis-database[1]: timeout: timed out
2016-07-20T15:16:15UTC deis-database[1]: <Greenlet at 0x7f17de5c7eb0: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:16:15UTC deis-database[1]:
2016-07-20T15:16:16UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:16:16UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:16:16UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:16:16UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:16:16UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:16:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:16:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:16:16UTC deis-database[1]: query_args=query_args
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:16:16UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:16:16UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:16:16UTC deis-database[1]: raise ex
2016-07-20T15:16:16UTC deis-database[1]: timeout: timed out
2016-07-20T15:16:16UTC deis-database[1]: <Greenlet at 0x7f17de5c7f50: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:16:16UTC deis-database[1]:
2016-07-20T15:16:17UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:16:17UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:16:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:16:17UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:16:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:16:17UTC deis-database[1]: group.join()
2016-07-20T15:16:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:16:17UTC deis-database[1]: raise val
2016-07-20T15:16:17UTC deis-database[1]: timeout: timed out
2016-07-20T15:16:17UTC deis-database[1]:
2016-07-20T15:16:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:17.431441-00 pid=668
2016-07-20T15:16:17UTC deis-database[1]: 2016-07-20 15:16:17 UTC LOG: archive command failed with exit code 2
2016-07-20T15:16:17UTC deis-database[1]: 2016-07-20 15:16:17 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:16:18UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:16:18UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:16:18UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:16:18UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:16:18UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:16:18UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:16:18UTC deis-database[1]: data=data)
2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:16:18UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:16:18UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:16:18UTC deis-database[1]: raise ex
2016-07-20T15:16:18UTC deis-database[1]: timeout: timed out
2016-07-20T15:16:18UTC deis-database[1]:
2016-07-20T15:16:18UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.421876-00 pid=3939
2016-07-20T15:16:18UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:16:18UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.795435-00 pid=833
2016-07-20T15:16:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.820306-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:16:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.966591-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.108685-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.247890-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.387682-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.528131-00 pid=833 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.673231-00 pid=833 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.816419-00 pid=833 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:17:14UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:17:14UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:17:14UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:17:14UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:17:14UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:17:14UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:17:14UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:17:14UTC deis-database[1]: query_args=query_args
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:17:14UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:17:14UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:17:14UTC deis-database[1]: raise ex
2016-07-20T15:17:14UTC deis-database[1]: timeout: timed out
2016-07-20T15:17:14UTC deis-database[1]: <Greenlet at 0x7fab9395d190: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:17:14UTC deis-database[1]:
2016-07-20T15:17:18UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:17:18UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:17:18UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:17:18UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:17:18UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:17:18UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:17:18UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:17:18UTC deis-database[1]: query_args=query_args
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:17:18UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:17:18UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:17:18UTC deis-database[1]: raise ex
2016-07-20T15:17:18UTC deis-database[1]: timeout: timed out
2016-07-20T15:17:18UTC deis-database[1]: <Greenlet at 0x7fab93bc5af0: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:17:18UTC deis-database[1]:
2016-07-20T15:17:21UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:17:21UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:17:21UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:17:21UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:17:21UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:17:21UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:17:21UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:17:21UTC deis-database[1]: query_args=query_args
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:17:21UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:17:21UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:17:21UTC deis-database[1]: raise ex
2016-07-20T15:17:21UTC deis-database[1]: timeout: timed out
2016-07-20T15:17:21UTC deis-database[1]: <Greenlet at 0x7fab9395d050: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:17:21UTC deis-database[1]:
2016-07-20T15:17:34UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:17:34UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:17:34UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:17:34UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:17:34UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:17:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:17:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:17:34UTC deis-database[1]: query_args=query_args
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:17:34UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:17:34UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:17:34UTC deis-database[1]: raise ex
2016-07-20T15:17:34UTC deis-database[1]: timeout: timed out
2016-07-20T15:17:34UTC deis-database[1]: <Greenlet at 0x7fab9395d0f0: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:17:34UTC deis-database[1]:
2016-07-20T15:17:44UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:17:44UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:17:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:17:44UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:17:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:17:44UTC deis-database[1]: group.join()
2016-07-20T15:17:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:17:44UTC deis-database[1]: raise val
2016-07-20T15:17:44UTC deis-database[1]: timeout: timed out
2016-07-20T15:17:44UTC deis-database[1]:
2016-07-20T15:17:44UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:44.968579-00 pid=833
2016-07-20T15:17:44UTC deis-database[1]: 2016-07-20 15:17:44 UTC LOG: archive command failed with exit code 2
2016-07-20T15:17:44UTC deis-database[1]: 2016-07-20 15:17:44 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:17:44UTC deis-database[1]: 2016-07-20 15:17:44 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:17:45UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:17:45UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.316185-00 pid=956
2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.341471-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.490105-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.628498-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.769306-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.914565-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:17:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:17:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:46.072386-00 pid=956 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:17:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:17:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:46.209305-00 pid=956 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:17:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:17:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:17:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:46.360380-00 pid=956 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:18:35UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:18:35UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:18:35UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:18:35UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:18:35UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:18:35UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:18:35UTC deis-database[1]: data=data)
2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:18:35UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:18:35UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:18:35UTC deis-database[1]: raise ex
2016-07-20T15:18:35UTC deis-database[1]: timeout: timed out
2016-07-20T15:18:35UTC deis-database[1]:
2016-07-20T15:18:35UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:18:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:18:35.336780-00 pid=3939
2016-07-20T15:18:58UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:18:58UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:18:58UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:18:58UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:18:58UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:18:58UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:18:58UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:18:58UTC deis-database[1]: query_args=query_args
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:18:58UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:18:58UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:18:58UTC deis-database[1]: raise ex
2016-07-20T15:18:58UTC deis-database[1]: timeout: timed out
2016-07-20T15:18:58UTC deis-database[1]: <Greenlet at 0x7fce507c2af0: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:18:58UTC deis-database[1]:
2016-07-20T15:19:03UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:19:03UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:19:03UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:19:03UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:19:03UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:19:03UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:19:03UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:19:03UTC deis-database[1]: query_args=query_args
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:19:03UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:19:03UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:19:03UTC deis-database[1]: raise ex
2016-07-20T15:19:03UTC deis-database[1]: timeout: timed out
2016-07-20T15:19:03UTC deis-database[1]: <Greenlet at 0x7fce5055a0f0: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:19:03UTC deis-database[1]:
2016-07-20T15:19:05UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:19:05UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:19:05UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:19:05UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:19:05UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:19:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:19:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:19:05UTC deis-database[1]: query_args=query_args
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:19:05UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:19:05UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:19:05UTC deis-database[1]: raise ex
2016-07-20T15:19:05UTC deis-database[1]: timeout: timed out
2016-07-20T15:19:05UTC deis-database[1]: <Greenlet at 0x7fce5055a190: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:19:05UTC deis-database[1]:
2016-07-20T15:19:17UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:19:17UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:19:17UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:19:17UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:19:17UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:19:17UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:19:17UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:19:17UTC deis-database[1]: query_args=query_args
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:19:17UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:19:17UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:19:17UTC deis-database[1]: raise ex
2016-07-20T15:19:17UTC deis-database[1]: timeout: timed out
2016-07-20T15:19:17UTC deis-database[1]: <Greenlet at 0x7fce5055a230: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:19:17UTC deis-database[1]:
2016-07-20T15:19:25UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:19:25UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:19:25UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:19:25UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:19:25UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:19:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:19:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:19:25UTC deis-database[1]: query_args=query_args
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:19:25UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:19:25UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:19:25UTC deis-database[1]: raise ex
2016-07-20T15:19:25UTC deis-database[1]: timeout: timed out
2016-07-20T15:19:25UTC deis-database[1]: <Greenlet at 0x7fce507c2eb0: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:19:25UTC deis-database[1]:
2016-07-20T15:19:28UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:19:28UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:19:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:19:28UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:19:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:19:28UTC deis-database[1]: group.join()
2016-07-20T15:19:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:19:28UTC deis-database[1]: raise val
2016-07-20T15:19:28UTC deis-database[1]: timeout: timed out
2016-07-20T15:19:28UTC deis-database[1]:
2016-07-20T15:19:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:28.122127-00 pid=956
2016-07-20T15:19:28UTC deis-database[1]: 2016-07-20 15:19:28 UTC LOG: archive command failed with exit code 2
2016-07-20T15:19:28UTC deis-database[1]: 2016-07-20 15:19:28 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:19:29UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:19:29UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.495208-00 pid=1103
2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.525129-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.676790-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.825022-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.967671-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.121673-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.266397-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.413886-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.565544-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:19:55UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:19:55UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:19:55UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:19:55UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:19:55UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:19:55UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:19:55UTC deis-database[1]: data=data)
2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:19:55UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:19:55UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:19:55UTC deis-database[1]: raise ex
2016-07-20T15:19:55UTC deis-database[1]: timeout: timed out
2016-07-20T15:19:55UTC deis-database[1]:
2016-07-20T15:19:55UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:19:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:55.498308-00 pid=3939
2016-07-20T15:20:20UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:20:20UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:20:20UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:20:20UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:20:20UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:20:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:20:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:20:20UTC deis-database[1]: query_args=query_args
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:20:20UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:20:20UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:20:20UTC deis-database[1]: raise ex
2016-07-20T15:20:20UTC deis-database[1]: timeout: timed out
2016-07-20T15:20:20UTC deis-database[1]: <Greenlet at 0x7fafc4f8b0f0: <wal_e.worker.upload.WalUploader object at 0x7fafc4f5d450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:20:20UTC deis-database[1]:
2016-07-20T15:20:40UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:20:40UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:20:40UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:20:40UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:20:40UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:20:40UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:20:40UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:20:40UTC deis-database[1]: query_args=query_args
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:20:40UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:20:40UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:20:40UTC deis-database[1]: raise ex
2016-07-20T15:20:40UTC deis-database[1]: timeout: timed out
2016-07-20T15:20:40UTC deis-database[1]: <Greenlet at 0x7fafc51f3eb0: <wal_e.worker.upload.WalUploader object at 0x7fafc4f5d450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:20:40UTC deis-database[1]:
2016-07-20T15:20:50UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:20:50UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:20:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:20:50UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:20:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:20:50UTC deis-database[1]: group.join()
2016-07-20T15:20:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:20:50UTC deis-database[1]: raise val
2016-07-20T15:20:50UTC deis-database[1]: timeout: timed out
2016-07-20T15:20:50UTC deis-database[1]:
2016-07-20T15:20:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:50.311307-00 pid=1103
2016-07-20T15:20:50UTC deis-database[1]: 2016-07-20 15:20:50 UTC LOG: archive command failed with exit code 2
2016-07-20T15:20:50UTC deis-database[1]: 2016-07-20 15:20:50 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:20:51UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:20:51UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.676403-00 pid=1216
2016-07-20T15:20:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.703263-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:20:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.840990-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:20:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.978360-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.123078-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.264729-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.404008-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.545761-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.697470-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:21:27UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:21:27UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:21:27UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:21:27UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:21:27UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:21:27UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:21:27UTC deis-database[1]: data=data)
2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:21:27UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:21:27UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:21:27UTC deis-database[1]: raise ex
2016-07-20T15:21:27UTC deis-database[1]: timeout: timed out
2016-07-20T15:21:27UTC deis-database[1]:
2016-07-20T15:21:27UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:21:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:21:27.734061-00 pid=3939
2016-07-20T15:22:19UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:22:19UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:22:19UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:22:19UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:22:19UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:22:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:22:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:22:19UTC deis-database[1]: query_args=query_args
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:22:19UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:22:19UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:22:19UTC deis-database[1]: raise ex
2016-07-20T15:22:19UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:19UTC deis-database[1]: <Greenlet at 0x7ff645e26af0: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:22:19UTC deis-database[1]:
2016-07-20T15:22:27UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:22:27UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:22:27UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:22:27UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:22:27UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:22:27UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:22:27UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:22:27UTC deis-database[1]: query_args=query_args
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:22:27UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:22:27UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:22:27UTC deis-database[1]: raise ex
2016-07-20T15:22:27UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:27UTC deis-database[1]: <Greenlet at 0x7ff645bbe050: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:22:27UTC deis-database[1]:
2016-07-20T15:22:35UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:22:35UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:22:35UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:22:35UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:22:35UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:22:35UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:22:35UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:22:35UTC deis-database[1]: query_args=query_args
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:22:35UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:22:35UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:22:35UTC deis-database[1]: raise ex
2016-07-20T15:22:35UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:35UTC deis-database[1]: <Greenlet at 0x7ff645e26eb0: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:22:35UTC deis-database[1]:
2016-07-20T15:22:37UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:22:37UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:22:37UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:22:37UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:22:37UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:22:37UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:22:37UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:22:37UTC deis-database[1]: query_args=query_args
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:22:37UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:22:37UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:22:37UTC deis-database[1]: raise ex
2016-07-20T15:22:37UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:37UTC deis-database[1]: <Greenlet at 0x7ff645e26b90: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:22:37UTC deis-database[1]:
2016-07-20T15:22:39UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:22:39UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:22:39UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:22:39UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:22:39UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:22:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:22:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:22:39UTC deis-database[1]: query_args=query_args
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:22:39UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:22:39UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:22:39UTC deis-database[1]: raise ex
2016-07-20T15:22:39UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:39UTC deis-database[1]: <Greenlet at 0x7ff645bbe190: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:22:39UTC deis-database[1]:
2016-07-20T15:22:43UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:22:43UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:22:43UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:22:43UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:22:43UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:22:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:22:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:22:43UTC deis-database[1]: query_args=query_args
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:22:43UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:22:43UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:22:43UTC deis-database[1]: raise ex
2016-07-20T15:22:43UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:43UTC deis-database[1]: <Greenlet at 0x7ff645e26f50: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:22:43UTC deis-database[1]:
2016-07-20T15:22:49UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:22:49UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:22:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:22:49UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:22:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:22:49UTC deis-database[1]: group.join()
2016-07-20T15:22:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:22:49UTC deis-database[1]: raise val
2016-07-20T15:22:49UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:49UTC deis-database[1]:
2016-07-20T15:22:49UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:49.648320-00 pid=1216
2016-07-20T15:22:49UTC deis-database[1]: 2016-07-20 15:22:49 UTC LOG: archive command failed with exit code 2
2016-07-20T15:22:49UTC deis-database[1]: 2016-07-20 15:22:49 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:22:49UTC deis-database[1]: 2016-07-20 15:22:49 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:22:50UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.008261-00 pid=1381
2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.032432-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.175039-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.318255-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.462605-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.604756-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.747163-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.885062-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:22:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:22:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:22:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:51.026287-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:22:52UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:22:52UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:22:52UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:22:52UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:22:52UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:22:52UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:22:52UTC deis-database[1]: data=data)
2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:22:52UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:22:52UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:22:52UTC deis-database[1]: raise ex
2016-07-20T15:22:52UTC deis-database[1]: timeout: timed out
2016-07-20T15:22:52UTC deis-database[1]:
2016-07-20T15:22:52UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:22:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:52.777145-00 pid=3939
2016-07-20T15:24:10UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:24:10UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:24:10UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:24:10UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:24:10UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:24:10UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:24:10UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:24:10UTC deis-database[1]: query_args=query_args
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:24:10UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:24:10UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:24:10UTC deis-database[1]: raise ex
2016-07-20T15:24:10UTC deis-database[1]: timeout: timed out
2016-07-20T15:24:10UTC deis-database[1]: <Greenlet at 0x7fb32d00eeb0: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:24:10UTC deis-database[1]:
2016-07-20T15:24:11UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:24:11UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:24:11UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:24:11UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:24:11UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:24:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:24:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:24:11UTC deis-database[1]: query_args=query_args
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:24:11UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:24:11UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:24:11UTC deis-database[1]: raise ex
2016-07-20T15:24:11UTC deis-database[1]: timeout: timed out
2016-07-20T15:24:11UTC deis-database[1]: <Greenlet at 0x7fb32d00ef50: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:24:11UTC deis-database[1]:
2016-07-20T15:24:15UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:24:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:24:15UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:24:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:24:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:24:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:24:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:24:15UTC deis-database[1]: query_args=query_args
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:24:15UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:24:15UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:24:15UTC deis-database[1]: raise ex
2016-07-20T15:24:15UTC deis-database[1]: timeout: timed out
2016-07-20T15:24:15UTC deis-database[1]: <Greenlet at 0x7fb32cda6050: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:24:15UTC deis-database[1]:
2016-07-20T15:24:20UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:24:20UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:24:20UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:24:20UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:24:20UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:24:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:24:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:24:20UTC deis-database[1]: query_args=query_args
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:24:20UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:24:20UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:24:20UTC deis-database[1]: raise ex
2016-07-20T15:24:20UTC deis-database[1]: timeout: timed out
2016-07-20T15:24:20UTC deis-database[1]: <Greenlet at 0x7fb32cda60f0: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:24:20UTC deis-database[1]:
2016-07-20T15:24:21UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:24:21UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:24:21UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:24:21UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:24:21UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:24:21UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:24:21UTC deis-database[1]: data=data)
2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:24:21UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:24:21UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:24:21UTC deis-database[1]: raise ex
2016-07-20T15:24:21UTC deis-database[1]: timeout: timed out
2016-07-20T15:24:21UTC deis-database[1]:
2016-07-20T15:24:21UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:24:21UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:21.850205-00 pid=3939
2016-07-20T15:24:22UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:24:22UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:24:22UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:24:22UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:24:22UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:24:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:24:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:24:22UTC deis-database[1]: query_args=query_args
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:24:22UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:24:22UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:24:22UTC deis-database[1]: raise ex
2016-07-20T15:24:22UTC deis-database[1]: timeout: timed out
2016-07-20T15:24:22UTC deis-database[1]: <Greenlet at 0x7fb32d00eb90: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:24:22UTC deis-database[1]:
2016-07-20T15:24:40UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:24:40UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:24:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:24:40UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:24:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:24:40UTC deis-database[1]: group.join()
2016-07-20T15:24:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:24:40UTC deis-database[1]: raise val
2016-07-20T15:24:40UTC deis-database[1]: timeout: timed out
2016-07-20T15:24:40UTC deis-database[1]:
2016-07-20T15:24:40UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:40.140473-00 pid=1381
2016-07-20T15:24:40UTC deis-database[1]: 2016-07-20 15:24:40 UTC LOG: archive command failed with exit code 2
2016-07-20T15:24:40UTC deis-database[1]: 2016-07-20 15:24:40 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:24:41UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:24:41UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.495699-00 pid=1534
2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.522590-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.670634-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.814237-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.966028-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.132804-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.284598-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.435181-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.582480-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:25:54UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:25:54UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:25:54UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:25:54UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:25:54UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:25:54UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:25:54UTC deis-database[1]: data=data)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:25:54UTC deis-database[1]: raise ex
2016-07-20T15:25:54UTC deis-database[1]: timeout: timed out
2016-07-20T15:25:54UTC deis-database[1]:
2016-07-20T15:25:54UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:25:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:25:54.229196-00 pid=3939
2016-07-20T15:25:54UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:25:54UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:25:54UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:25:54UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:25:54UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:25:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:25:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:25:54UTC deis-database[1]: query_args=query_args
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:25:54UTC deis-database[1]: raise ex
2016-07-20T15:25:54UTC deis-database[1]: timeout: timed out
2016-07-20T15:25:54UTC deis-database[1]: <Greenlet at 0x7fbf6d816190: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:25:54UTC deis-database[1]:
2016-07-20T15:26:00UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:26:00UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:26:00UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:26:00UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:26:00UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:26:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:26:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:26:00UTC deis-database[1]: query_args=query_args
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:26:00UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:26:00UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:26:00UTC deis-database[1]: raise ex
2016-07-20T15:26:00UTC deis-database[1]: timeout: timed out
2016-07-20T15:26:00UTC deis-database[1]: <Greenlet at 0x7fbf6da7eb90: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:26:00UTC deis-database[1]:
2016-07-20T15:26:15UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:26:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:26:15UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:26:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:26:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:26:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:26:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:26:15UTC deis-database[1]: query_args=query_args
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:26:15UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:26:15UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:26:15UTC deis-database[1]: raise ex
2016-07-20T15:26:15UTC deis-database[1]: timeout: timed out
2016-07-20T15:26:15UTC deis-database[1]: <Greenlet at 0x7fbf6da7eeb0: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:26:15UTC deis-database[1]:
2016-07-20T15:26:16UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:26:16UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:26:16UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:26:16UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:26:16UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:26:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:26:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:26:16UTC deis-database[1]: query_args=query_args
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:26:16UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:26:16UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:26:16UTC deis-database[1]: raise ex
2016-07-20T15:26:16UTC deis-database[1]: timeout: timed out
2016-07-20T15:26:16UTC deis-database[1]: <Greenlet at 0x7fbf6d816230: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:26:16UTC deis-database[1]:
2016-07-20T15:26:24UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:26:24UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:26:24UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:26:24UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:26:24UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:26:24UTC deis-database[1]: group.join()
2016-07-20T15:26:24UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:26:24UTC deis-database[1]: raise val
2016-07-20T15:26:24UTC deis-database[1]: timeout: timed out
2016-07-20T15:26:24UTC deis-database[1]:
2016-07-20T15:26:24UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:24.464774-00 pid=1534
2016-07-20T15:26:24UTC deis-database[1]: 2016-07-20 15:26:24 UTC LOG: archive command failed with exit code 2
2016-07-20T15:26:24UTC deis-database[1]: 2016-07-20 15:26:24 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:26:25UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:26:25UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:26:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:25.820185-00 pid=1675
2016-07-20T15:26:25UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:25UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:26:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:25.848009-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:26:25UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:25UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:26:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:25.992467-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.136684-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.285525-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.437192-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.586899-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.727482-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.869152-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:27:41UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:27:41UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:27:41UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:27:41UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:27:41UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:27:41UTC deis-database[1]: query_args=query_args
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:27:41UTC deis-database[1]: raise ex
2016-07-20T15:27:41UTC deis-database[1]: timeout: timed out
2016-07-20T15:27:41UTC deis-database[1]: <Greenlet at 0x7ff25647af50: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:27:41UTC deis-database[1]:
2016-07-20T15:27:41UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:27:41UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:27:41UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:27:41UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:27:41UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:27:41UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:27:41UTC deis-database[1]: data=data)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:27:41UTC deis-database[1]: raise ex
2016-07-20T15:27:41UTC deis-database[1]: timeout: timed out
2016-07-20T15:27:41UTC deis-database[1]:
2016-07-20T15:27:41UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:27:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:27:41.630147-00 pid=3939
2016-07-20T15:27:41UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:27:41UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:27:41UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:27:41UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:27:41UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:27:41UTC deis-database[1]: query_args=query_args
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:27:41UTC deis-database[1]: raise ex
2016-07-20T15:27:41UTC deis-database[1]: timeout: timed out
2016-07-20T15:27:41UTC deis-database[1]: <Greenlet at 0x7ff25647aaf0: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:27:41UTC deis-database[1]:
2016-07-20T15:27:53UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:27:53UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:27:53UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:27:53UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:27:53UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:27:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:27:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:27:53UTC deis-database[1]: query_args=query_args
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:27:53UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:27:53UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:27:53UTC deis-database[1]: raise ex
2016-07-20T15:27:53UTC deis-database[1]: timeout: timed out
2016-07-20T15:27:53UTC deis-database[1]: <Greenlet at 0x7ff256212230: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:27:53UTC deis-database[1]:
2016-07-20T15:28:00UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:28:00UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:28:00UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:28:00UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:28:00UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:28:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:28:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:28:00UTC deis-database[1]: query_args=query_args
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:28:00UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:28:00UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:28:00UTC deis-database[1]: raise ex
2016-07-20T15:28:00UTC deis-database[1]: timeout: timed out
2016-07-20T15:28:00UTC deis-database[1]: <Greenlet at 0x7ff25647ab90: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:28:00UTC deis-database[1]:
2016-07-20T15:28:09UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:28:09UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:28:09UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:28:09UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:28:09UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:28:09UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:28:09UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:28:09UTC deis-database[1]: query_args=query_args
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:28:09UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:28:09UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:28:09UTC deis-database[1]: raise ex
2016-07-20T15:28:09UTC deis-database[1]: timeout: timed out
2016-07-20T15:28:09UTC deis-database[1]: <Greenlet at 0x7ff256212050: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:28:09UTC deis-database[1]:
2016-07-20T15:28:11UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:28:11UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:28:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:28:11UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:28:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:28:11UTC deis-database[1]: group.join()
2016-07-20T15:28:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:28:11UTC deis-database[1]: raise val
2016-07-20T15:28:11UTC deis-database[1]: timeout: timed out
2016-07-20T15:28:11UTC deis-database[1]:
2016-07-20T15:28:11UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:11.569622-00 pid=1675
2016-07-20T15:28:11UTC deis-database[1]: 2016-07-20 15:28:11 UTC LOG: archive command failed with exit code 2
2016-07-20T15:28:11UTC deis-database[1]: 2016-07-20 15:28:11 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:28:11UTC deis-database[1]: 2016-07-20 15:28:11 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:28:11UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:28:11UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:28:11UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:11.927515-00 pid=1815
2016-07-20T15:28:11UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:11UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:28:11UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:11.952909-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.092486-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.234062-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.369795-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.505606-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.643469-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.786116-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.930001-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:29:01UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:29:01UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:29:01UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:29:01UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:29:01UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:29:01UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:29:01UTC deis-database[1]: data=data)
2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:29:01UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:29:01UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:29:01UTC deis-database[1]: raise ex
2016-07-20T15:29:01UTC deis-database[1]: timeout: timed out
2016-07-20T15:29:01UTC deis-database[1]:
2016-07-20T15:29:01UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:29:01UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:01.411161-00 pid=3939
2016-07-20T15:29:02UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:29:02UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:29:02UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:29:02UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:29:02UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:29:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:29:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:29:02UTC deis-database[1]: query_args=query_args
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:29:02UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:29:02UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:29:02UTC deis-database[1]: raise ex
2016-07-20T15:29:02UTC deis-database[1]: timeout: timed out
2016-07-20T15:29:02UTC deis-database[1]: <Greenlet at 0x7fdafdfedf50: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:29:02UTC deis-database[1]:
2016-07-20T15:29:05UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:29:05UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:29:05UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:29:05UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:29:05UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:29:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:29:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:29:05UTC deis-database[1]: query_args=query_args
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:29:05UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:29:05UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:29:05UTC deis-database[1]: raise ex
2016-07-20T15:29:05UTC deis-database[1]: timeout: timed out
2016-07-20T15:29:05UTC deis-database[1]: <Greenlet at 0x7fdafdd85050: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:29:05UTC deis-database[1]:
2016-07-20T15:29:15UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:29:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:29:15UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:29:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:29:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:29:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:29:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:29:15UTC deis-database[1]: query_args=query_args
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:29:15UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:29:15UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:29:15UTC deis-database[1]: raise ex
2016-07-20T15:29:15UTC deis-database[1]: timeout: timed out
2016-07-20T15:29:15UTC deis-database[1]: <Greenlet at 0x7fdafdfedaf0: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:29:15UTC deis-database[1]:
2016-07-20T15:29:30UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:29:30UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:29:30UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:29:30UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:29:30UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:29:30UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:29:30UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:29:30UTC deis-database[1]: query_args=query_args
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:29:30UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:29:30UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:29:30UTC deis-database[1]: raise ex
2016-07-20T15:29:30UTC deis-database[1]: timeout: timed out
2016-07-20T15:29:30UTC deis-database[1]: <Greenlet at 0x7fdafdd85190: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:29:30UTC deis-database[1]:
2016-07-20T15:29:32UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:29:32UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:29:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:29:32UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:29:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:29:32UTC deis-database[1]: group.join()
2016-07-20T15:29:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:29:32UTC deis-database[1]: raise val
2016-07-20T15:29:32UTC deis-database[1]: timeout: timed out
2016-07-20T15:29:32UTC deis-database[1]:
2016-07-20T15:29:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:32.926825-00 pid=1815
2016-07-20T15:29:32UTC deis-database[1]: 2016-07-20 15:29:32 UTC LOG: archive command failed with exit code 2
2016-07-20T15:29:32UTC deis-database[1]: 2016-07-20 15:29:32 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:29:34UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:29:34UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.291263-00 pid=1943
2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.315443-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.465485-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.600021-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.748890-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.889596-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:29:35UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:35UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:29:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:35.024928-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:29:35UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:35UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:29:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:35.164896-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:29:35UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:29:35UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:29:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:35.307267-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:30:26UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:30:26UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:30:26UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:30:26UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:30:26UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:30:26UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:30:26UTC deis-database[1]: data=data)
2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:30:26UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:30:26UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:30:26UTC deis-database[1]: raise ex
2016-07-20T15:30:26UTC deis-database[1]: timeout: timed out
2016-07-20T15:30:26UTC deis-database[1]:
2016-07-20T15:30:26UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:30:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:30:26.970691-00 pid=3939
2016-07-20T15:30:36UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:30:36UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:30:36UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:30:36UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:30:36UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:30:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:30:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:30:36UTC deis-database[1]: query_args=query_args
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:30:36UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:30:36UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:30:36UTC deis-database[1]: raise ex
2016-07-20T15:30:36UTC deis-database[1]: timeout: timed out
2016-07-20T15:30:36UTC deis-database[1]: <Greenlet at 0x7f415ad70b90: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:30:36UTC deis-database[1]:
2016-07-20T15:30:44UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:30:44UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:30:44UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:30:44UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:30:44UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:30:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:30:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:30:44UTC deis-database[1]: query_args=query_args
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:30:44UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:30:44UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:30:44UTC deis-database[1]: raise ex
2016-07-20T15:30:44UTC deis-database[1]: timeout: timed out
2016-07-20T15:30:44UTC deis-database[1]: <Greenlet at 0x7f415ab08050: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:30:44UTC deis-database[1]:
2016-07-20T15:30:45UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:30:45UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:30:45UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:30:45UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:30:45UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:30:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:30:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:30:45UTC deis-database[1]: query_args=query_args
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:30:45UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:30:45UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:30:45UTC deis-database[1]: raise ex
2016-07-20T15:30:45UTC deis-database[1]: timeout: timed out
2016-07-20T15:30:45UTC deis-database[1]: <Greenlet at 0x7f415ad70f50: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:30:45UTC deis-database[1]:
2016-07-20T15:30:48UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:30:48UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:30:48UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:30:48UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:30:48UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:30:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:30:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:30:48UTC deis-database[1]: query_args=query_args
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:30:48UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:30:48UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:30:48UTC deis-database[1]: raise ex
2016-07-20T15:30:48UTC deis-database[1]: timeout: timed out
2016-07-20T15:30:48UTC deis-database[1]: <Greenlet at 0x7f415ab08230: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:30:48UTC deis-database[1]:
2016-07-20T15:31:06UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:31:06UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:31:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:31:06UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:31:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:31:06UTC deis-database[1]: group.join()
2016-07-20T15:31:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:31:06UTC deis-database[1]: raise val
2016-07-20T15:31:06UTC deis-database[1]: timeout: timed out
2016-07-20T15:31:06UTC deis-database[1]:
2016-07-20T15:31:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:06.550339-00 pid=1943
2016-07-20T15:31:06UTC deis-database[1]: 2016-07-20 15:31:06 UTC LOG: archive command failed with exit code 2
2016-07-20T15:31:06UTC deis-database[1]: 2016-07-20 15:31:06 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:31:07UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:31:07UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:31:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:07.914574-00 pid=2071
2016-07-20T15:31:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:07UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:31:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:07.940828-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.095672-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.245805-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.385529-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.527124-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.668800-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.812164-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.960555-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:31:56UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:31:56UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:31:56UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:31:56UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:31:56UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:31:56UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:31:56UTC deis-database[1]: data=data)
2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:31:56UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:31:56UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:31:56UTC deis-database[1]: raise ex
2016-07-20T15:31:56UTC deis-database[1]: timeout: timed out
2016-07-20T15:31:56UTC deis-database[1]:
2016-07-20T15:31:56UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:31:56UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:56.321967-00 pid=3939
2016-07-20T15:32:16UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:32:16UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:32:16UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:32:16UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:32:16UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:32:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:32:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:32:16UTC deis-database[1]: query_args=query_args
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:32:16UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:32:16UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:32:16UTC deis-database[1]: raise ex
2016-07-20T15:32:16UTC deis-database[1]: timeout: timed out
2016-07-20T15:32:16UTC deis-database[1]: <Greenlet at 0x7f4654d27050: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:32:16UTC deis-database[1]:
2016-07-20T15:32:19UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:32:19UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:32:19UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:32:19UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:32:19UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:32:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:32:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:32:19UTC deis-database[1]: query_args=query_args
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:32:19UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:32:19UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:32:19UTC deis-database[1]: raise ex
2016-07-20T15:32:19UTC deis-database[1]: timeout: timed out
2016-07-20T15:32:19UTC deis-database[1]: <Greenlet at 0x7f4654d27190: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:32:19UTC deis-database[1]:
2016-07-20T15:32:25UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:32:25UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:32:25UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:32:25UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:32:25UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:32:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:32:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:32:25UTC deis-database[1]: query_args=query_args
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:32:25UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:32:25UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:32:25UTC deis-database[1]: raise ex
2016-07-20T15:32:25UTC deis-database[1]: timeout: timed out
2016-07-20T15:32:25UTC deis-database[1]: <Greenlet at 0x7f4654d27230: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:32:25UTC deis-database[1]:
2016-07-20T15:32:28UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:32:28UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:32:28UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:32:28UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:32:28UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:32:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:32:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:32:28UTC deis-database[1]: query_args=query_args
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:32:28UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:32:28UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:32:28UTC deis-database[1]: raise ex
2016-07-20T15:32:28UTC deis-database[1]: timeout: timed out
2016-07-20T15:32:28UTC deis-database[1]: <Greenlet at 0x7f4654f8feb0: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:32:28UTC deis-database[1]:
2016-07-20T15:32:29UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:32:29UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:32:29UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:32:29UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:32:29UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:32:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:32:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:32:29UTC deis-database[1]: query_args=query_args
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:32:29UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:32:29UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:32:29UTC deis-database[1]: raise ex
2016-07-20T15:32:29UTC deis-database[1]: timeout: timed out
2016-07-20T15:32:29UTC deis-database[1]: <Greenlet at 0x7f4654f8faf0: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:32:29UTC deis-database[1]:
2016-07-20T15:32:39UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:32:39UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:32:39UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:32:39UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:32:39UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:32:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:32:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:32:39UTC deis-database[1]: query_args=query_args
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:32:39UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:32:39UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:32:39UTC deis-database[1]: raise ex
2016-07-20T15:32:39UTC deis-database[1]: timeout: timed out
2016-07-20T15:32:39UTC deis-database[1]: <Greenlet at 0x7f4654f8ff50: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:32:39UTC deis-database[1]:
2016-07-20T15:32:46UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:32:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:32:46UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:32:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:32:46UTC deis-database[1]: group.join()
2016-07-20T15:32:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:32:46UTC deis-database[1]: raise val
2016-07-20T15:32:46UTC deis-database[1]: timeout: timed out
2016-07-20T15:32:46UTC deis-database[1]:
2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.105032-00 pid=2071
2016-07-20T15:32:46UTC deis-database[1]: 2016-07-20 15:32:46 UTC LOG: archive command failed with exit code 2
2016-07-20T15:32:46UTC deis-database[1]: 2016-07-20 15:32:46 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:32:46UTC deis-database[1]: 2016-07-20 15:32:46 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:32:46UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:32:46UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.455597-00 pid=2211
2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.480692-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.622949-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.760061-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.904812-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.042464-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.182300-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.317402-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.454389-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:33:56UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:33:56UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:33:56UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:33:56UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:33:56UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:33:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:33:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:33:56UTC deis-database[1]: query_args=query_args
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:33:56UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:33:56UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:33:56UTC deis-database[1]: raise ex
2016-07-20T15:33:56UTC deis-database[1]: timeout: timed out
2016-07-20T15:33:56UTC deis-database[1]: <Greenlet at 0x7f63e95ae190: <wal_e.worker.upload.WalUploader object at 0x7f63e9580450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:33:56UTC deis-database[1]:
2016-07-20T15:33:57UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:33:57UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:33:57UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:33:57UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:33:57UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:33:57UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:33:57UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:33:57UTC deis-database[1]: query_args=query_args
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:33:57UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:33:57UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:33:57UTC deis-database[1]: raise ex
2016-07-20T15:33:57UTC deis-database[1]: timeout: timed out
2016-07-20T15:33:57UTC deis-database[1]: <Greenlet at 0x7f63e9816f50: <wal_e.worker.upload.WalUploader object at 0x7f63e9580450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:33:57UTC deis-database[1]:
2016-07-20T15:33:59UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:33:59UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:33:59UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:33:59UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:33:59UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:33:59UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:33:59UTC deis-database[1]: data=data)
2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:33:59UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:33:59UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:33:59UTC deis-database[1]: raise ex
2016-07-20T15:33:59UTC deis-database[1]: timeout: timed out
2016-07-20T15:33:59UTC deis-database[1]:
2016-07-20T15:33:59UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:33:59UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:33:59.245414-00 pid=3939
2016-07-20T15:34:26UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:34:26UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:34:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:34:26UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:34:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:34:26UTC deis-database[1]: group.join()
2016-07-20T15:34:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:34:26UTC deis-database[1]: raise val
2016-07-20T15:34:26UTC deis-database[1]: timeout: timed out
2016-07-20T15:34:26UTC deis-database[1]:
2016-07-20T15:34:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:26.636581-00 pid=2211
2016-07-20T15:34:26UTC deis-database[1]: 2016-07-20 15:34:26 UTC LOG: archive command failed with exit code 2
2016-07-20T15:34:26UTC deis-database[1]: 2016-07-20 15:34:26 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:34:27UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:34:27UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:34:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:27.985965-00 pid=2352
2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.011209-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.157331-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.300840-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.443553-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.586209-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.722892-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.862425-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:34:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:34:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:34:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:29.013908-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:35:00UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:35:00UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:35:00UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:35:00UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:35:00UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:35:00UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:35:00UTC deis-database[1]: data=data)
2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:35:00UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:35:00UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:35:00UTC deis-database[1]: raise ex
2016-07-20T15:35:00UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:00UTC deis-database[1]:
2016-07-20T15:35:00UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:35:00UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:00.068286-00 pid=3939
2016-07-20T15:35:23UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:35:23UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:35:23UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:35:23UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:35:23UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:35:23UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:35:23UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:35:23UTC deis-database[1]: query_args=query_args
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:35:23UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:35:23UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:35:23UTC deis-database[1]: raise ex
2016-07-20T15:35:23UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:23UTC deis-database[1]: <Greenlet at 0x7fb203dfdb90: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:35:23UTC deis-database[1]:
2016-07-20T15:35:36UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:35:36UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:35:36UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:35:36UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:35:36UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:35:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:35:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:35:36UTC deis-database[1]: query_args=query_args
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:35:36UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:35:36UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:35:36UTC deis-database[1]: raise ex
2016-07-20T15:35:36UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:36UTC deis-database[1]: <Greenlet at 0x7fb203b95190: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:35:36UTC deis-database[1]:
2016-07-20T15:35:38UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:35:38UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:35:38UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:35:38UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:35:38UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:35:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:35:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:35:38UTC deis-database[1]: query_args=query_args
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:35:38UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:35:38UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:35:38UTC deis-database[1]: raise ex
2016-07-20T15:35:38UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:38UTC deis-database[1]: <Greenlet at 0x7fb203dfdaf0: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:35:38UTC deis-database[1]:
2016-07-20T15:35:44UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:35:44UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:35:44UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:35:44UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:35:44UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:35:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:35:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:35:44UTC deis-database[1]: query_args=query_args
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:35:44UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:35:44UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:35:44UTC deis-database[1]: raise ex
2016-07-20T15:35:44UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:44UTC deis-database[1]: <Greenlet at 0x7fb203b95230: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:35:44UTC deis-database[1]:
2016-07-20T15:35:49UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:35:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:35:49UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:35:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:35:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:35:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:35:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:35:49UTC deis-database[1]: query_args=query_args
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:35:49UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:35:49UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:35:49UTC deis-database[1]: raise ex
2016-07-20T15:35:49UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:49UTC deis-database[1]: <Greenlet at 0x7fb203b950f0: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:35:49UTC deis-database[1]:
2016-07-20T15:35:52UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:35:52UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:35:52UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:35:52UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:35:52UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:35:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:35:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:35:52UTC deis-database[1]: query_args=query_args
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:35:52UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:35:52UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:35:52UTC deis-database[1]: raise ex
2016-07-20T15:35:52UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:52UTC deis-database[1]: <Greenlet at 0x7fb203dfdeb0: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:35:52UTC deis-database[1]:
2016-07-20T15:35:53UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:35:53UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:35:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:35:53UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:35:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:35:53UTC deis-database[1]: group.join()
2016-07-20T15:35:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:35:53UTC deis-database[1]: raise val
2016-07-20T15:35:53UTC deis-database[1]: timeout: timed out
2016-07-20T15:35:53UTC deis-database[1]:
2016-07-20T15:35:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:53.620273-00 pid=2352
2016-07-20T15:35:53UTC deis-database[1]: 2016-07-20 15:35:53 UTC LOG: archive command failed with exit code 2
2016-07-20T15:35:53UTC deis-database[1]: 2016-07-20 15:35:53 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:35:54UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:35:54UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:35:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:54.964755-00 pid=2468
2016-07-20T15:35:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:35:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:54.992378-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.140222-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.284697-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.451196-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.598838-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.743298-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.885363-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:35:56UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:35:56UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:35:56UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:56.026037-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:36:13UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:36:13UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:36:13UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:36:13UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:36:13UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:36:13UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:36:13UTC deis-database[1]: data=data)
2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:36:13UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:36:13UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:36:13UTC deis-database[1]: raise ex
2016-07-20T15:36:13UTC deis-database[1]: timeout: timed out
2016-07-20T15:36:13UTC deis-database[1]:
2016-07-20T15:36:13UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:36:13UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:36:13.500027-00 pid=3939
2016-07-20T15:36:50UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:36:50UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:36:50UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:36:50UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:36:50UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:36:50UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:36:50UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:36:50UTC deis-database[1]: query_args=query_args
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:36:50UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:36:50UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:36:50UTC deis-database[1]: raise ex
2016-07-20T15:36:50UTC deis-database[1]: timeout: timed out
2016-07-20T15:36:50UTC deis-database[1]: <Greenlet at 0x7f587d0bf0f0: <wal_e.worker.upload.WalUploader object at 0x7f587d091450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:36:50UTC deis-database[1]:
2016-07-20T15:36:55UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:36:55UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:36:55UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:36:55UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:36:55UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:36:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:36:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:36:55UTC deis-database[1]: query_args=query_args
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:36:55UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:36:55UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:36:55UTC deis-database[1]: raise ex
2016-07-20T15:36:55UTC deis-database[1]: timeout: timed out
2016-07-20T15:36:55UTC deis-database[1]: <Greenlet at 0x7f587d327eb0: <wal_e.worker.upload.WalUploader object at 0x7f587d091450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:36:55UTC deis-database[1]:
2016-07-20T15:37:04UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:37:04UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:37:04UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:37:04UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:37:04UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:37:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:37:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:37:04UTC deis-database[1]: query_args=query_args
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:37:04UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:37:04UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:37:04UTC deis-database[1]: raise ex
2016-07-20T15:37:04UTC deis-database[1]: timeout: timed out
2016-07-20T15:37:04UTC deis-database[1]: <Greenlet at 0x7f587d0bf190: <wal_e.worker.upload.WalUploader object at 0x7f587d091450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:37:04UTC deis-database[1]:
2016-07-20T15:37:20UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:37:20UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:37:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:37:20UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:37:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:37:20UTC deis-database[1]: group.join()
2016-07-20T15:37:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:37:20UTC deis-database[1]: raise val
2016-07-20T15:37:20UTC deis-database[1]: timeout: timed out
2016-07-20T15:37:20UTC deis-database[1]:
2016-07-20T15:37:20UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:20.613984-00 pid=2468
2016-07-20T15:37:20UTC deis-database[1]: 2016-07-20 15:37:20 UTC LOG: archive command failed with exit code 2
2016-07-20T15:37:20UTC deis-database[1]: 2016-07-20 15:37:20 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:37:20UTC deis-database[1]: 2016-07-20 15:37:20 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:37:38UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:37:38UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.173720-00 pid=2608
2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.198996-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.341395-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.492550-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.636081-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.781720-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.922592-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:37:39UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:39UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:37:39UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:39.099734-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:37:39UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:37:39UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:37:39UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:39.243137-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:38:05UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:38:05UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:38:05UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:38:05UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:38:05UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:38:05UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:38:05UTC deis-database[1]: data=data)
2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:38:05UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:38:05UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:38:05UTC deis-database[1]: raise ex
2016-07-20T15:38:05UTC deis-database[1]: timeout: timed out
2016-07-20T15:38:05UTC deis-database[1]:
2016-07-20T15:38:05UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:38:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:38:05.631248-00 pid=3939
2016-07-20T15:38:44UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:38:44UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:38:44UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:38:44UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:38:44UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:38:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:38:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:38:44UTC deis-database[1]: query_args=query_args
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:38:44UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:38:44UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:38:44UTC deis-database[1]: raise ex
2016-07-20T15:38:44UTC deis-database[1]: timeout: timed out
2016-07-20T15:38:44UTC deis-database[1]: <Greenlet at 0x7f855c806230: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:38:44UTC deis-database[1]:
2016-07-20T15:38:53UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:38:53UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:38:53UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:38:53UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:38:53UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:38:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:38:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:38:53UTC deis-database[1]: query_args=query_args
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:38:53UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:38:53UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:38:53UTC deis-database[1]: raise ex
2016-07-20T15:38:53UTC deis-database[1]: timeout: timed out
2016-07-20T15:38:53UTC deis-database[1]: <Greenlet at 0x7f855c806190: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:38:53UTC deis-database[1]:
2016-07-20T15:39:02UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:39:02UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:39:02UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:39:02UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:39:02UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:39:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:39:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:39:02UTC deis-database[1]: query_args=query_args
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:39:02UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:39:02UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:39:02UTC deis-database[1]: raise ex
2016-07-20T15:39:02UTC deis-database[1]: timeout: timed out
2016-07-20T15:39:02UTC deis-database[1]: <Greenlet at 0x7f855ca6eaf0: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:39:02UTC deis-database[1]:
2016-07-20T15:39:11UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:39:11UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:39:11UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:39:11UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:39:11UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:39:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:39:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:39:11UTC deis-database[1]: query_args=query_args
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:39:11UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:39:11UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:39:11UTC deis-database[1]: raise ex
2016-07-20T15:39:11UTC deis-database[1]: timeout: timed out
2016-07-20T15:39:11UTC deis-database[1]: <Greenlet at 0x7f855ca6eeb0: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:39:11UTC deis-database[1]:
2016-07-20T15:39:14UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:39:14UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:39:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:39:14UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:39:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:39:14UTC deis-database[1]: group.join()
2016-07-20T15:39:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:39:14UTC deis-database[1]: raise val
2016-07-20T15:39:14UTC deis-database[1]: timeout: timed out
2016-07-20T15:39:14UTC deis-database[1]:
2016-07-20T15:39:14UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:14.834065-00 pid=2608
2016-07-20T15:39:14UTC deis-database[1]: 2016-07-20 15:39:14 UTC LOG: archive command failed with exit code 2
2016-07-20T15:39:14UTC deis-database[1]: 2016-07-20 15:39:14 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:39:16UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:39:16UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.174055-00 pid=2748
2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.201868-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.347398-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.490148-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.640133-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.785058-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.921151-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:39:17UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:17UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:39:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:17.070456-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:39:17UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:39:17UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:39:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:17.208249-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:40:10UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:40:10UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:40:10UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:40:10UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:40:10UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:40:10UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:40:10UTC deis-database[1]: data=data)
2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:40:10UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:40:10UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:40:10UTC deis-database[1]: raise ex
2016-07-20T15:40:10UTC deis-database[1]: timeout: timed out
2016-07-20T15:40:10UTC deis-database[1]:
2016-07-20T15:40:10UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:40:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:10.985192-00 pid=3939
2016-07-20T15:40:26UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:40:26UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:40:26UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:40:26UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:40:26UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:40:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:40:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:40:26UTC deis-database[1]: query_args=query_args
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:40:26UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:40:26UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:40:26UTC deis-database[1]: raise ex
2016-07-20T15:40:26UTC deis-database[1]: timeout: timed out
2016-07-20T15:40:26UTC deis-database[1]: <Greenlet at 0x7fc45483f0f0: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:40:26UTC deis-database[1]:
2016-07-20T15:40:28UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:40:28UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:40:28UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:40:28UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:40:28UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:40:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:40:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:40:28UTC deis-database[1]: query_args=query_args
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:40:28UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:40:28UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:40:28UTC deis-database[1]: raise ex
2016-07-20T15:40:28UTC deis-database[1]: timeout: timed out
2016-07-20T15:40:28UTC deis-database[1]: <Greenlet at 0x7fc45483f190: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:40:28UTC deis-database[1]:
2016-07-20T15:40:32UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:40:32UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:40:32UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:40:32UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:40:32UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:40:32UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:40:32UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:40:32UTC deis-database[1]: query_args=query_args
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:40:32UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:40:32UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:40:32UTC deis-database[1]: raise ex
2016-07-20T15:40:32UTC deis-database[1]: timeout: timed out
2016-07-20T15:40:32UTC deis-database[1]: <Greenlet at 0x7fc45483f230: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:40:32UTC deis-database[1]:
2016-07-20T15:40:34UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:40:34UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:40:34UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:40:34UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:40:34UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:40:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:40:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:40:34UTC deis-database[1]: query_args=query_args
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:40:34UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:40:34UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:40:34UTC deis-database[1]: raise ex
2016-07-20T15:40:34UTC deis-database[1]: timeout: timed out
2016-07-20T15:40:34UTC deis-database[1]: <Greenlet at 0x7fc45483f050: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:40:34UTC deis-database[1]:
2016-07-20T15:40:45UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:40:45UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:40:45UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:40:45UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:40:45UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:40:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:40:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:40:45UTC deis-database[1]: query_args=query_args
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:40:45UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:40:45UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:40:45UTC deis-database[1]: raise ex
2016-07-20T15:40:45UTC deis-database[1]: timeout: timed out
2016-07-20T15:40:45UTC deis-database[1]: <Greenlet at 0x7fc454aa7f50: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:40:45UTC deis-database[1]:
2016-07-20T15:40:56UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:40:56UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:40:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:40:56UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:40:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:40:56UTC deis-database[1]: group.join()
2016-07-20T15:40:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:40:56UTC deis-database[1]: raise val
2016-07-20T15:40:56UTC deis-database[1]: timeout: timed out
2016-07-20T15:40:56UTC deis-database[1]:
2016-07-20T15:40:56UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:56.734755-00 pid=2748
2016-07-20T15:40:56UTC deis-database[1]: 2016-07-20 15:40:56 UTC LOG: archive command failed with exit code 2
2016-07-20T15:40:56UTC deis-database[1]: 2016-07-20 15:40:56 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:40:58UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.074102-00 pid=2888
2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.099153-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.238059-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.373391-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.511076-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.653949-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.788161-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.927898-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:40:59UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:40:59UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:40:59UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:59.067591-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:41:47UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:41:47UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:41:47UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:41:47UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:41:47UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:41:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:41:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:41:47UTC deis-database[1]: query_args=query_args
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:41:47UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:41:47UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:41:47UTC deis-database[1]: raise ex
2016-07-20T15:41:47UTC deis-database[1]: timeout: timed out
2016-07-20T15:41:47UTC deis-database[1]: <Greenlet at 0x7f7ed76bab90: <wal_e.worker.upload.WalUploader object at 0x7f7ed7424450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:41:47UTC deis-database[1]:
2016-07-20T15:42:05UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:42:05UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:42:05UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:42:05UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:42:05UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:42:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:42:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:42:05UTC deis-database[1]: query_args=query_args
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:42:05UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:42:05UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:42:05UTC deis-database[1]: raise ex
2016-07-20T15:42:05UTC deis-database[1]: timeout: timed out
2016-07-20T15:42:05UTC deis-database[1]: <Greenlet at 0x7f7ed76baeb0: <wal_e.worker.upload.WalUploader object at 0x7f7ed7424450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:42:05UTC deis-database[1]:
2016-07-20T15:42:17UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:42:17UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:42:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:42:17UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:42:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:42:17UTC deis-database[1]: group.join()
2016-07-20T15:42:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:42:17UTC deis-database[1]: raise val
2016-07-20T15:42:17UTC deis-database[1]: timeout: timed out
2016-07-20T15:42:17UTC deis-database[1]:
2016-07-20T15:42:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:17.378498-00 pid=2888
2016-07-20T15:42:17UTC deis-database[1]: 2016-07-20 15:42:17 UTC LOG: archive command failed with exit code 2
2016-07-20T15:42:17UTC deis-database[1]: 2016-07-20 15:42:17 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:42:17UTC deis-database[1]: 2016-07-20 15:42:17 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:42:25UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:42:25UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:42:25UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:42:25UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:42:25UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:42:25UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:42:25UTC deis-database[1]: data=data)
2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:42:25UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:42:25UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:42:25UTC deis-database[1]: raise ex
2016-07-20T15:42:25UTC deis-database[1]: timeout: timed out
2016-07-20T15:42:25UTC deis-database[1]:
2016-07-20T15:42:25UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:42:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:25.035232-00 pid=3939
2016-07-20T15:42:37UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:42:37UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.616912-00 pid=3031
2016-07-20T15:42:37UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:37UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.643983-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:42:37UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:37UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.788249-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:42:37UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:37UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.924596-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.076934-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.217048-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.353703-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.503778-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.646837-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:43:49UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:43:49UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:43:49UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:43:49UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:43:49UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:43:49UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:43:49UTC deis-database[1]: data=data)
2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:43:49UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:43:49UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:43:49UTC deis-database[1]: raise ex
2016-07-20T15:43:49UTC deis-database[1]: timeout: timed out
2016-07-20T15:43:49UTC deis-database[1]:
2016-07-20T15:43:49UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:43:49UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:43:49.953199-00 pid=3939
2016-07-20T15:44:00UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:44:00UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:44:00UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:44:00UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:44:00UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:44:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:44:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:44:00UTC deis-database[1]: query_args=query_args
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:44:00UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:44:00UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:44:00UTC deis-database[1]: raise ex
2016-07-20T15:44:00UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:00UTC deis-database[1]: <Greenlet at 0x7fb46d3ef0f0: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:44:00UTC deis-database[1]:
2016-07-20T15:44:06UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:44:06UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:44:06UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:44:06UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:44:06UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:44:06UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:44:06UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:44:06UTC deis-database[1]: query_args=query_args
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:44:06UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:44:06UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:44:06UTC deis-database[1]: raise ex
2016-07-20T15:44:06UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:06UTC deis-database[1]: <Greenlet at 0x7fb46d657af0: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:44:06UTC deis-database[1]:
2016-07-20T15:44:11UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:44:11UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:44:11UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:44:11UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:44:11UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:44:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:44:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:44:11UTC deis-database[1]: query_args=query_args
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:44:11UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:44:11UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:44:11UTC deis-database[1]: raise ex
2016-07-20T15:44:11UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:11UTC deis-database[1]: <Greenlet at 0x7fb46d3ef050: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:44:11UTC deis-database[1]:
2016-07-20T15:44:19UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:44:19UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:44:19UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:44:19UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:44:19UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:44:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:44:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:44:19UTC deis-database[1]: query_args=query_args
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:44:19UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:44:19UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:44:19UTC deis-database[1]: raise ex
2016-07-20T15:44:19UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:19UTC deis-database[1]: <Greenlet at 0x7fb46d657eb0: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:44:19UTC deis-database[1]:
2016-07-20T15:44:26UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:44:26UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:44:26UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:44:26UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:44:26UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:44:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:44:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:44:26UTC deis-database[1]: query_args=query_args
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:44:26UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:44:26UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:44:26UTC deis-database[1]: raise ex
2016-07-20T15:44:26UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:26UTC deis-database[1]: <Greenlet at 0x7fb46d3ef230: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:44:26UTC deis-database[1]:
2016-07-20T15:44:29UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:44:29UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:44:29UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:44:29UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:44:29UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:44:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:44:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:44:29UTC deis-database[1]: query_args=query_args
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:44:29UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:44:29UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:44:29UTC deis-database[1]: raise ex
2016-07-20T15:44:29UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:29UTC deis-database[1]: <Greenlet at 0x7fb46d3ef190: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:44:29UTC deis-database[1]:
2016-07-20T15:44:30UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:44:30UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:44:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:44:30UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:44:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:44:30UTC deis-database[1]: group.join()
2016-07-20T15:44:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:44:30UTC deis-database[1]: raise val
2016-07-20T15:44:30UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:30UTC deis-database[1]:
2016-07-20T15:44:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:30.496921-00 pid=3031
2016-07-20T15:44:30UTC deis-database[1]: 2016-07-20 15:44:30 UTC LOG: archive command failed with exit code 2
2016-07-20T15:44:30UTC deis-database[1]: 2016-07-20 15:44:30 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:44:31UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:44:31UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:44:31UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:31.846497-00 pid=3185
2016-07-20T15:44:31UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:31UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:44:31UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:31.872108-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.017260-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.170971-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.307232-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.442922-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.580884-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.724810-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.863568-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:44:43UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:44:43UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:44:43UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:44:43UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:44:43UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:44:43UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:44:43UTC deis-database[1]: data=data)
2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:44:43UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:44:43UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:44:43UTC deis-database[1]: raise ex
2016-07-20T15:44:43UTC deis-database[1]: timeout: timed out
2016-07-20T15:44:43UTC deis-database[1]:
2016-07-20T15:44:43UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:44:43UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:43.478962-00 pid=3939
2016-07-20T15:45:46UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:45:46UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:45:46UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:45:46UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:45:46UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:45:46UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:45:46UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:45:46UTC deis-database[1]: query_args=query_args
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:45:46UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:45:46UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:45:46UTC deis-database[1]: raise ex
2016-07-20T15:45:46UTC deis-database[1]: timeout: timed out
2016-07-20T15:45:46UTC deis-database[1]: <Greenlet at 0x7ff578292230: <wal_e.worker.upload.WalUploader object at 0x7ff578264450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:45:46UTC deis-database[1]:
2016-07-20T15:45:52UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:45:52UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:45:52UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:45:52UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:45:52UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:45:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:45:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:45:52UTC deis-database[1]: query_args=query_args
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:45:52UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:45:52UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:45:52UTC deis-database[1]: raise ex
2016-07-20T15:45:52UTC deis-database[1]: timeout: timed out
2016-07-20T15:45:52UTC deis-database[1]: <Greenlet at 0x7ff5784faeb0: <wal_e.worker.upload.WalUploader object at 0x7ff578264450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:45:52UTC deis-database[1]:
2016-07-20T15:46:09UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:46:09UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:46:09UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:46:09UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:46:09UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:46:09UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:46:09UTC deis-database[1]: data=data)
2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:46:09UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:46:09UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:46:09UTC deis-database[1]: raise ex
2016-07-20T15:46:09UTC deis-database[1]: timeout: timed out
2016-07-20T15:46:09UTC deis-database[1]:
2016-07-20T15:46:09UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:46:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:09.137421-00 pid=3939
2016-07-20T15:46:16UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:46:16UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:46:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:46:16UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:46:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:46:16UTC deis-database[1]: group.join()
2016-07-20T15:46:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:46:16UTC deis-database[1]: raise val
2016-07-20T15:46:16UTC deis-database[1]: timeout: timed out
2016-07-20T15:46:16UTC deis-database[1]:
2016-07-20T15:46:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:16.763932-00 pid=3185
2016-07-20T15:46:16UTC deis-database[1]: 2016-07-20 15:46:16 UTC LOG: archive command failed with exit code 2
2016-07-20T15:46:16UTC deis-database[1]: 2016-07-20 15:46:16 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:46:18UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:46:18UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.131062-00 pid=3336
2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.157612-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.298275-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.440470-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.605423-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.748424-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.895359-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:46:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:46:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:19.072569-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:46:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:46:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:46:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:19.215672-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:47:28UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:47:28UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:47:28UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:47:28UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:47:28UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:47:28UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:47:28UTC deis-database[1]: data=data)
2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:47:28UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:47:28UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:47:28UTC deis-database[1]: raise ex
2016-07-20T15:47:28UTC deis-database[1]: timeout: timed out
2016-07-20T15:47:28UTC deis-database[1]:
2016-07-20T15:47:28UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:47:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:47:28.152232-00 pid=3939
2016-07-20T15:47:34UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:47:34UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:47:34UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:47:34UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:47:34UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:47:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:47:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:47:34UTC deis-database[1]: query_args=query_args
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:47:34UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:47:34UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:47:34UTC deis-database[1]: raise ex
2016-07-20T15:47:34UTC deis-database[1]: timeout: timed out
2016-07-20T15:47:34UTC deis-database[1]: <Greenlet at 0x7fa6a44bf0f0: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:47:34UTC deis-database[1]:
2016-07-20T15:47:47UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:47:47UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:47:47UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:47:47UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:47:47UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:47:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:47:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:47:47UTC deis-database[1]: query_args=query_args
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:47:47UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:47:47UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:47:47UTC deis-database[1]: raise ex
2016-07-20T15:47:47UTC deis-database[1]: timeout: timed out
2016-07-20T15:47:47UTC deis-database[1]: <Greenlet at 0x7fa6a4727b90: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:47:47UTC deis-database[1]:
2016-07-20T15:47:49UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:47:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:47:49UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:47:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:47:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:47:49UTC deis-database[1]: query_args=query_args
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:47:49UTC deis-database[1]: raise ex
2016-07-20T15:47:49UTC deis-database[1]: timeout: timed out
2016-07-20T15:47:49UTC deis-database[1]: <Greenlet at 0x7fa6a44bf190: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:47:49UTC deis-database[1]:
2016-07-20T15:47:49UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:47:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:47:49UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:47:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:47:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:47:49UTC deis-database[1]: query_args=query_args
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:47:49UTC deis-database[1]: raise ex
2016-07-20T15:47:49UTC deis-database[1]: timeout: timed out
2016-07-20T15:47:49UTC deis-database[1]: <Greenlet at 0x7fa6a4727af0: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:47:49UTC deis-database[1]:
2016-07-20T15:47:53UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:47:53UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:47:53UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:47:53UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:47:53UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:47:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:47:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:47:53UTC deis-database[1]: query_args=query_args
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:47:53UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:47:53UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:47:53UTC deis-database[1]: raise ex
2016-07-20T15:47:53UTC deis-database[1]: timeout: timed out
2016-07-20T15:47:53UTC deis-database[1]: <Greenlet at 0x7fa6a44bf050: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:47:53UTC deis-database[1]:
2016-07-20T15:47:55UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:47:55UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:47:55UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:47:55UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:47:55UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:47:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:47:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:47:55UTC deis-database[1]: query_args=query_args
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:47:55UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:47:55UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:47:55UTC deis-database[1]: raise ex
2016-07-20T15:47:55UTC deis-database[1]: timeout: timed out
2016-07-20T15:47:55UTC deis-database[1]: <Greenlet at 0x7fa6a44bf230: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:47:55UTC deis-database[1]:
2016-07-20T15:48:04UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:48:04UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:48:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:48:04UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:48:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:48:04UTC deis-database[1]: group.join()
2016-07-20T15:48:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:48:04UTC deis-database[1]: raise val
2016-07-20T15:48:04UTC deis-database[1]: timeout: timed out
2016-07-20T15:48:04UTC deis-database[1]:
2016-07-20T15:48:04UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:04.872569-00 pid=3336
2016-07-20T15:48:04UTC deis-database[1]: 2016-07-20 15:48:04 UTC LOG: archive command failed with exit code 2
2016-07-20T15:48:04UTC deis-database[1]: 2016-07-20 15:48:04 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:48:04UTC deis-database[1]: 2016-07-20 15:48:04 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:48:05UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:48:05UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.225375-00 pid=3477
2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.250716-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.393853-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.542910-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.687694-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.823241-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.960127-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:48:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:06UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:48:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:06.099615-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:48:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:48:06UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:48:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:06.249689-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:48:31UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:48:31UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:48:31UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:48:31UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:48:31UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:48:31UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:48:31UTC deis-database[1]: data=data)
2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:48:31UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:48:31UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:48:31UTC deis-database[1]: raise ex
2016-07-20T15:48:31UTC deis-database[1]: timeout: timed out
2016-07-20T15:48:31UTC deis-database[1]:
2016-07-20T15:48:31UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:48:31UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:31.333767-00 pid=3939
2016-07-20T15:49:22UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:49:22UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:49:22UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:49:22UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:49:22UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:49:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:49:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:49:22UTC deis-database[1]: query_args=query_args
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:49:22UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:49:22UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:49:22UTC deis-database[1]: raise ex
2016-07-20T15:49:22UTC deis-database[1]: timeout: timed out
2016-07-20T15:49:22UTC deis-database[1]: <Greenlet at 0x7f5a8b709230: <wal_e.worker.upload.WalUploader object at 0x7f5a8b6db450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:49:22UTC deis-database[1]:
2016-07-20T15:49:51UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:49:51UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:49:51UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:49:51UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:49:51UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:49:51UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:49:51UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:49:51UTC deis-database[1]: query_args=query_args
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:49:51UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:49:51UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:49:51UTC deis-database[1]: raise ex
2016-07-20T15:49:51UTC deis-database[1]: timeout: timed out
2016-07-20T15:49:51UTC deis-database[1]: <Greenlet at 0x7f5a8b971b90: <wal_e.worker.upload.WalUploader object at 0x7f5a8b6db450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:49:51UTC deis-database[1]:
2016-07-20T15:49:52UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:49:52UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:49:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:49:52UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:49:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:49:52UTC deis-database[1]: group.join()
2016-07-20T15:49:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:49:52UTC deis-database[1]: raise val
2016-07-20T15:49:52UTC deis-database[1]: timeout: timed out
2016-07-20T15:49:52UTC deis-database[1]:
2016-07-20T15:49:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:52.259243-00 pid=3477
2016-07-20T15:49:52UTC deis-database[1]: 2016-07-20 15:49:52 UTC LOG: archive command failed with exit code 2
2016-07-20T15:49:52UTC deis-database[1]: 2016-07-20 15:49:52 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:49:53UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:49:53UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.614013-00 pid=3630
2016-07-20T15:49:53UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:53UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.639713-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:49:53UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:53UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.781682-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:49:53UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:53UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.927603-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.069398-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.222308-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.369621-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.519578-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.671798-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:50:27UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:50:27UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:50:27UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:50:27UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:50:27UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:50:27UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:50:27UTC deis-database[1]: data=data)
2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:50:27UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:50:27UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:50:27UTC deis-database[1]: raise ex
2016-07-20T15:50:27UTC deis-database[1]: timeout: timed out
2016-07-20T15:50:27UTC deis-database[1]:
2016-07-20T15:50:27UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:50:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:50:27.925245-00 pid=3939
2016-07-20T15:50:56UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:50:56UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:50:56UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:50:56UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:50:56UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:50:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:50:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:50:56UTC deis-database[1]: query_args=query_args
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:50:56UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:50:56UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:50:56UTC deis-database[1]: raise ex
2016-07-20T15:50:56UTC deis-database[1]: timeout: timed out
2016-07-20T15:50:56UTC deis-database[1]: <Greenlet at 0x7f9bfb6ebaf0: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:50:56UTC deis-database[1]:
2016-07-20T15:51:13UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:51:13UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:51:13UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:51:13UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:51:13UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:51:13UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:51:13UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:51:13UTC deis-database[1]: query_args=query_args
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:51:13UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:51:13UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:51:13UTC deis-database[1]: raise ex
2016-07-20T15:51:13UTC deis-database[1]: timeout: timed out
2016-07-20T15:51:13UTC deis-database[1]: <Greenlet at 0x7f9bfb6ebeb0: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:51:13UTC deis-database[1]:
2016-07-20T15:51:20UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:51:20UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:51:20UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:51:20UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:51:20UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:51:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:51:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:51:20UTC deis-database[1]: query_args=query_args
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:51:20UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:51:20UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:51:20UTC deis-database[1]: raise ex
2016-07-20T15:51:20UTC deis-database[1]: timeout: timed out
2016-07-20T15:51:20UTC deis-database[1]: <Greenlet at 0x7f9bfb483050: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:51:20UTC deis-database[1]:
2016-07-20T15:51:25UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:51:25UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:51:25UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:51:25UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:51:25UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:51:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:51:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:51:25UTC deis-database[1]: query_args=query_args
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:51:25UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:51:25UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:51:25UTC deis-database[1]: raise ex
2016-07-20T15:51:25UTC deis-database[1]: timeout: timed out
2016-07-20T15:51:25UTC deis-database[1]: <Greenlet at 0x7f9bfb6ebf50: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:51:25UTC deis-database[1]:
2016-07-20T15:51:26UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:51:26UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:51:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:51:26UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:51:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:51:26UTC deis-database[1]: group.join()
2016-07-20T15:51:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:51:26UTC deis-database[1]: raise val
2016-07-20T15:51:26UTC deis-database[1]: timeout: timed out
2016-07-20T15:51:26UTC deis-database[1]:
2016-07-20T15:51:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:26.385369-00 pid=3630
2016-07-20T15:51:26UTC deis-database[1]: 2016-07-20 15:51:26 UTC LOG: archive command failed with exit code 2
2016-07-20T15:51:26UTC deis-database[1]: 2016-07-20 15:51:26 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:51:27UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:51:27UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:51:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:27.770945-00 pid=3761
2016-07-20T15:51:27UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:27UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:51:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:27.805291-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:51:27UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:27UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:51:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:27.947909-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.085938-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.222462-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.360340-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.506321-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.649026-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.795619-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T15:51:59UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception
2016-07-20T15:51:59UTC deis-database[1]: DETAIL: Exception information dump:
2016-07-20T15:51:59UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim
2016-07-20T15:51:59UTC deis-database[1]: return f(*args, **kwargs)
2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch
2016-07-20T15:51:59UTC deis-database[1]: bucket.delete_keys([key.name for key in page])
2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys
2016-07-20T15:51:59UTC deis-database[1]: while delete_keys2(headers):
2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2
2016-07-20T15:51:59UTC deis-database[1]: data=data)
2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:51:59UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:51:59UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:51:59UTC deis-database[1]: raise ex
2016-07-20T15:51:59UTC deis-database[1]: timeout: timed out
2016-07-20T15:51:59UTC deis-database[1]:
2016-07-20T15:51:59UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises.
2016-07-20T15:51:59UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:59.294902-00 pid=3939
2016-07-20T15:52:38UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:52:38UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:52:38UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:52:38UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:52:38UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:52:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:52:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:52:38UTC deis-database[1]: query_args=query_args
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:52:38UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:52:38UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:52:38UTC deis-database[1]: raise ex
2016-07-20T15:52:38UTC deis-database[1]: timeout: timed out
2016-07-20T15:52:38UTC deis-database[1]: <Greenlet at 0x7f76f9cd4050: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:52:38UTC deis-database[1]:
2016-07-20T15:52:42UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:52:42UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:52:42UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:52:42UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:52:42UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:52:42UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:52:42UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:52:42UTC deis-database[1]: query_args=query_args
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:52:42UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:52:42UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:52:42UTC deis-database[1]: raise ex
2016-07-20T15:52:42UTC deis-database[1]: timeout: timed out
2016-07-20T15:52:42UTC deis-database[1]: <Greenlet at 0x7f76f9f3cf50: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:52:42UTC deis-database[1]:
2016-07-20T15:52:43UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:52:43UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:52:43UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:52:43UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:52:43UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:52:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:52:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:52:43UTC deis-database[1]: query_args=query_args
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:52:43UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:52:43UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:52:43UTC deis-database[1]: raise ex
2016-07-20T15:52:43UTC deis-database[1]: timeout: timed out
2016-07-20T15:52:43UTC deis-database[1]: <Greenlet at 0x7f76f9f3cb90: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:52:43UTC deis-database[1]:
2016-07-20T15:52:49UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:52:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:52:49UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:52:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:52:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:52:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:52:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:52:49UTC deis-database[1]: query_args=query_args
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:52:49UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:52:49UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:52:49UTC deis-database[1]: raise ex
2016-07-20T15:52:49UTC deis-database[1]: timeout: timed out
2016-07-20T15:52:49UTC deis-database[1]: <Greenlet at 0x7f76f9cd40f0: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:52:49UTC deis-database[1]:
2016-07-20T15:52:54UTC deis-database[1]: Traceback (most recent call last):
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
2016-07-20T15:52:54UTC deis-database[1]: result = self._run(*self.args, **self.kwargs)
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__
2016-07-20T15:52:54UTC deis-database[1]: self.gpg_key_id)
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
2016-07-20T15:52:54UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf)
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file
2016-07-20T15:52:54UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True)
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-07-20T15:52:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-07-20T15:52:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size)
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-07-20T15:52:54UTC deis-database[1]: query_args=query_args
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
2016-07-20T15:52:54UTC deis-database[1]: retry_handler=retry_handler
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-07-20T15:52:54UTC deis-database[1]: retry_handler=retry_handler)
2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-07-20T15:52:54UTC deis-database[1]: raise ex
2016-07-20T15:52:54UTC deis-database[1]: timeout: timed out
2016-07-20T15:52:54UTC deis-database[1]: <Greenlet at 0x7f76f9cd4190: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout
2016-07-20T15:52:54UTC deis-database[1]:
2016-07-20T15:53:09UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Traceback (most recent call last):
2016-07-20T15:53:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main
2016-07-20T15:53:09UTC deis-database[1]: concurrency=args.pool_size)
2016-07-20T15:53:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive
2016-07-20T15:53:09UTC deis-database[1]: group.join()
2016-07-20T15:53:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
2016-07-20T15:53:09UTC deis-database[1]: raise val
2016-07-20T15:53:09UTC deis-database[1]: timeout: timed out
2016-07-20T15:53:09UTC deis-database[1]:
2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.000469-00 pid=3761
2016-07-20T15:53:09UTC deis-database[1]: 2016-07-20 15:53:09 UTC LOG: archive command failed with exit code 2
2016-07-20T15:53:09UTC deis-database[1]: 2016-07-20 15:53:09 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034
2016-07-20T15:53:09UTC deis-database[1]: 2016-07-20 15:53:09 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later
2016-07-20T15:53:09UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T15:53:09UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.357210-00 pid=3922
2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo".
2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.382696-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin
2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo".
2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.529969-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin
2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo".
2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.699376-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin
2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo".
2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.851905-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin
2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo".
2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.997291-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin
2016-07-20T15:53:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:10UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo".
2016-07-20T15:53:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:10.142817-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin
2016-07-20T15:53:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:10UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo".
2016-07-20T15:53:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:10.287335-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin
2016-07-20T15:53:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T15:53:10UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo".
2016-07-20T15:53:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:10.428345-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin
2016-07-20T16:05:44UTC deis-database[1]: database: waiting for confd to write initial templates...
2016-07-20T16:05:54UTC deis-database[1]: The files belonging to this database system will be owned by user "postgres".
2016-07-20T16:05:54UTC deis-database[1]: This user must also own the server process.
2016-07-20T16:05:54UTC deis-database[1]:
2016-07-20T16:05:54UTC deis-database[1]: The database cluster will be initialized with locales
2016-07-20T16:05:54UTC deis-database[1]: COLLATE: C
2016-07-20T16:05:54UTC deis-database[1]: CTYPE: C.UTF-8
2016-07-20T16:05:54UTC deis-database[1]: MESSAGES: C
2016-07-20T16:05:54UTC deis-database[1]: MONETARY: C
2016-07-20T16:05:54UTC deis-database[1]: NUMERIC: C
2016-07-20T16:05:54UTC deis-database[1]: TIME: C
2016-07-20T16:05:54UTC deis-database[1]: The default database encoding has accordingly been set to "UTF8".
2016-07-20T16:05:54UTC deis-database[1]: The default text search configuration will be set to "english".
2016-07-20T16:05:54UTC deis-database[1]:
2016-07-20T16:05:54UTC deis-database[1]: Data page checksums are disabled.
2016-07-20T16:05:54UTC deis-database[1]:
2016-07-20T16:05:54UTC deis-database[1]: creating directory /var/lib/postgresql/9.3/main ... ok
2016-07-20T16:05:54UTC deis-database[1]: creating subdirectories ... ok
2016-07-20T16:05:54UTC deis-database[1]: selecting default max_connections ... 100
2016-07-20T16:05:54UTC deis-database[1]: selecting default shared_buffers ... 128MB
2016-07-20T16:05:54UTC deis-database[1]: creating configuration files ... ok
2016-07-20T16:05:55UTC deis-database[1]: creating template1 database in /var/lib/postgresql/9.3/main/base/1 ... ok
2016-07-20T16:05:55UTC deis-database[1]: initializing pg_authid ... ok
2016-07-20T16:05:55UTC deis-database[1]: initializing dependencies ... ok
2016-07-20T16:05:55UTC deis-database[1]: creating system views ... ok
2016-07-20T16:05:55UTC deis-database[1]: loading system objects' descriptions ... ok
2016-07-20T16:05:55UTC deis-database[1]: sh: locale: not found
2016-07-20T16:05:55UTC deis-database[1]: creating collations ... ok
2016-07-20T16:05:55UTC deis-database[1]: No usable system locales were found.
2016-07-20T16:05:55UTC deis-database[1]: Use the option "--debug" to see details.
2016-07-20T16:05:55UTC deis-database[1]: creating conversions ... ok
2016-07-20T16:05:55UTC deis-database[1]: creating dictionaries ... ok
2016-07-20T16:05:55UTC deis-database[1]: setting privileges on built-in objects ... ok
2016-07-20T16:05:55UTC deis-database[1]: creating information schema ... ok
2016-07-20T16:05:55UTC deis-database[1]: loading PL/pgSQL server-side language ... ok
2016-07-20T16:05:56UTC deis-database[1]: vacuuming database template1 ... ok
2016-07-20T16:05:56UTC deis-database[1]: copying template1 to template0 ... ok
2016-07-20T16:05:56UTC deis-database[1]: copying template1 to postgres ... ok
2016-07-20T16:06:00UTC deis-database[1]:
2016-07-20T16:06:00UTC deis-database[1]: WARNING: enabling "trust" authentication for local connections
2016-07-20T16:06:00UTC deis-database[1]: You can change this by editing pg_hba.conf or using the option -A, or
2016-07-20T16:06:00UTC deis-database[1]: --auth-local and --auth-host, the next time you run initdb.
2016-07-20T16:06:00UTC deis-database[1]: syncing data to disk ... ok
2016-07-20T16:06:00UTC deis-database[1]:
2016-07-20T16:06:00UTC deis-database[1]: Success.
2016-07-20T16:06:00UTC deis-database[1]:
2016-07-20T16:06:01UTC deis-database[1]: database: expecting initialization id: e45a7062-84a0-4dd5-978b-f471e0d88138
2016-07-20T16:06:01UTC deis-database[1]: database: no existing database found or it is outdated.
2016-07-20T16:06:01UTC deis-database[1]: database: no backups found. Initializing a new database...
2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: could not open usermap file "/etc/postgresql/main/pg_ident.conf": No such file or directory
2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: database system was shut down at 2016-07-20 16:05:56 UTC
2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: MultiXact member wraparound protections are now enabled
2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: database system is ready to accept connections
2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: autovacuum launcher started
2016-07-20T16:06:02UTC deis-database[1]: ALTER ROLE
2016-07-20T16:06:02UTC deis-database[1]: CREATE ROLE
2016-07-20T16:06:02UTC deis-database[1]: ALTER ROLE
2016-07-20T16:06:03UTC deis-database[1]: CREATE DATABASE
2016-07-20T16:06:03UTC deis-database[1]: database: performing an initial backup...
2016-07-20T16:06:03UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:06:03UTC deis-database[1]: DETAIL: The subcommand is "backup-push".
2016-07-20T16:06:03UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:03.624440-00 pid=167
2016-07-20T16:06:03UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:06:03UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:06:03UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:03.973826-00 pid=177
2016-07-20T16:06:03UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:06:03UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000001" to "s3://db_wal/wal_005/000000010000000000000001.lzo".
2016-07-20T16:06:03UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:03.997350-00 pid=177 action=push-wal key=s3://db_wal/wal_005/000000010000000000000001.lzo prefix= seg=000000010000000000000001 state=begin
2016-07-20T16:06:04UTC deis-database[1]: wal_e.operator.backup INFO MSG: start upload postgres version metadata
2016-07-20T16:06:04UTC deis-database[1]: DETAIL: Uploading to s3://db_wal/basebackups_005/base_000000010000000000000002_00000040/extended_version.txt.
2016-07-20T16:06:04UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:04.412497-00 pid=167
2016-07-20T16:06:05UTC deis-database[1]: wal_e.operator.backup INFO MSG: postgres version metadata upload complete
2016-07-20T16:06:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:05.364806-00 pid=167
2016-07-20T16:06:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: beginning volume compression
2016-07-20T16:06:05UTC deis-database[1]: DETAIL: Building volume 0.
2016-07-20T16:06:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:05.407180-00 pid=167
2016-07-20T16:06:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin uploading a base backup volume
2016-07-20T16:06:06UTC deis-database[1]: DETAIL: Uploading to "s3://db_wal/basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo".
2016-07-20T16:06:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:06.174663-00 pid=167
2016-07-20T16:06:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:06:06UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000001.lzo" complete at 1437.06KiB/s.
2016-07-20T16:06:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:06.416166-00 pid=177 action=push-wal key=s3://db_wal/wal_005/000000010000000000000001.lzo prefix= rate=1437.06 seg=000000010000000000000001 state=complete
2016-07-20T16:06:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: finish uploading a base backup volume
2016-07-20T16:06:07UTC deis-database[1]: DETAIL: Uploading to "s3://db_wal/basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo" complete at 3564.64KiB/s.
2016-07-20T16:06:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:07.370915-00 pid=167
2016-07-20T16:06:07UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:06:07UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:06:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:07.843843-00 pid=191
2016-07-20T16:06:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:06:07UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000002" to "s3://db_wal/wal_005/000000010000000000000002.lzo".
2016-07-20T16:06:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:07.866923-00 pid=191 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.lzo prefix= seg=000000010000000000000002 state=begin
2016-07-20T16:06:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:06:08UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000002.lzo" complete at 439.525KiB/s.
2016-07-20T16:06:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:08.354917-00 pid=191 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.lzo prefix= rate=439.525 seg=000000010000000000000002 state=complete
2016-07-20T16:06:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:06:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:06:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:08.664902-00 pid=198
2016-07-20T16:06:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:06:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000002.00000028.backup" to "s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo".
2016-07-20T16:06:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:08.687202-00 pid=198 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo prefix= seg=000000010000000000000002.00000028.backup state=begin
2016-07-20T16:06:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:06:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo" complete at 00KiB/s.
2016-07-20T16:06:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:09.105333-00 pid=198 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo prefix= rate=00 seg=000000010000000000000002.00000028.backup state=complete
2016-07-20T16:06:09UTC deis-database[1]: NOTICE: pg_stop_backup complete, all required WAL segments have been archived
2016-07-20T16:06:09UTC deis-database[1]: database: postgres is running...
2016-07-20T16:07:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:07:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:07:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:07:08.936854-00 pid=284
2016-07-20T16:07:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:07:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000003" to "s3://db_wal/wal_005/000000010000000000000003.lzo".
2016-07-20T16:07:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:07:08.957889-00 pid=284 action=push-wal key=s3://db_wal/wal_005/000000010000000000000003.lzo prefix= seg=000000010000000000000003 state=begin
2016-07-20T16:07:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:07:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000003.lzo" complete at 492.032KiB/s.
2016-07-20T16:07:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:07:09.657617-00 pid=284 action=push-wal key=s3://db_wal/wal_005/000000010000000000000003.lzo prefix= rate=492.032 seg=000000010000000000000003 state=complete
2016-07-20T16:08:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:08:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:08:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:08:08.217418-00 pid=369
2016-07-20T16:08:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:08:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000004" to "s3://db_wal/wal_005/000000010000000000000004.lzo".
2016-07-20T16:08:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:08:08.275400-00 pid=369 action=push-wal key=s3://db_wal/wal_005/000000010000000000000004.lzo prefix= seg=000000010000000000000004 state=begin
2016-07-20T16:08:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:08:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000004.lzo" complete at 478.695KiB/s.
2016-07-20T16:08:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:08:09.799474-00 pid=369 action=push-wal key=s3://db_wal/wal_005/000000010000000000000004.lzo prefix= rate=478.695 seg=000000010000000000000004 state=complete
2016-07-20T16:09:07UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:09:07UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:09:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:09:07.625836-00 pid=441
2016-07-20T16:09:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:09:07UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000005" to "s3://db_wal/wal_005/000000010000000000000005.lzo".
2016-07-20T16:09:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:09:07.647171-00 pid=441 action=push-wal key=s3://db_wal/wal_005/000000010000000000000005.lzo prefix= seg=000000010000000000000005 state=begin
2016-07-20T16:09:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:09:10UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000005.lzo" complete at 40.4188KiB/s.
2016-07-20T16:09:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:09:10.259547-00 pid=441 action=push-wal key=s3://db_wal/wal_005/000000010000000000000005.lzo prefix= rate=40.4188 seg=000000010000000000000005 state=complete
2016-07-20T16:14:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:14:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:14:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:14:08.500846-00 pid=807
2016-07-20T16:14:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:14:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000006" to "s3://db_wal/wal_005/000000010000000000000006.lzo".
2016-07-20T16:14:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:14:08.590810-00 pid=807 action=push-wal key=s3://db_wal/wal_005/000000010000000000000006.lzo prefix= seg=000000010000000000000006 state=begin
2016-07-20T16:14:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:14:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000006.lzo" complete at 9505.92KiB/s.
2016-07-20T16:14:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:14:09.768045-00 pid=807 action=push-wal key=s3://db_wal/wal_005/000000010000000000000006.lzo prefix= rate=9505.92 seg=000000010000000000000006 state=complete
2016-07-20T16:18:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:18:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:18:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:18:08.304874-00 pid=1111
2016-07-20T16:18:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:18:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000007" to "s3://db_wal/wal_005/000000010000000000000007.lzo".
2016-07-20T16:18:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:18:08.335939-00 pid=1111 action=push-wal key=s3://db_wal/wal_005/000000010000000000000007.lzo prefix= seg=000000010000000000000007 state=begin
2016-07-20T16:18:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:18:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000007.lzo" complete at 1149.33KiB/s.
2016-07-20T16:18:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:18:09.227802-00 pid=1111 action=push-wal key=s3://db_wal/wal_005/000000010000000000000007.lzo prefix= rate=1149.33 seg=000000010000000000000007 state=complete
2016-07-20T16:23:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:23:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:23:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:23:08.483431-00 pid=1489
2016-07-20T16:23:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:23:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000008" to "s3://db_wal/wal_005/000000010000000000000008.lzo".
2016-07-20T16:23:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:23:08.514281-00 pid=1489 action=push-wal key=s3://db_wal/wal_005/000000010000000000000008.lzo prefix= seg=000000010000000000000008 state=begin
2016-07-20T16:23:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:23:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000008.lzo" complete at 523.868KiB/s.
2016-07-20T16:23:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:23:09.107577-00 pid=1489 action=push-wal key=s3://db_wal/wal_005/000000010000000000000008.lzo prefix= rate=523.868 seg=000000010000000000000008 state=complete
2016-07-20T16:28:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:28:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:28:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:28:08.506632-00 pid=1855
2016-07-20T16:28:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:28:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000009" to "s3://db_wal/wal_005/000000010000000000000009.lzo".
2016-07-20T16:28:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:28:08.529235-00 pid=1855 action=push-wal key=s3://db_wal/wal_005/000000010000000000000009.lzo prefix= seg=000000010000000000000009 state=begin
2016-07-20T16:28:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:28:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000009.lzo" complete at 2422.68KiB/s.
2016-07-20T16:28:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:28:09.389993-00 pid=1855 action=push-wal key=s3://db_wal/wal_005/000000010000000000000009.lzo prefix= rate=2422.68 seg=000000010000000000000009 state=complete
2016-07-20T16:33:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:33:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:33:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:33:08.666435-00 pid=2233
2016-07-20T16:33:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:33:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000A" to "s3://db_wal/wal_005/00000001000000000000000A.lzo".
2016-07-20T16:33:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:33:08.688055-00 pid=2233 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000A.lzo prefix= seg=00000001000000000000000A state=begin
2016-07-20T16:33:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:33:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000A.lzo" complete at 598.405KiB/s.
2016-07-20T16:33:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:33:09.202731-00 pid=2233 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000A.lzo prefix= rate=598.405 seg=00000001000000000000000A state=complete
2016-07-20T16:38:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:38:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:38:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:38:08.878056-00 pid=2598
2016-07-20T16:38:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:38:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000B" to "s3://db_wal/wal_005/00000001000000000000000B.lzo".
2016-07-20T16:38:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:38:08.914100-00 pid=2598 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000B.lzo prefix= seg=00000001000000000000000B state=begin
2016-07-20T16:38:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:38:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000B.lzo" complete at 6696.11KiB/s.
2016-07-20T16:38:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:38:09.900464-00 pid=2598 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000B.lzo prefix= rate=6696.11 seg=00000001000000000000000B state=complete
2016-07-20T16:43:09UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:43:09UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:43:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:43:09.005446-00 pid=2976
2016-07-20T16:43:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:43:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000C" to "s3://db_wal/wal_005/00000001000000000000000C.lzo".
2016-07-20T16:43:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:43:09.033349-00 pid=2976 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000C.lzo prefix= seg=00000001000000000000000C state=begin
2016-07-20T16:43:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:43:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000C.lzo" complete at 3187.16KiB/s.
2016-07-20T16:43:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:43:09.491392-00 pid=2976 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000C.lzo prefix= rate=3187.16 seg=00000001000000000000000C state=complete
2016-07-20T16:48:09UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E
2016-07-20T16:48:09UTC deis-database[1]: DETAIL: The subcommand is "wal-push".
2016-07-20T16:48:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:48:09.132989-00 pid=3354
2016-07-20T16:48:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file
2016-07-20T16:48:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000D" to "s3://db_wal/wal_005/00000001000000000000000D.lzo".
2016-07-20T16:48:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:48:09.160614-00 pid=3354 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000D.lzo prefix= seg=00000001000000000000000D state=begin
2016-07-20T16:48:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file
2016-07-20T16:48:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000D.lzo" complete at 1523.32KiB/s.
2016-07-20T16:48:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:48:09.465534-00 pid=3354 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000D.lzo prefix= rate=1523.32 seg=00000001000000000000000D state=complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment