wal-e error on secondary, after pulling down a basebackup from a pg_dump/pg_restore

log.log

2013-09-25 05:46:27 UTC FATAL:  the database system is starting up
Traceback (most recent call last):
lzop: <stdin>: not a lzop file
  File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/s3_worker.py", line 260, in write_and_close_thread
    key.get_contents_to_file(stream)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1226, in get_contents_to_file
    response_headers=response_headers)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1102, in get_file
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 215, in open
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 169, in open_read
    self.resp.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>wal_005/000000010000016100000061.lzo</Key><RequestId>F3576CF3F63331E5</RequestId><HostId>gEkyLpAcIVR3eNuMkW7OqXCOhYS/ryR2B78/y
isZ10+a5LYyXAbffFkmdP4bFhIH</HostId></Error>
<Greenlet at 0x19d0cd0: write_and_close_thread(<Key: facebook-mirror-wal-e-production,/wal_005/00, <wal_e.piper.NonBlockPipeFileWrap object at 0x1a3c)> failed with S3ResponseError

wal_e.worker.s3_worker INFO     MSG: could not locate object while performing wal restore
        DETAIL: The absolute URI that could not be located is s3://facebook-mirror-wal-e-production/wal_005/000000010000016100000061.lzo.
        HINT: This can be normal when Postgres is trying to detect what timelines are available during restoration.
        STRUCTURED: time=2013-09-25T05:46:27.622183-00 pid=9956
wal_e.operator.s3_operator INFO     MSG: complete wal restore
        STRUCTURED: time=2013-09-25T05:46:27.623049-00 pid=9956 action=wal-fetch key=s3://facebook-mirror-wal-e-production/wal_005/000000010000016100000061.lzo prefix=s3://facebook-mirror-wal-e-production seg=000
000010000016100000061 state=complete
2013-09-25 05:46:27 UTC FATAL:  the database system is starting up
2013-09-25 05:46:27 UTC LOG:  consistent recovery state reached at 161/61003450
2013-09-25 05:46:27 UTC LOG:  record with zero length at 161/61003450
2013-09-25 05:46:27 UTC LOG:  database system is ready to accept read only connections
wal_e.operator.s3_operator INFO     MSG: begin wal restore
        STRUCTURED: time=2013-09-25T05:46:27.794913-00 pid=9983 action=wal-fetch key=s3://facebook-mirror-wal-e-production/wal_005/000000010000016100000061.lzo prefix=s3://facebook-mirror-wal-e-production seg=000
000010000016100000061 state=begin
Traceback (most recent call last):
lzop: <stdin>:   File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
not a lzop file
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/s3_worker.py", line 260, in write_and_close_thread
    key.get_contents_to_file(stream)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1226, in get_contents_to_file
    response_headers=response_headers)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1102, in get_file
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 215, in open
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 169, in open_read
    self.resp.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>wal_005/000000010000016100000061.lzo</Key><RequestId>4FB2DE09A534D706</RequestId><HostId>QR65Zfb0cm9ZhYBnb6toaL8NFUrM5HDErCSRVLmEu1OTSorkjZhZpeC+v7K4Oxuf</HostId></Error>
<Greenlet at 0x15aecd0: write_and_close_thread(<Key: facebook-mirror-wal-e-production,/wal_005/00, <wal_e.piper.NonBlockPipeFileWrap object at 0x161a)> failed with S3ResponseError

wal_e.worker.s3_worker INFO     MSG: could not locate object while performing wal restore
        DETAIL: The absolute URI that could not be located is s3://facebook-mirror-wal-e-production/wal_005/000000010000016100000061.lzo.
        HINT: This can be normal when Postgres is trying to detect what timelines are available during restoration.
        STRUCTURED: time=2013-09-25T05:46:27.837472-00 pid=9983
wal_e.operator.s3_operator INFO     MSG: complete wal restore
        STRUCTURED: time=2013-09-25T05:46:27.838260-00 pid=9983 action=wal-fetch key=s3://facebook-mirror-wal-e-production/wal_005/000000010000016100000061.lzo prefix=s3://facebook-mirror-wal-e-production seg=000000010000016100000061 state=complete
2013-09-25 05:46:27 UTC LOG:  streaming replication successfully connected to primary
2