Skip to content

Instantly share code, notes, and snippets.

@winks

winks/cassandra.log Secret

Created September 25, 2015 14:59
Show Gist options
  • Save winks/989ff567c11879646d3e to your computer and use it in GitHub Desktop.
Save winks/989ff567c11879646d3e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
cassandra.log
cassandra@cassandra-1:~$ cassandra-snapshotter --aws-access-key-id=REDACTED --aws-secret-access-key=SECRET --s3-bucket-region=eu-west-1 --s3-bucket-name=COMPANY-cassandra-backup --s3-base-path=prod/COMPANY_cluster/$(date +%u) backup --hosts=172.31.5.185,172.31.6.149,172.31.5.20,172.31.10.42 --user=cassandra --cassandra-conf-path=/etc/cassandra --backup-schema
WARNING:root:Response: u'The specified key does not exist.' manifest_path: u'prod/COMPANY_cluster/5/20150925135442//manifest.json'
[172.31.5.185] Executing task 'node_start_backup'
[172.31.6.149] Executing task 'node_start_backup'
[172.31.5.20] Executing task 'node_start_backup'
[172.31.10.42] Executing task 'node_start_backup'
[172.31.5.185] Executing task 'upload_node_backups'
[172.31.6.149] Executing task 'upload_node_backups'
[172.31.5.20] Executing task 'upload_node_backups'
[172.31.10.42] Executing task 'upload_node_backups'
[172.31.10.42] run: cassandra-snapshotter-agent create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20150925141717 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra
[172.31.5.20] run: cassandra-snapshotter-agent create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20150925141717 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra
[172.31.5.185] run: cassandra-snapshotter-agent create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20150925141717 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra
[172.31.6.149] run: cassandra-snapshotter-agent create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20150925141717 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra
[172.31.5.185] run: cassandra-snapshotter-agent put --aws-access-key-id=REDACTED --aws-secret-access-key=SECRET --s3-bucket-name=COMPANY-cassandra-backup --s3-bucket-region=eu-west-1 --s3-base-path=prod/COMPANY_cluster/5/20150925141717/172.31.5.185 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4
[172.31.5.185] out: lzop 1.03
[172.31.5.185] out: LZO library 2.06
[172.31.5.185] out: Copyright (C) 1996-2010 Markus Franz Xaver Johannes Oberhumer
[172.31.10.42] run: cassandra-snapshotter-agent put --aws-access-key-id=REDACTED --aws-secret-access-key=SECRET --s3-bucket-name=COMPANY-cassandra-backup --s3-bucket-region=eu-west-1 --s3-base-path=prod/COMPANY_cluster/5/20150925141717/172.31.10.42 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4
[172.31.6.149] run: cassandra-snapshotter-agent put --aws-access-key-id=REDACTED --aws-secret-access-key=SECRET --s3-bucket-name=COMPANY-cassandra-backup --s3-bucket-region=eu-west-1 --s3-base-path=prod/COMPANY_cluster/5/20150925141717/172.31.6.149 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4
[172.31.5.20] run: cassandra-snapshotter-agent put --aws-access-key-id=REDACTED --aws-secret-access-key=SECRET --s3-bucket-name=COMPANY-cassandra-backup --s3-bucket-region=eu-west-1 --s3-base-path=prod/COMPANY_cluster/5/20150925141717/172.31.5.20 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4
[172.31.10.42] out: lzop 1.03
[172.31.10.42] out: LZO library 2.06
[172.31.10.42] out: Copyright (C) 1996-2010 Markus Franz Xaver Johannes Oberhumer
[172.31.5.20] out: lzop 1.03
[172.31.5.20] out: LZO library 2.06
[172.31.5.20] out: Copyright (C) 1996-2010 Markus Franz Xaver Johannes Oberhumer
[172.31.6.149] out: lzop 1.03
[172.31.6.149] out: LZO library 2.06
[172.31.6.149] out: Copyright (C) 1996-2010 Markus Franz Xaver Johannes Oberhumer
[172.31.10.42] out: Traceback (most recent call last):
[172.31.10.42] out: File "/usr/local/bin/cassandra-snapshotter-agent", line 9, in <module>
[172.31.10.42] out: load_entry_point('cassandra-snapshotter==0.4.0', 'console_scripts', 'cassandra-snapshotter-agent')()
[172.31.10.42] out: File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/agent.py", line 266, in main
[172.31.10.42] out: args.incremental_backups
[172.31.10.42] out: File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/agent.py", line 145, in put_from_manifest
[172.31.10.42] out: for _ in pool.imap(upload_file, ((bucket, f, destination_path(s3_base_path, f), s3_ssenc, buffer_size) for f in files)):
[172.31.10.42] out: File "/usr/lib/python2.7/multiprocessing/pool.py", line 626, in next
[172.31.10.42] out: raise value
[172.31.10.42] out: boto.exception.S3ResponseError: S3ResponseError: 200 OK
[172.31.10.42] out: <?xml version="1.0" encoding="UTF-8"?>
[172.31.10.42] out: <Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>213B56EDD1DA15E9</RequestId><HostId>jcI63cX22z+XG91LQIDAt5bSP9kEYureXx6QdYu/kbkVVhnuD0T150rbzgvhKVUNVMU9GqObLYA=</HostId></Error>
[172.31.10.42] out:
Fatal error: run() received nonzero return code 1 while executing!
Requested: cassandra-snapshotter-agent put --aws-access-key-id=REDACTED --aws-secret-access-key=SECRET --s3-bucket-name=COMPANY-cassandra-backup --s3-bucket-region=eu-west-1 --s3-base-path=prod/COMPANY_cluster/5/20150925141717/172.31.10.42 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4
Executed: /bin/bash -l -c "cassandra-snapshotter-agent put --aws-access-key-id=REDACTED --aws-secret-access-key=SECRET --s3-bucket-name=COMPANY-cassandra-backup --s3-bucket-region=eu-west-1 --s3-base-path=prod/COMPANY_cluster/5/20150925141717/172.31.10.42 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4"
Aborting.
[172.31.5.20] out:
[172.31.5.185] out:
[172.31.6.149] out:
Fatal error: One or more hosts failed while executing task 'upload_node_backups'
Aborting.
[172.31.5.185] Executing task 'clear_node_snapshot'
[172.31.6.149] Executing task 'clear_node_snapshot'
[172.31.5.20] Executing task 'clear_node_snapshot'
[172.31.10.42] Executing task 'clear_node_snapshot'
[172.31.10.42] run: /usr/bin/nodetool clearsnapshot -t "20150925141717"
[172.31.5.20] run: /usr/bin/nodetool clearsnapshot -t "20150925141717"
[172.31.5.185] run: /usr/bin/nodetool clearsnapshot -t "20150925141717"
[172.31.6.149] run: /usr/bin/nodetool clearsnapshot -t "20150925141717"
[172.31.5.185] out: Requested clearing snapshot for: all keyspaces
[172.31.5.185] out:
[172.31.10.42] out: Requested clearing snapshot for: all keyspaces
[172.31.6.149] out: Requested clearing snapshot for: all keyspaces
[172.31.10.42] out:
[172.31.5.20] out: Requested clearing snapshot for: all keyspaces
[172.31.6.149] out:
[172.31.5.20] out:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment