Skip to content

Instantly share code, notes, and snippets.

@DorianZaccaria

DorianZaccaria/gist:2ea8384929d50a7255d0

Created January 5, 2015 22:24
Show Gist options
  • Save DorianZaccaria/2ea8384929d50a7255d0 to your computer and use it in GitHub Desktop.
Save DorianZaccaria/2ea8384929d50a7255d0 to your computer and use it in GitHub Desktop.
Download ZIP
Cassandra error in prod (on mcnulty-query)
Raw
gistfile1.txt
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1506, in run_add_or_renew_pool
new_pool = HostConnectionPool(host, distance, self)
File "/usr/local/lib/python2.7/dist-packages/cassandra/pool.py", line 397, in __init__
for i in range(core_conns)]
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 645, in connection_factory
return self.connection_class.factory(address, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 162, in factory
conn = cls(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 195, in __init__
raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('10.31.49.38', 9042)]. Last error: Connection refused
2015-01-05T22:22:24,615 INFO [cassandra.cluster] [20994] New Cassandra host <Host: 10.63.152.227 aws-us-east-1> discovered
2015-01-05T22:22:24,615 INFO [cassandra.cluster] [20994] New Cassandra host <Host: 10.31.49.38 aws-us-east-1> discovered
2015-01-05T22:22:24,616 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.63.191.178 aws-us-east-1> discovered
2015-01-05T22:22:24,616 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.95.133.250 aws-us-east-1> discovered
2015-01-05T22:22:24,616 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.168.77.46 aws-us-east-1> discovered
2015-01-05T22:22:24,617 INFO [ddcassandog.common] [21010] Establishing a Cassandra session: keyspace=Intake, hosts=10.150.54.107,10.168.77.46,10.179.179.67,10.181.43.237,10.237.148.86,10.237.175.39,10.30.229.111,10.31.49.38,10.63.152.227,10.63.184.33,10.63.191.178,10.67.184.172,10.69.62.10,10.95.133.250,10.95.133.32,10.97.133.107
2015-01-05T22:22:24,617 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.181.43.237 aws-us-east-1> discovered
2015-01-05T22:22:24,617 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.67.184.172 aws-us-east-1> discovered
2015-01-05T22:22:24,617 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.63.184.33 aws-us-east-1> discovered
2015-01-05T22:22:24,618 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.30.229.111 aws-us-east-1> discovered
2015-01-05T22:22:24,618 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.237.148.86 aws-us-east-1> discovered
2015-01-05T22:22:24,618 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.237.175.39 aws-us-east-1> discovered
2015-01-05T22:22:24,618 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.95.133.32 aws-us-east-1> discovered
2015-01-05T22:22:24,619 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.63.152.227 aws-us-east-1> discovered
2015-01-05T22:22:24,619 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.31.49.38 aws-us-east-1> discovered
2015-01-05T22:22:24,619 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.150.54.107 aws-us-east-1> discovered
2015-01-05T22:22:24,619 INFO [cassandra.cluster] [20984] New Cassandra host <Host: 10.69.62.10 aws-us-east-1> discovered
2015-01-05T22:22:24,619 WARNI [cassandra.cluster] [20973] Failed to create connection pool for new host 10.31.49.38:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1506, in run_add_or_renew_pool
new_pool = HostConnectionPool(host, distance, self)
File "/usr/local/lib/python2.7/dist-packages/cassandra/pool.py", line 397, in __init__
for i in range(core_conns)]
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 645, in connection_factory
return self.connection_class.factory(address, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 162, in factory
conn = cls(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 195, in __init__
raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('10.31.49.38', 9042)]. Last error: Connection refused
2015-01-05T22:22:24,623 WARNI [cassandra.cluster] [20972] Host 10.31.49.38 has been marked down
2015-01-05T22:22:24,624 WARNI [cassandra.cluster] [20974] Host 10.31.49.38 has been marked down
2015-01-05T22:22:24,641 INFO [ddcassandog.common] [20975] Establishing a Cassandra session: keyspace=Intake, hosts=10.150.54.107,10.168.77.46,10.179.179.67,10.181.43.237,10.237.148.86,10.237.175.39,10.30.229.111,10.31.49.38,10.63.152.227,10.63.184.33,10.63.191.178,10.67.184.172,10.69.62.10,10.95.133.250,10.95.133.32,10.97.133.107
2015-01-05T22:22:24,657 WARNI [cassandra.cluster] [20971] Host 10.31.49.38 has been marked down
2015-01-05T22:22:24,657 INFO [dd.dogapi] [20998] Initializing dog api to use statsd: localhost, 8125
2015-01-05T22:22:24,659 INFO [dd.dd.utils.db] [20998] dogweb is connecting to postgres @ dog@localhost:15433/dogdataprod. Pgbouncer: True
2015-01-05T22:22:24,715 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.63.191.178 aws-us-east-1> discovered
2015-01-05T22:22:24,716 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.95.133.250 aws-us-east-1> discovered
2015-01-05T22:22:24,716 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.97.133.107 aws-us-east-1> discovered
2015-01-05T22:22:24,716 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.168.77.46 aws-us-east-1> discovered
2015-01-05T22:22:24,716 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.181.43.237 aws-us-east-1> discovered
2015-01-05T22:22:24,716 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.67.184.172 aws-us-east-1> discovered
2015-01-05T22:22:24,716 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.179.179.67 aws-us-east-1> discovered
2015-01-05T22:22:24,717 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.30.229.111 aws-us-east-1> discovered
2015-01-05T22:22:24,717 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.237.148.86 aws-us-east-1> discovered
2015-01-05T22:22:24,717 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.237.175.39 aws-us-east-1> discovered
2015-01-05T22:22:24,717 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.95.133.32 aws-us-east-1> discovered
2015-01-05T22:22:24,717 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.31.49.38 aws-us-east-1> discovered
2015-01-05T22:22:24,718 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.150.54.107 aws-us-east-1> discovered
2015-01-05T22:22:24,718 INFO [cassandra.cluster] [20986] New Cassandra host <Host: 10.69.62.10 aws-us-east-1> discovered
2015-01-05T22:22:24,720 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.63.191.178 aws-us-east-1> discovered
2015-01-05T22:22:24,720 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.95.133.250 aws-us-east-1> discovered
2015-01-05T22:22:24,720 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.97.133.107 aws-us-east-1> discovered
2015-01-05T22:22:24,721 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.168.77.46 aws-us-east-1> discovered
2015-01-05T22:22:24,721 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.181.43.237 aws-us-east-1> discovered
2015-01-05T22:22:24,721 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.179.179.67 aws-us-east-1> discovered
2015-01-05T22:22:24,721 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.63.184.33 aws-us-east-1> discovered
2015-01-05T22:22:24,722 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.237.148.86 aws-us-east-1> discovered
2015-01-05T22:22:24,722 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.237.175.39 aws-us-east-1> discovered
2015-01-05T22:22:24,722 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.95.133.32 aws-us-east-1> discovered
2015-01-05T22:22:24,722 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.63.152.227 aws-us-east-1> discovered
2015-01-05T22:22:24,722 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.31.49.38 aws-us-east-1> discovered
2015-01-05T22:22:24,722 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.150.54.107 aws-us-east-1> discovered
2015-01-05T22:22:24,723 INFO [cassandra.cluster] [20985] New Cassandra host <Host: 10.69.62.10 aws-us-east-1> discovered
2015-01-05T22:22:24,756 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.63.191.178 aws-us-east-1> discovered
2015-01-05T22:22:24,756 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.95.133.250 aws-us-east-1> discovered
2015-01-05T22:22:24,756 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.97.133.107 aws-us-east-1> discovered
2015-01-05T22:22:24,756 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.181.43.237 aws-us-east-1> discovered
2015-01-05T22:22:24,757 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.67.184.172 aws-us-east-1> discovered
2015-01-05T22:22:24,757 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.179.179.67 aws-us-east-1> discovered
2015-01-05T22:22:24,757 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.63.184.33 aws-us-east-1> discovered
2015-01-05T22:22:24,757 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.30.229.111 aws-us-east-1> discovered
2015-01-05T22:22:24,757 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.237.148.86 aws-us-east-1> discovered
2015-01-05T22:22:24,757 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.237.175.39 aws-us-east-1> discovered
2015-01-05T22:22:24,758 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.95.133.32 aws-us-east-1> discovered
2015-01-05T22:22:24,758 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.63.152.227 aws-us-east-1> discovered
2015-01-05T22:22:24,758 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.31.49.38 aws-us-east-1> discovered
2015-01-05T22:22:24,758 INFO [cassandra.cluster] [21010] New Cassandra host <Host: 10.150.54.107 aws-us-east-1> discovered
2015-01-05T22:22:24,757 WARNI [cassandra.cluster] [20994] Failed to create connection pool for new host 10.31.49.38:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1506, in run_add_or_renew_pool
new_pool = HostConnectionPool(host, distance, self)
File "/usr/local/lib/python2.7/dist-packages/cassandra/pool.py", line 397, in __init__
for i in range(core_conns)]
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 645, in connection_factory
return self.connection_class.factory(address, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 162, in factory
conn = cls(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 195, in __init__
raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('10.31.49.38', 9042)]. Last error: Connection refused
2015-01-05T22:22:24,763 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.63.191.178 aws-us-east-1> discovered
2015-01-05T22:22:24,763 INFO [dd.dogapi] [21008] Initializing dog api to use statsd: localhost, 8125
2015-01-05T22:22:24,764 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.95.133.250 aws-us-east-1> discovered
2015-01-05T22:22:24,764 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.97.133.107 aws-us-east-1> discovered
2015-01-05T22:22:24,765 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.168.77.46 aws-us-east-1> discovered
2015-01-05T22:22:24,764 INFO [dd.dd.utils.db] [21008] dogweb is connecting to postgres @ dog@localhost:15433/dogdataprod. Pgbouncer: True
2015-01-05T22:22:24,765 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.181.43.237 aws-us-east-1> discovered
2015-01-05T22:22:24,765 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.67.184.172 aws-us-east-1> discovered
2015-01-05T22:22:24,765 INFO [ddcassandog.common] [20998] Establishing a Cassandra session: keyspace=Intake, hosts=10.150.54.107,10.168.77.46,10.179.179.67,10.181.43.237,10.237.148.86,10.237.175.39,10.30.229.111,10.31.49.38,10.63.152.227,10.63.184.33,10.63.191.178,10.67.184.172,10.69.62.10,10.95.133.250,10.95.133.32,10.97.133.107
2015-01-05T22:22:24,766 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.179.179.67 aws-us-east-1> discovered
2015-01-05T22:22:24,766 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.63.184.33 aws-us-east-1> discovered
2015-01-05T22:22:24,766 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.30.229.111 aws-us-east-1> discovered
2015-01-05T22:22:24,767 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.237.148.86 aws-us-east-1> discovered
2015-01-05T22:22:24,767 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.237.175.39 aws-us-east-1> discovered
2015-01-05T22:22:24,767 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.95.133.32 aws-us-east-1> discovered
2015-01-05T22:22:24,767 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.63.152.227 aws-us-east-1> discovered
2015-01-05T22:22:24,768 INFO [cassandra.cluster] [20991] New Cassandra host <Host: 10.69.62.10 aws-us-east-1> discovered
2015-01-05T22:22:24,768 WARNI [cassandra.cluster] [20998] [control connection] Error connecting to 10.31.49.38:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1770, in _reconnect_internal
return self._try_connect(host)
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1787, in _try_connect
connection = self._cluster.connection_factory(host.address, is_control_connection=True)
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 645, in connection_factory
return self.connection_class.factory(address, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 162, in factory
conn = cls(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 195, in __init__
raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('10.31.49.38', 9042)]. Last error: Connection refused
2015-01-05T22:22:24,772 WARNI [cassandra.cluster] [20984] Failed to create connection pool for new host 10.31.49.38:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1506, in run_add_or_renew_pool
new_pool = HostConnectionPool(host, distance, self)
File "/usr/local/lib/python2.7/dist-packages/cassandra/pool.py", line 397, in __init__
for i in range(core_conns)]
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 645, in connection_factory
return self.connection_class.factory(address, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 162, in factory
conn = cls(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 195, in __init__
raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('10.31.49.38', 9042)]. Last error: Connection refused
2015-01-05T22:22:24,819 WARNI [cassandra.cluster] [20973] Host 10.31.49.38 has been marked down
2015-01-05T22:22:24,821 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.95.133.250 aws-us-east-1> discovered
2015-01-05T22:22:24,822 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.97.133.107 aws-us-east-1> discovered
2015-01-05T22:22:24,822 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.168.77.46 aws-us-east-1> discovered
2015-01-05T22:22:24,822 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.181.43.237 aws-us-east-1> discovered
2015-01-05T22:22:24,822 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.67.184.172 aws-us-east-1> discovered
2015-01-05T22:22:24,822 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.63.184.33 aws-us-east-1> discovered
2015-01-05T22:22:24,823 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.30.229.111 aws-us-east-1> discovered
2015-01-05T22:22:24,823 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.237.148.86 aws-us-east-1> discovered
2015-01-05T22:22:24,823 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.237.175.39 aws-us-east-1> discovered
2015-01-05T22:22:24,823 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.95.133.32 aws-us-east-1> discovered
2015-01-05T22:22:24,823 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.63.152.227 aws-us-east-1> discovered
2015-01-05T22:22:24,823 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.31.49.38 aws-us-east-1> discovered
2015-01-05T22:22:24,824 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.150.54.107 aws-us-east-1> discovered
2015-01-05T22:22:24,824 INFO [cassandra.cluster] [20993] New Cassandra host <Host: 10.69.62.10 aws-us-east-1> discovered
2015-01-05T22:22:24,825 INFO [ddcassandog.common] [21008] Establishing a Cassandra session: keyspace=Intake, hosts=10.150.54.107,10.168.77.46,10.179.179.67,10.181.43.237,10.237.148.86,10.237.175.39,10.30.229.111,10.31.49.38,10.63.152.227,10.63.184.33,10.63.191.178,10.67.184.172,10.69.62.10,10.95.133.250,10.95.133.32,10.97.133.107
2015-01-05T22:22:24,827 WARNI [cassandra.cluster] [21008] [control connection] Error connecting to 10.31.49.38:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1770, in _reconnect_internal
return self._try_connect(host)
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1787, in _try_connect
connection = self._cluster.connection_factory(host.address, is_control_connection=True)
File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 645, in connection_factory
return self.connection_class.factory(address, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 162, in factory
conn = cls(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cassandra/io/asyncorereactor.py", line 195, in __init__
raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('10.31.49.38', 9042)]. Last error: Connection refused
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment