Skip to content

Instantly share code, notes, and snippets.

@krislindgren

krislindgren/Fail after reconnect

Created March 16, 2015 22:12
Show Gist options
  • Save krislindgren/6a05e91263801de94ca1 to your computer and use it in GitHub Desktop.
Save krislindgren/6a05e91263801de94ca1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Fail after reconnect
2015-03-16 15:05:36.141 17667 INFO nova.compute.resource_tracker [-] Compute_service record updated for g1dlgenlab11.dev.glbt1.gdg:g1dlgenlab11.dev.glbt1.gdg
2015-03-16 15:06:04.998 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'conductor': Socket closed
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 827, in execute_method
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1024, in _publish
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit **kwargs)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 429, in __init__
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit **options)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 375, in __init__
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit self.reconnect(channel)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 383, in reconnect
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit self.revive(self._channel)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit self.declare()
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit self.exchange.declare()
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 613, in exchange_declare
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit self._send_method((40, 10), args)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 251, in _write
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit raise IOError('Socket closed')
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit IOError: Socket closed
2015-03-16 15:06:04.998 17667 TRACE oslo_messaging._drivers.impl_rabbit
2015-03-16 15:06:05.001 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server openstack-test.int.godaddy.com:5672 closed the connection. Check login credentials: Socket closed
2015-03-16 15:06:05.005 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'conductor': connection already closed
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 827, in execute_method
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1024, in _publish
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit **kwargs)
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 429, in __init__
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit **options)
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 375, in __init__
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit self.reconnect(channel)
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 383, in reconnect
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit self.revive(self._channel)
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit self.declare()
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit self.exchange.declare()
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 613, in exchange_declare
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit self._send_method((40, 10), args)
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in _send_method
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit raise RecoverableConnectionError('connection already closed')
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit RecoverableConnectionError: connection already closed
2015-03-16 15:06:05.005 17667 TRACE oslo_messaging._drivers.impl_rabbit
2015-03-16 15:06:05.007 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on openstack-test.int.godaddy.com:5672 is unreachable: connection already closed. Trying again in 1 seconds.
Raw
Restart rabbitmq logs
2015-03-16 15:02:49.789 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 827, in execute_method
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1003, in _consume
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=poll_timeout)
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-03-16 15:02:49.789 17667 TRACE oslo_messaging._drivers.impl_rabbit
2015-03-16 15:02:49.792 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on openstack-test.int.godaddy.com:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2015-03-16 15:02:49.934 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 827, in execute_method
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1003, in _consume
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=poll_timeout)
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-03-16 15:02:49.934 17667 TRACE oslo_messaging._drivers.impl_rabbit
2015-03-16 15:02:49.936 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on openstack-test.int.godaddy.com:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2015-03-16 15:02:49.990 17667 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel errors occurs, try to reconnect.
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 930, in _heartbeat_thread_job
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit self.connection.drain_events(timeout=0.001)
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-03-16 15:02:49.990 17667 TRACE oslo_messaging._drivers.impl_rabbit
2015-03-16 15:02:50.127 17667 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel errors occurs, try to reconnect.
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 930, in _heartbeat_thread_job
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit self.connection.drain_events(timeout=0.001)
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-03-16 15:02:50.127 17667 TRACE oslo_messaging._drivers.impl_rabbit
2015-03-16 15:02:50.861 17667 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on openstack-test.int.godaddy.com:5672
2015-03-16 15:02:50.992 17667 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on openstack-test.int.godaddy.com:5672
2015-03-16 15:02:54.910 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'conductor': connection already closed
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 827, in execute_method
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1024, in _publish
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit **kwargs)
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 429, in __init__
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit **options)
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 375, in __init__
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit self.reconnect(channel)
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 383, in reconnect
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit self.revive(self._channel)
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit self.declare()
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit self.exchange.declare()
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 613, in exchange_declare
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit self._send_method((40, 10), args)
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in _send_method
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit raise RecoverableConnectionError('connection already closed')
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit RecoverableConnectionError: connection already closed
2015-03-16 15:02:54.910 17667 TRACE oslo_messaging._drivers.impl_rabbit
2015-03-16 15:02:54.913 17667 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on openstack-test.int.godaddy.com:5672 is unreachable: connection already closed. Trying again in 1 seconds.
2015-03-16 15:02:55.950 17667 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on openstack-test.int.godaddy.com:5672
2015-03-16 15:02:57.837 17667 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel errors occurs, try to reconnect.
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 930, in _heartbeat_thread_job
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit self.connection.drain_events(timeout=0.001)
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-03-16 15:02:57.837 17667 TRACE oslo_messaging._drivers.impl_rabbit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment