Skip to content

Instantly share code, notes, and snippets.

@noelbk

noelbk/gist:f8476952809dac8a295c

Last active August 29, 2015 14:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save noelbk/f8476952809dac8a295c to your computer and use it in GitHub Desktop.
Save noelbk/f8476952809dac8a295c to your computer and use it in GitHub Desktop.
Download ZIP
rabbit still drops reply if auto_delete=False, durable=True
Raw
gistfile1.txt
Search for !!! below for milestones
!!! server consuming
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599308161744 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue network -> <Exchange nova(topic) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fdfd65c2c90>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>, 'kwargs': {'exclusive': False, 'name': 'network', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>}}), TopicConsumer({'queue': <Queue network.10.35.0.14 -> <Exchange nova(topic) bound to chan:1> -> network.10.35.0.14 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fdfd65c2c90>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>, 'kwargs': {'exclusive': False, 'name': 'network.10.35.0.14', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network.10.35.0.14', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>}}), FanoutConsumer({'queue': <Queue network_fanout_083e9a4049044be0a851eac8ef12d2bc -> <Exchange network_fanout(fanout) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fdfd65c2c90>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>, 'kwargs': {'exclusive': False, 'name': 'network_fanout_083e9a4049044be0a851eac8ef12d2bc', 'exchange': <unbound Exchange network_fanout(fanout)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>}})]
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599308161744 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'validate_networks' message={u'args': {u'networks': [[u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', None]]}, u'version': u'1.0', u'method': u'validate_networks'}
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_219953bb685947cc964137cc09045fb9', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 messaging messaging.py _publish NBK: _publish routing_key=u'reply_219953bb685947cc964137cc09045fb9' exchange=u'reply_219953bb685947cc964137cc09045fb9' message.body='{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_219953bb685947cc964137cc09045fb9', immediate=False, exchange=u'reply_219953bb685947cc964137cc09045fb9') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_219953bb685947cc964137cc09045fb9', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 1, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 1, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>}
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 messaging messaging.py _publish NBK: _publish routing_key=u'reply_219953bb685947cc964137cc09045fb9' exchange=u'reply_219953bb685947cc964137cc09045fb9' message.body='{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_219953bb685947cc964137cc09045fb9', immediate=False, exchange=u'reply_219953bb685947cc964137cc09045fb9') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"437669a0fb8a4148b1599680c06acf43\\", \\"failure\\": null, \\"result\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='f0973986fd7245aa861d98a0fbfb6fa1' timeout=60 self.incoming=[{u'_unique_id': u'437669a0fb8a4148b1599680c06acf43', u'failure': None, u'result': None, u'_msg_id': u'f0973986fd7245aa861d98a0fbfb6fa1'}]
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='f0973986fd7245aa861d98a0fbfb6fa1' got incoming_msg_id=u'f0973986fd7245aa861d98a0fbfb6fa1' message_data={u'_unique_id': u'437669a0fb8a4148b1599680c06acf43', u'failure': None, u'result': None}
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='f0973986fd7245aa861d98a0fbfb6fa1' timeout=60 self.incoming=[]
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 connection connection.py _debug [Kombu connection:0x7fdfd65c7490] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_219953bb685947cc964137cc09045fb9 -> <Exchange reply_219953bb685947cc964137cc09045fb9(direct) bound to chan:1> -> reply_219953bb685947cc964137cc09045fb9 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7f51a00b8990>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>, 'kwargs': {'exclusive': False, 'name': 'reply_219953bb685947cc964137cc09045fb9', 'exchange': <unbound Exchange reply_219953bb685947cc964137cc09045fb9(direct)>, 'durable': True, 'routing_key': 'reply_219953bb685947cc964137cc09045fb9', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>}})]
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 2, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 2, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>}
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:07 10.35.0.14 nova-network 9862 140599325781680 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'validate_networks' took=0.010 got_reply=True exceptions=[] message={u'args': {u'networks': [[u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', None]]}, u'version': u'1.0', u'method': u'validate_networks'}
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"57edab6a15fb4fc186be8be53c82c1df\\", \\"failure\\": null, \\"_msg_id\\": \\"f0973986fd7245aa861d98a0fbfb6fa1\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='f0973986fd7245aa861d98a0fbfb6fa1' timeout=60 self.incoming=[{u'_unique_id': u'57edab6a15fb4fc186be8be53c82c1df', u'failure': None, u'_msg_id': u'f0973986fd7245aa861d98a0fbfb6fa1', u'result': None, u'ending': True}]
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='f0973986fd7245aa861d98a0fbfb6fa1' got incoming_msg_id=u'f0973986fd7245aa861d98a0fbfb6fa1' message_data={u'_unique_id': u'57edab6a15fb4fc186be8be53c82c1df', u'failure': None, u'result': None, u'ending': True}
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 client client.py call NBK: rpc client call method='validate_networks' target=<Target topic=network, version=1.0> took=0.070 timeout=60 got_result=True exceptions=[] kwargs={'networks': [(u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', None)]}
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 quota quota.py reserve Created reservations ['775379dc-0371-4d58-ac83-3c514fcc9189', 'f564fea1-1934-428a-8d79-ecce60d4e7b3', '72d91a0c-ff19-46df-abe4-8ca0443d6f2e'] reserve /usr/lib64/python2.7/site-packages/nova/quota.py:1272
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 http http.py log_curl_request curl -i -X HEAD -H 'X-Service-Catalog: [{"endpoints": [{"adminURL": "http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4", "region": "RegionOne", "internalURL": "http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4", "publicURL": "http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4"}], "endpoints_links": [], "type": "volume", "name": "Volume Service"}]' -H 'X-Identity-Status: Confirmed' -H 'X-Roles: _member_,Admin,ProjectAdmin,admin' -H 'User-Agent: python-glanceclient' -H 'X-Tenant-Id: e57d3cef3061401a900a9751b1fb0bc4' -H 'X-User-Id: 6bb7c6d04a8b43dc83cf9e851b92b3fb' -H 'X-Auth-Token: 925d73112a9f441e9f5599509efc94ee' -H 'Content-Type: application/octet-stream' http://10.35.0.13:9292/v1/images/8fb48209-3c53-4767-a4a4-a47d5f85db49 log_curl_request /usr/lib64/python2.7/site-packages/glanceclient/common/http.py:142
Jul 11 22:15:07 10.35.0.15 pound: 10.35.2.3:35357 10.35.2.7 - - [11/Jul/2014:22:15:07 +0000] "GET / HTTP/1.1" 300 948 "" "python-requests/2.3.0 CPython/2.7.6 Linux/3.10.47"
Jul 11 22:15:07 10.35.0.13 glance-api 2014-07-11 22:15:07.591 3431 INFO keystoneclient.middleware.auth_token [-] Auth Token confirmed use of v2.0 apis
Jul 11 22:15:07 10.35.0.15 pound: 10.35.2.3:35357 10.35.2.7 - - [11/Jul/2014:22:15:07 +0000] "POST /v2.0/tokens HTTP/1.1" 200 3232 "" "python-requests/2.3.0 CPython/2.7.6 Linux/3.10.47"
Jul 11 22:15:07 10.35.0.15 pound: 10.35.2.3:35357 10.35.2.7 - - [11/Jul/2014:22:15:07 +0000] "GET /v2.0/tokens/925d73112a9f441e9f5599509efc94ee HTTP/1.1" 200 3288 "" "python-requests/2.3.0 CPython/2.7.6 Linux/3.10.47"
Jul 11 22:15:07 10.35.0.13 glance-registry 2014-07-11 22:15:07.949 3423 INFO glance.registry.api.v1.images [20f63ce9-24c0-4cae-a251-edb7952349f2 6bb7c6d04a8b43dc83cf9e851b92b3fb e57d3cef3061401a900a9751b1fb0bc4 - - -] Successfully retrieved image 8fb48209-3c53-4767-a4a4-a47d5f85db49
Jul 11 22:15:07 10.35.0.13 glance-registry 2014-07-11 22:15:07.950 3423 INFO glance.wsgi.server [20f63ce9-24c0-4cae-a251-edb7952349f2 6bb7c6d04a8b43dc83cf9e851b92b3fb e57d3cef3061401a900a9751b1fb0bc4 - - -] 10.35.0.13 - - [11/Jul/2014 22:15:07] "GET /images/8fb48209-3c53-4767-a4a4-a47d5f85db49 HTTP/1.1" 200 933 0.017401
Jul 11 22:15:07 10.35.0.13 glance-api 2014-07-11 22:15:07.953 3431 INFO glance.wsgi.server [4d179f33-f79b-4b0a-94f2-03177df5229b 6bb7c6d04a8b43dc83cf9e851b92b3fb e57d3cef3061401a900a9751b1fb0bc4 - - -] 10.35.0.13 - - [11/Jul/2014 22:15:07] "HEAD /v1/images/8fb48209-3c53-4767-a4a4-a47d5f85db49 HTTP/1.1" 200 875 0.661783
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 http http.py log_http_response
HTTP/1.1 200 OK
content-length: 0
x-image-meta-id: 8fb48209-3c53-4767-a4a4-a47d5f85db49
date: Fri, 11 Jul 2014 22:15:07 GMT
x-image-meta-deleted: False
x-image-meta-checksum: eef3c287653e85dcd96a6fddc6cec9bc
x-image-meta-container_format: bare
x-image-meta-protected: False
x-image-meta-min_disk: 0
x-image-meta-created_at: 2014-07-11T22:07:23
x-image-meta-size: 18350080
x-image-meta-status: active
etag: eef3c287653e85dcd96a6fddc6cec9bc
location: http://10.35.0.13:9292/v1/images/8fb48209-3c53-4767-a4a4-a47d5f85db49
x-image-meta-is_public: True
x-image-meta-min_ram: 0
x-image-meta-owner: cfd0bf8abaa74c1bbabeb44095e44eaa
x-image-meta-updated_at: 2014-07-11T22:07:25
content-type: text/html; charset=UTF-8
x-openstack-request-id: req-4d179f33-f79b-4b0a-94f2-03177df5229b
x-image-meta-disk_format: qcow2
x-image-meta-name: Demo Image CirrOS 0.3.1 9
log_http_response /usr/lib64/python2.7/site-packages/glanceclient/common/http.py:152
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 http http.py log_curl_request curl -i -X HEAD -H 'X-Service-Catalog: [{"endpoints": [{"adminURL": "http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4", "region": "RegionOne", "internalURL": "http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4", "publicURL": "http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4"}], "endpoints_links": [], "type": "volume", "name": "Volume Service"}]' -H 'X-Identity-Status: Confirmed' -H 'X-Roles: _member_,Admin,ProjectAdmin,admin' -H 'User-Agent: python-glanceclient' -H 'X-Tenant-Id: e57d3cef3061401a900a9751b1fb0bc4' -H 'X-User-Id: 6bb7c6d04a8b43dc83cf9e851b92b3fb' -H 'X-Auth-Token: 925d73112a9f441e9f5599509efc94ee' -H 'Content-Type: application/octet-stream' http://10.35.0.13:9292/v1/images/8fb48209-3c53-4767-a4a4-a47d5f85db49 log_curl_request /usr/lib64/python2.7/site-packages/glanceclient/common/http.py:142
Jul 11 22:15:07 10.35.0.13 glance-registry 2014-07-11 22:15:07.982 3423 INFO glance.registry.api.v1.images [2f52b777-e93a-4f83-b8f7-bc93946f10a9 6bb7c6d04a8b43dc83cf9e851b92b3fb e57d3cef3061401a900a9751b1fb0bc4 - - -] Successfully retrieved image 8fb48209-3c53-4767-a4a4-a47d5f85db49
Jul 11 22:15:07 10.35.0.13 glance-registry 2014-07-11 22:15:07.983 3423 INFO glance.wsgi.server [2f52b777-e93a-4f83-b8f7-bc93946f10a9 6bb7c6d04a8b43dc83cf9e851b92b3fb e57d3cef3061401a900a9751b1fb0bc4 - - -] 10.35.0.13 - - [11/Jul/2014 22:15:07] "GET /images/8fb48209-3c53-4767-a4a4-a47d5f85db49 HTTP/1.1" 200 933 0.017328
Jul 11 22:15:07 10.35.0.13 glance-api 2014-07-11 22:15:07.987 3431 INFO glance.wsgi.server [74202132-a998-4123-af0d-6555879942d2 6bb7c6d04a8b43dc83cf9e851b92b3fb e57d3cef3061401a900a9751b1fb0bc4 - - -] 10.35.0.13 - - [11/Jul/2014 22:15:07] "HEAD /v1/images/8fb48209-3c53-4767-a4a4-a47d5f85db49 HTTP/1.1" 200 875 0.025457
Jul 11 22:15:07 10.35.0.13 nova-api 3620 139988495936240 http http.py log_http_response
HTTP/1.1 200 OK
content-length: 0
x-image-meta-id: 8fb48209-3c53-4767-a4a4-a47d5f85db49
date: Fri, 11 Jul 2014 22:15:07 GMT
x-image-meta-deleted: False
x-image-meta-checksum: eef3c287653e85dcd96a6fddc6cec9bc
x-image-meta-container_format: bare
x-image-meta-protected: False
x-image-meta-min_disk: 0
x-image-meta-created_at: 2014-07-11T22:07:23
x-image-meta-size: 18350080
x-image-meta-status: active
etag: eef3c287653e85dcd96a6fddc6cec9bc
location: http://10.35.0.13:9292/v1/images/8fb48209-3c53-4767-a4a4-a47d5f85db49
x-image-meta-is_public: True
x-image-meta-min_ram: 0
x-image-meta-owner: cfd0bf8abaa74c1bbabeb44095e44eaa
x-image-meta-updated_at: 2014-07-11T22:07:25
content-type: text/html; charset=UTF-8
x-openstack-request-id: req-74202132-a998-4123-af0d-6555879942d2
x-image-meta-disk_format: qcow2
x-image-meta-name: Demo Image CirrOS 0.3.1 9
log_http_response /usr/lib64/python2.7/site-packages/glanceclient/common/http.py:152
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 client client.py call NBK: rpc client call method='select_destinations' target=<Target topic=scheduler, version=3.0> timeout=60 rpc_response_timeout=60 kwargs={'request_spec': {'num_instances': 1, 'block_device_mapping': [{'guest_format': None, 'boot_index': 0L, 'connection_info': None, 'volume_id': None, 'source_type': u'image', 'device_type': u'disk', 'snapshot_id': None, 'image_id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', 'no_device': None, 'device_name': u'/dev/vda', 'disk_bus': None, 'volume_size': None, 'destination_type': u'local', 'delete_on_termination': True}], 'image': {'status': u'active', 'name': u'Demo Image CirrOS 0.3.1 9', 'deleted': False, 'container_format': u'bare', 'created_at': '2014-07-11T22:07:23.000000', 'disk_format': u'qcow2', 'updated_at': '2014-07-11T22:07:25.000000', 'properties': {}, 'min_disk': 0, 'min_ram': 0, 'checksum': u'eef3c287653e85dcd96a6fddc6cec9bc', 'owner': u'cfd0bf8abaa74c1bbabeb44095e44eaa', 'is_public': True, 'deleted_at': None, 'id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', 'size': 18350080}, 'instance_type': {'root_gb': 1, 'name': 'm1.tiny', 'ephemeral_gb': 0, 'memory_mb': 512, 'vcpus': 1, 'swap': 0, 'rxtx_factor': 1.0, 'flavorid': '1', 'vcpu_weight': None, 'id': 2}, 'instance_properties': {'vm_state': u'building', 'internal_id': None, 'availability_zone': None, 'terminated_at': None, 'ephemeral_gb': 0L, 'instance_type_id': 2L, 'user_data': u'', 'cleaned': 0L, 'vm_mode': None, 'deleted_at': None, 'reservation_id': u'r-k4rso6fz', 'id': 5L, 'security_groups': ['default'], 'disable_terminate': False, 'root_device_name': u'/dev/vda', 'user_id': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', 'uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', 'default_swap_device': None, 'hostname': u'vm-93808201693-0-clone', 'launched_on': None, 'display_description': u'vm-93808201693', 'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\n', 'deleted': 0L, 'config_drive': u'', 'power_state': 0L, 'default_ephemeral_device': None, 'progress': None, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'launched_at': None, 'scheduled_at': None, 'node': None, 'ramdisk_id': u'', 'access_ip_v6': None, 'access_ip_v4': None, 'kernel_id': u'', 'key_name': u'keypair-8197170298', 'updated_at': None, 'host': None, 'root_gb': 1L, 'display_name': u'vm-93808201693-0-clone', 'system_metadata': {u'instance_type_name': u'm1.tiny', u'images': u'5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382', u'image_min_ram': u'0', u'instance_type_ephemeral_gb': u'0', u'clean_attempts': u'1', u'network_allocated': u'True', u'image_base_image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'gc_src_host': u'10.35.0.14', u'logical_volumes': u'', u'instance_type_vcpu_weight': None, u'instance_type_root_gb': u'1', u'instance_type_id': u'2', u'instance_type_rxtx_factor': u'1.0', u'gc_dst_host': u'10.35.0.15', u'instance_type_vcpus': u'1', u'image_disk_format': u'qcow2', u'instance_type_memory_mb': u'512', u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e', u'attached_networks': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'instance_type_flavorid': u'1', u'image_container_format': u'bare', u'image_min_disk': u'1', u'instance_type_swap': u'0'}, 'task_state': u'scheduling', 'shutdown_terminate': False, 'cell_name': None, 'ephemeral_key_uuid': None, 'locked': False, 'name': 'instance-00000005', 'created_at': datetime.datetime(2014, 7, 11, 22, 15, 7), 'locked_by': None, 'launch_index': 0L, 'memory_mb': 512L, 'vcpus': 1L, 'image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', 'architecture': None, 'auto_disk_config': False, 'os_type': None, 'metadata': {u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e'}}, 'security_group': ['default'], 'instance_uuids': [u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c']}, 'filter_properties': {'scheduler_hints': {}}}
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange='nova', auto_delete=False, passive=False, arguments=None, type='topic', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {'ttl': 60000}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', headers={'ttl': 60000}, properties={'delivery_mode': 2}
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 messaging messaging.py _publish NBK: _publish routing_key='scheduler' exchange='nova' message.body='{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='scheduler', immediate=False, exchange='nova') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 connection connection.py _debug [Kombu connection:0x7f51a033c150] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'scheduler', 'delivery_tag': 1, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='a631ea46cb914b138367dc563a8e89dd' timeout=60 self.incoming=[]
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'scheduler', 'delivery_tag': 1, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>}
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_219953bb685947cc964137cc09045fb9 -> <Exchange reply_219953bb685947cc964137cc09045fb9(direct) bound to chan:1> -> reply_219953bb685947cc964137cc09045fb9 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7f51a00b8990>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>, 'kwargs': {'exclusive': False, 'name': 'reply_219953bb685947cc964137cc09045fb9', 'exchange': <unbound Exchange reply_219953bb685947cc964137cc09045fb9(direct)>, 'durable': True, 'routing_key': 'reply_219953bb685947cc964137cc09045fb9', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>}})]
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:08 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.370 sec retry=1 got_ret=True exceptions=['connection_errors: [Errno 104] Connection reset by peer']
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"request_spec\\": {\\"num_instances\\": 1, \\"block_device_mapping\\": [{\\"guest_format\\": null, \\"boot_index\\": 0, \\"connection_info\\": null, \\"volume_id\\": null, \\"source_type\\": \\"image\\", \\"device_type\\": \\"disk\\", \\"snapshot_id\\": null, \\"image_id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"no_device\\": null, \\"device_name\\": \\"/dev/vda\\", \\"disk_bus\\": null, \\"volume_size\\": null, \\"destination_type\\": \\"local\\", \\"delete_on_termination\\": true}], \\"image\\": {\\"status\\": \\"active\\", \\"name\\": \\"Demo Image CirrOS 0.3.1 9\\", \\"deleted\\": false, \\"container_format\\": \\"bare\\", \\"created_at\\": \\"2014-07-11T22:07:23.000000\\", \\"disk_format\\": \\"qcow2\\", \\"updated_at\\": \\"2014-07-11T22:07:25.000000\\", \\"properties\\": {}, \\"min_disk\\": 0, \\"min_ram\\": 0, \\"checksum\\": \\"eef3c287653e85dcd96a6fddc6cec9bc\\", \\"owner\\": \\"cfd0bf8abaa74c1bbabeb44095e44eaa\\", \\"is_public\\": true, \\"deleted_at\\": null, \\"id\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"size\\": 18350080}, \\"instance_type\\": {\\"root_gb\\": 1, \\"name\\": \\"m1.tiny\\", \\"ephemeral_gb\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"swap\\": 0, \\"rxtx_factor\\": 1.0, \\"flavorid\\": \\"1\\", \\"vcpu_weight\\": null, \\"id\\": 2}, \\"instance_properties\\": {\\"vm_state\\": \\"building\\", \\"internal_id\\": null, \\"availability_zone\\": null, \\"terminated_at\\": null, \\"ephemeral_gb\\": 0, \\"instance_type_id\\": 2, \\"user_data\\": \\"\\", \\"cleaned\\": 0, \\"vm_mode\\": null, \\"deleted_at\\": null, \\"reservation_id\\": \\"r-k4rso6fz\\", \\"id\\": 5, \\"security_groups\\": [\\"default\\"], \\"disable_terminate\\": false, \\"root_device_name\\": \\"/dev/vda\\", \\"user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"default_swap_device\\": null, \\"hostname\\": \\"vm-93808201693-0-clone\\", \\"launched_on\\": null, \\"display_description\\": \\"vm-93808201693\\", \\"key_data\\": \\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\\\\n\\", \\"deleted\\": 0, \\"config_drive\\": \\"\\", \\"power_state\\": 0, \\"default_ephemeral_device\\": null, \\"progress\\": null, \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"launched_at\\": null, \\"scheduled_at\\": null, \\"node\\": null, \\"ramdisk_id\\": \\"\\", \\"access_ip_v6\\": null, \\"access_ip_v4\\": null, \\"kernel_id\\": \\"\\", \\"key_name\\": \\"keypair-8197170298\\", \\"updated_at\\": null, \\"host\\": null, \\"root_gb\\": 1, \\"display_name\\": \\"vm-93808201693-0-clone\\", \\"system_metadata\\": {\\"instance_type_name\\": \\"m1.tiny\\", \\"images\\": \\"5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382\\", \\"image_min_ram\\": \\"0\\", \\"instance_type_ephemeral_gb\\": \\"0\\", \\"clean_attempts\\": \\"1\\", \\"network_allocated\\": \\"True\\", \\"image_base_image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"gc_src_host\\": \\"10.35.0.14\\", \\"logical_volumes\\": \\"\\", \\"instance_type_vcpu_weight\\": null, \\"instance_type_root_gb\\": \\"1\\", \\"instance_type_id\\": \\"2\\", \\"instance_type_rxtx_factor\\": \\"1.0\\", \\"gc_dst_host\\": \\"10.35.0.15\\", \\"instance_type_vcpus\\": \\"1\\", \\"image_disk_format\\": \\"qcow2\\", \\"instance_type_memory_mb\\": \\"512\\", \\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\", \\"attached_networks\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"instance_type_flavorid\\": \\"1\\", \\"image_container_format\\": \\"bare\\", \\"image_min_disk\\": \\"1\\", \\"instance_type_swap\\": \\"0\\"}, \\"task_state\\": \\"scheduling\\", \\"shutdown_terminate\\": false, \\"cell_name\\": null, \\"ephemeral_key_uuid\\": null, \\"locked\\": false, \\"name\\": \\"instance-00000005\\", \\"created_at\\": \\"2014-07-11T22:15:07.000000\\", \\"locked_by\\": null, \\"launch_index\\": 0, \\"memory_mb\\": 512, \\"vcpus\\": 1, \\"image_ref\\": \\"8fb48209-3c53-4767-a4a4-a47d5f85db49\\", \\"architecture\\": null, \\"auto_disk_config\\": false, \\"os_type\\": null, \\"metadata\\": {\\"launched_from\\": \\"adb82c12-7ea3-40e1-b485-ccf1e91d3b8e\\"}}, \\"security_group\\": [\\"default\\"], \\"instance_uuids\\": [\\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\"]}, \\"filter_properties\\": {\\"scheduler_hints\\": {}}}, \\"_unique_id\\": \\"cb209b7e6b654020a104c6611dd04941\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_219953bb685947cc964137cc09045fb9\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"3.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"select_destinations\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue scheduler -> <Exchange nova(topic) bound to chan:1> -> scheduler bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f248e23b810>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>, 'kwargs': {'exclusive': False, 'name': 'scheduler', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'scheduler', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>}}), TopicConsumer({'queue': <Queue scheduler.10.35.0.13 -> <Exchange nova(topic) bound to chan:1> -> scheduler.10.35.0.13 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f248e23b810>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>, 'kwargs': {'exclusive': False, 'name': 'scheduler.10.35.0.13', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'scheduler.10.35.0.13', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>}}), FanoutConsumer({'queue': <Queue scheduler_fanout_8b3d4052eed044ce985ad934cc1aab93 -> <Exchange scheduler_fanout(fanout) bound to chan:1> -> scheduler bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f248e23b810>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>, 'kwargs': {'exclusive': False, 'name': 'scheduler_fanout_8b3d4052eed044ce985ad934cc1aab93', 'exchange': <unbound Exchange scheduler_fanout(fanout)>, 'durable': True, 'routing_key': 'scheduler', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f248e6fb350>}})]
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794942295408 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'select_destinations' message={u'args': {u'request_spec': {u'num_instances': 1, u'block_device_mapping': [{u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True, u'image_id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'connection_info': None, u'no_device': None, u'snapshot_id': None, u'device_name': u'/dev/vda', u'disk_bus': None, u'source_type': u'image', u'device_type': u'disk', u'volume_id': None, u'destination_type': u'local', u'volume_size': None}], u'image': {u'status': u'active', u'name': u'Demo Image CirrOS 0.3.1 9', u'deleted': False, u'container_format': u'bare', u'created_at': u'2014-07-11T22:07:23.000000', u'disk_format': u'qcow2', u'updated_at': u'2014-07-11T22:07:25.000000', u'id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'owner': u'cfd0bf8abaa74c1bbabeb44095e44eaa', u'min_ram': 0, u'checksum': u'eef3c287653e85dcd96a6fddc6cec9bc', u'min_disk': 0, u'is_public': True, u'deleted_at': None, u'properties': {}, u'size': 18350080}, u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb': 0, u'memory_mb': 512, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2}, u'instance_properties': {u'vm_state': u'building', u'internal_id': None, u'availability_zone': None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 2, u'user_data': u'', u'cleaned': 0, u'vm_mode': None, u'deleted_at': None, u'reservation_id': u'r-k4rso6fz', u'id': 5, u'security_groups': [u'default'], u'disable_terminate': False, u'user_id': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', u'uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'default_swap_device': None, u'hostname': u'vm-93808201693-0-clone', u'launched_on': None, u'display_description': u'vm-93808201693', u'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\n', u'kernel_id': u'', u'power_state': 0, u'default_ephemeral_device': None, u'progress': None, u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'launched_at': None, u'config_drive': u'', u'node': None, u'ramdisk_id': u'', u'access_ip_v6': None, u'access_ip_v4': None, u'deleted': 0, u'key_name': u'keypair-8197170298', u'updated_at': None, u'host': None, u'ephemeral_key_uuid': None, u'architecture': None, u'display_name': u'vm-93808201693-0-clone', u'system_metadata': {u'instance_type_id': u'2', u'images': u'5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382', u'image_min_ram': u'0', u'instance_type_ephemeral_gb': u'0', u'clean_attempts': u'1', u'network_allocated': u'True', u'image_base_image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'gc_src_host': u'10.35.0.14', u'logical_volumes': u'', u'instance_type_vcpu_weight': None, u'instance_type_root_gb': u'1', u'instance_type_name': u'm1.tiny', u'instance_type_rxtx_factor': u'1.0', u'gc_dst_host': u'10.35.0.15', u'instance_type_vcpus': u'1', u'image_disk_format': u'qcow2', u'instance_type_memory_mb': u'512', u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e', u'attached_networks': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'instance_type_flavorid': u'1', u'image_container_format': u'bare', u'image_min_disk': u'1', u'instance_type_swap': u'0'}, u'task_state': u'scheduling', u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1, u'locked': False, u'name': u'instance-00000005', u'created_at': u'2014-07-11T22:15:07.000000', u'locked_by': None, u'launch_index': 0, u'metadata': {u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e'}, u'memory_mb': 512, u'vcpus': 1, u'image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'root_device_name': u'/dev/vda', u'auto_disk_config': False, u'os_type': None, u'scheduled_at': None}, u'security_group': [u'default'], u'instance_uuids': [u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c']}, u'filter_properties': {u'scheduler_hints': {}}}, u'version': u'3.0', u'method': u'select_destinations'}
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Starting with 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:70
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 retry_filter retry_filter.py host_passes Host [u'10.35.0.14', u'node-10-35-0-14'] passes. Previously tried hosts: [] host_passes /usr/lib64/python2.7/site-packages/nova/scheduler/filters/retry_filter.py:45
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 retry_filter retry_filter.py host_passes Host [u'10.35.0.15', u'node-10-35-0-15'] passes. Previously tried hosts: [] host_passes /usr/lib64/python2.7/site-packages/nova/scheduler/filters/retry_filter.py:45
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 retry_filter retry_filter.py host_passes Host [u'10.35.0.13', u'node-10-35-0-13'] passes. Previously tried hosts: [] host_passes /usr/lib64/python2.7/site-packages/nova/scheduler/filters/retry_filter.py:45
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter RetryFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter AvailabilityZoneFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter RamFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 api api.py service_is_up Check if the given member [{u'binary': u'nova-compute', u'deleted': 0L, u'created_at': datetime.datetime(2014, 7, 11, 22, 6, 17), u'updated_at': datetime.datetime(2014, 7, 11, 22, 15, 3), u'report_count': 52L, u'topic': u'compute', u'host': u'10.35.0.14', u'disabled': False, u'deleted_at': None, u'disabled_reason': None, u'id': 12L}] is part of the ServiceGroup, is up service_is_up /usr/lib64/python2.7/site-packages/nova/servicegroup/api.py:116
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 db db.py is_up DB_Driver.is_up last_heartbeat = 2014-07-11 22:15:03 elapsed = 5.021458 is_up /usr/lib64/python2.7/site-packages/nova/servicegroup/drivers/db.py:71
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 api api.py service_is_up Check if the given member [{u'binary': u'nova-compute', u'deleted': 0L, u'created_at': datetime.datetime(2014, 7, 11, 22, 6, 16), u'updated_at': datetime.datetime(2014, 7, 11, 22, 15, 2), u'report_count': 52L, u'topic': u'compute', u'host': u'10.35.0.15', u'disabled': False, u'deleted_at': None, u'disabled_reason': None, u'id': 8L}] is part of the ServiceGroup, is up service_is_up /usr/lib64/python2.7/site-packages/nova/servicegroup/api.py:116
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 db db.py is_up DB_Driver.is_up last_heartbeat = 2014-07-11 22:15:02 elapsed = 6.022046 is_up /usr/lib64/python2.7/site-packages/nova/servicegroup/drivers/db.py:71
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 api api.py service_is_up Check if the given member [{u'binary': u'nova-compute', u'deleted': 0L, u'created_at': datetime.datetime(2014, 7, 11, 22, 6, 15), u'updated_at': datetime.datetime(2014, 7, 11, 22, 15, 1), u'report_count': 52L, u'topic': u'compute', u'host': u'10.35.0.13', u'disabled': False, u'deleted_at': None, u'disabled_reason': None, u'id': 5L}] is part of the ServiceGroup, is up service_is_up /usr/lib64/python2.7/site-packages/nova/servicegroup/api.py:116
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 db db.py is_up DB_Driver.is_up last_heartbeat = 2014-07-11 22:15:01 elapsed = 7.022622 is_up /usr/lib64/python2.7/site-packages/nova/servicegroup/drivers/db.py:71
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter ComputeFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 extension extension.py _load_plugins found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib64/python2.7/site-packages/stevedore/extension.py:156
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 extension extension.py _load_plugins found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib64/python2.7/site-packages/stevedore/extension.py:156
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter DifferentHostFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 extension extension.py _load_plugins found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib64/python2.7/site-packages/stevedore/extension.py:156
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 extension extension.py _load_plugins found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib64/python2.7/site-packages/stevedore/extension.py:156
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter SameHostFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 extension extension.py _load_plugins found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib64/python2.7/site-packages/stevedore/extension.py:156
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 extension extension.py _load_plugins found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib64/python2.7/site-packages/stevedore/extension.py:156
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter GroupAntiAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter ComputeCapabilitiesFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filters filters.py get_filtered_objects Filter ImagePropertiesFilter returned 3 host(s) get_filtered_objects /usr/lib64/python2.7/site-packages/nova/filters.py:88
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filter_scheduler filter_scheduler.py _schedule Filtered [(10.35.0.14, node-10-35-0-14) ram:47729 disk:1658880 io_ops:0 instances:1, (10.35.0.15, node-10-35-0-15) ram:46705 disk:1656832 io_ops:0 instances:2, (10.35.0.13, node-10-35-0-13) ram:47217 disk:1657856 io_ops:0 instances:1] _schedule /usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py:332
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 filter_scheduler filter_scheduler.py _schedule Weighed [WeighedHost [host: 10.35.0.14, weight: 1.0], WeighedHost [host: 10.35.0.13, weight: 0.989272769176], WeighedHost [host: 10.35.0.15, weight: 0.978545538352]] _schedule /usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py:337
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_219953bb685947cc964137cc09045fb9', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 impl_rabbit impl_rabbit.py _error_callback Failed to publish message to topic 'reply_219953bb685947cc964137cc09045fb9': [Errno 104] Connection reset by peer
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 698, in ensure
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit ret = method(*args, **kwargs)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 831, in _publish
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 424, in __init__
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 390, in __init__
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 398, in reconnect
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 85, in __init__
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 224, in revive
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit self.declare()
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 105, in declare
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/kombu/entity.py", line 172, in declare
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit ret = self.channel.exchange_declare(**args)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/kombu/utils/debug.py", line 57, in __wrapped
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit ret = meth(*args, **kwargs)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/amqp/channel.py", line 613, in exchange_declare
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((40, 10), args)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 307, in sendall
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 293, in send
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2014-07-11 22:15:08.051 3514 TRACE oslo.messaging._drivers.impl_rabbit
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 impl_rabbit impl_rabbit.py ensure NBK: ensure reconnecting...
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 impl_rabbit impl_rabbit.py _connect Reconnecting to AMQP server on 10.35.0.3:5672
Jul 11 22:15:08 10.35.0.13 nova-scheduler 3514 139794943988912 impl_rabbit impl_rabbit.py _connect Delaying reconnect for 1.0 seconds...
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 connection connection.py _debug [Kombu connection:0x7f248e57b210] closed _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 connection connection.py _debug [Kombu connection:0x7f248e57b210] establishing connection... _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:09 10.35.0.15 rabbitmq.log: =INFO REPORT==== 11-Jul-2014::22:15:09 ===
Jul 11 22:15:09 10.35.0.15 rabbitmq.log: accepting AMQP connection <0.1452.0> (10.35.0.13:50637 -> 10.35.0.3:5672)
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 connection connection.py _debug [Kombu connection:0x7f248e57b210] connection established: <kombu.transport.pyamqp.Connection object at 0x7f248e587b90> _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 connection connection.py _debug [Kombu connection:0x7f248e57b210] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 impl_rabbit impl_rabbit.py _connect Connected to AMQP server on 10.35.0.3:5672
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_219953bb685947cc964137cc09045fb9', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 messaging messaging.py _publish NBK: _publish routing_key=u'reply_219953bb685947cc964137cc09045fb9' exchange=u'reply_219953bb685947cc964137cc09045fb9' message.body='{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_219953bb685947cc964137cc09045fb9', immediate=False, exchange=u'reply_219953bb685947cc964137cc09045fb9') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.020 sec retry=1 got_ret=True exceptions=['connection_errors: [Errno 104] Connection reset by peer']
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_219953bb685947cc964137cc09045fb9', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 3, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 3, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>}
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 messaging messaging.py _publish NBK: _publish routing_key=u'reply_219953bb685947cc964137cc09045fb9' exchange=u'reply_219953bb685947cc964137cc09045fb9' message.body='{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_219953bb685947cc964137cc09045fb9', immediate=False, exchange=u'reply_219953bb685947cc964137cc09045fb9') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"ea28727f14654d3bbd94e0cd84759e62\\", \\"failure\\": null, \\"result\\": [{\\"host\\": \\"10.35.0.14\\", \\"nodename\\": \\"node-10-35-0-14\\", \\"limits\\": {\\"memory_mb\\": 72361.5}}], \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='a631ea46cb914b138367dc563a8e89dd' timeout=60 self.incoming=[{u'_unique_id': u'ea28727f14654d3bbd94e0cd84759e62', u'failure': None, u'result': [{u'host': u'10.35.0.14', u'nodename': u'node-10-35-0-14', u'limits': {u'memory_mb': 72361.5}}], u'_msg_id': u'a631ea46cb914b138367dc563a8e89dd'}]
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='a631ea46cb914b138367dc563a8e89dd' got incoming_msg_id=u'a631ea46cb914b138367dc563a8e89dd' message_data={u'_unique_id': u'ea28727f14654d3bbd94e0cd84759e62', u'failure': None, u'result': [{u'host': u'10.35.0.14', u'nodename': u'node-10-35-0-14', u'limits': {u'memory_mb': 72361.5}}]}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='a631ea46cb914b138367dc563a8e89dd' timeout=60 self.incoming=[]
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 connection connection.py _debug [Kombu connection:0x7f248e57b210] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_219953bb685947cc964137cc09045fb9 -> <Exchange reply_219953bb685947cc964137cc09045fb9(direct) bound to chan:1> -> reply_219953bb685947cc964137cc09045fb9 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7f51a00b8990>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>, 'kwargs': {'exclusive': False, 'name': 'reply_219953bb685947cc964137cc09045fb9', 'exchange': <unbound Exchange reply_219953bb685947cc964137cc09045fb9(direct)>, 'durable': True, 'routing_key': 'reply_219953bb685947cc964137cc09045fb9', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>}})]
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 4, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_219953bb685947cc964137cc09045fb9', 'delivery_tag': 4, 'exchange': u'reply_219953bb685947cc964137cc09045fb9'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f51a00b8710>}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:09 10.35.0.13 nova-scheduler 3514 139794943988912 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'select_destinations' took=0.070 got_reply=True exceptions=[] message={u'args': {u'request_spec': {u'num_instances': 1, u'block_device_mapping': [{u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True, u'image_id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'connection_info': None, u'no_device': None, u'snapshot_id': None, u'device_name': u'/dev/vda', u'disk_bus': None, u'source_type': u'image', u'device_type': u'disk', u'volume_id': None, u'destination_type': u'local', u'volume_size': None}], u'image': {u'status': u'active', u'name': u'Demo Image CirrOS 0.3.1 9', u'deleted': False, u'container_format': u'bare', u'created_at': u'2014-07-11T22:07:23.000000', u'disk_format': u'qcow2', u'updated_at': u'2014-07-11T22:07:25.000000', u'id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'owner': u'cfd0bf8abaa74c1bbabeb44095e44eaa', u'min_ram': 0, u'checksum': u'eef3c287653e85dcd96a6fddc6cec9bc', u'min_disk': 0, u'is_public': True, u'deleted_at': None, u'properties': {}, u'size': 18350080}, u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb': 0, u'memory_mb': 512, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2}, u'instance_properties': {u'vm_state': u'building', u'internal_id': None, u'availability_zone': None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 2, u'user_data': u'', u'cleaned': 0, u'vm_mode': None, u'deleted_at': None, u'reservation_id': u'r-k4rso6fz', u'id': 5, u'security_groups': [u'default'], u'disable_terminate': False, u'user_id': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', u'uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'default_swap_device': None, u'hostname': u'vm-93808201693-0-clone', u'launched_on': None, u'display_description': u'vm-93808201693', u'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\n', u'kernel_id': u'', u'power_state': 0, u'default_ephemeral_device': None, u'progress': None, u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'launched_at': None, u'config_drive': u'', u'node': None, u'ramdisk_id': u'', u'access_ip_v6': None, u'access_ip_v4': None, u'deleted': 0, u'key_name': u'keypair-8197170298', u'updated_at': None, u'host': None, u'ephemeral_key_uuid': None, u'architecture': None, u'display_name': u'vm-93808201693-0-clone', u'system_metadata': {u'instance_type_id': u'2', u'images': u'5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382', u'image_min_ram': u'0', u'instance_type_ephemeral_gb': u'0', u'clean_attempts': u'1', u'network_allocated': u'True', u'image_base_image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'gc_src_host': u'10.35.0.14', u'logical_volumes': u'', u'instance_type_vcpu_weight': None, u'instance_type_root_gb': u'1', u'instance_type_name': u'm1.tiny', u'instance_type_rxtx_factor': u'1.0', u'gc_dst_host': u'10.35.0.15', u'instance_type_vcpus': u'1', u'image_disk_format': u'qcow2', u'instance_type_memory_mb': u'512', u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e', u'attached_networks': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'instance_type_flavorid': u'1', u'image_container_format': u'bare', u'image_min_disk': u'1', u'instance_type_swap': u'0'}, u'task_state': u'scheduling', u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1, u'locked': False, u'name': u'instance-00000005', u'created_at': u'2014-07-11T22:15:07.000000', u'locked_by': None, u'launch_index': 0, u'metadata': {u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e'}, u'memory_mb': 512, u'vcpus': 1, u'image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'root_device_name': u'/dev/vda', u'auto_disk_config': False, u'os_type': None, u'scheduled_at': None}, u'security_group': [u'default'], u'instance_uuids': [u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c']}, u'filter_properties': {'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb': 0, u'memory_mb': 512, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2}, 'config_options': {}, 'retry': {'num_attempts': 1, 'hosts': []}, 'context': <nova.context.RequestContext object at 0x7f248e480a10>, 'request_spec': {u'num_instances': 1, u'block_device_mapping': [{u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True, u'image_id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'connection_info': None, u'no_device': None, u'snapshot_id': None, u'device_name': u'/dev/vda', u'disk_bus': None, u'source_type': u'image', u'device_type': u'disk', u'volume_id': None, u'destination_type': u'local', u'volume_size': None}], u'image': {u'status': u'active', u'name': u'Demo Image CirrOS 0.3.1 9', u'deleted': False, u'container_format': u'bare', u'created_at': u'2014-07-11T22:07:23.000000', u'disk_format': u'qcow2', u'updated_at': u'2014-07-11T22:07:25.000000', u'id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'owner': u'cfd0bf8abaa74c1bbabeb44095e44eaa', u'min_ram': 0, u'checksum': u'eef3c287653e85dcd96a6fddc6cec9bc', u'min_disk': 0, u'is_public': True, u'deleted_at': None, u'properties': {}, u'size': 18350080}, u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb': 0, u'memory_mb': 512, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2}, u'instance_properties': {u'vm_state': u'building', u'internal_id': None, u'availability_zone': None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 2, u'user_data': u'', u'cleaned': 0, u'vm_mode': None, u'deleted_at': None, u'reservation_id': u'r-k4rso6fz', u'id': 5, u'security_groups': [u'default'], u'disable_terminate': False, u'user_id': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', u'uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'default_swap_device': None, u'hostname': u'vm-93808201693-0-clone', u'launched_on': None, u'display_description': u'vm-93808201693', u'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\n', u'kernel_id': u'', u'power_state': 0, u'default_ephemeral_device': None, u'progress': None, u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'launched_at': None, u'config_drive': u'', u'node': None, u'ramdisk_id': u'', u'access_ip_v6': None, u'access_ip_v4': None, u'deleted': 0, u'key_name': u'keypair-8197170298', u'updated_at': None, u'host': None, u'ephemeral_key_uuid': None, u'architecture': None, u'display_name': u'vm-93808201693-0-clone', u'system_metadata': {u'instance_type_id': u'2', u'images': u'5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382', u'image_min_ram': u'0', u'instance_type_ephemeral_gb': u'0', u'clean_attempts': u'1', u'network_allocated': u'True', u'image_base_image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'gc_src_host': u'10.35.0.14', u'logical_volumes': u'', u'instance_type_vcpu_weight': None, u'instance_type_root_gb': u'1', u'instance_type_name': u'm1.tiny', u'instance_type_rxtx_factor': u'1.0', u'gc_dst_host': u'10.35.0.15', u'instance_type_vcpus': u'1', u'image_disk_format': u'qcow2', u'instance_type_memory_mb': u'512', u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e', u'attached_networks': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'instance_type_flavorid': u'1', u'image_container_format': u'bare', u'image_min_disk': u'1', u'instance_type_swap': u'0'}, u'task_state': u'scheduling', u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1, u'locked': False, u'name': u'instance-00000005', u'created_at': u'2014-07-11T22:15:07.000000', u'locked_by': None, u'launch_index': 0, u'metadata': {u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e'}, u'memory_mb': 512, u'vcpus': 1, u'image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'root_device_name': u'/dev/vda', u'auto_disk_config': False, u'os_type': None, u'scheduled_at': None}, u'security_group': [u'default'], u'instance_uuids': [u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c']}, 'os_type': None, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'scheduler_hints': {}}}, u'version': u'3.0', u'method': u'select_destinations'}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"85aa0a2ef92948d1a31b0d51f2fcc963\\", \\"failure\\": null, \\"_msg_id\\": \\"a631ea46cb914b138367dc563a8e89dd\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='a631ea46cb914b138367dc563a8e89dd' timeout=60 self.incoming=[{u'_unique_id': u'85aa0a2ef92948d1a31b0d51f2fcc963', u'failure': None, u'_msg_id': u'a631ea46cb914b138367dc563a8e89dd', u'result': None, u'ending': True}]
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='a631ea46cb914b138367dc563a8e89dd' got incoming_msg_id=u'a631ea46cb914b138367dc563a8e89dd' message_data={u'_unique_id': u'85aa0a2ef92948d1a31b0d51f2fcc963', u'failure': None, u'result': None, u'ending': True}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 client client.py call NBK: rpc client call method='select_destinations' target=<Target topic=scheduler, version=3.0> took=0.010 timeout=60 got_result=True exceptions=[] kwargs={'request_spec': {'num_instances': 1, 'block_device_mapping': [{'guest_format': None, 'boot_index': 0L, 'connection_info': None, 'volume_id': None, 'source_type': u'image', 'device_type': u'disk', 'snapshot_id': None, 'image_id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', 'no_device': None, 'device_name': u'/dev/vda', 'disk_bus': None, 'volume_size': None, 'destination_type': u'local', 'delete_on_termination': True}], 'image': {'status': u'active', 'name': u'Demo Image CirrOS 0.3.1 9', 'deleted': False, 'container_format': u'bare', 'created_at': '2014-07-11T22:07:23.000000', 'disk_format': u'qcow2', 'updated_at': '2014-07-11T22:07:25.000000', 'properties': {}, 'min_disk': 0, 'min_ram': 0, 'checksum': u'eef3c287653e85dcd96a6fddc6cec9bc', 'owner': u'cfd0bf8abaa74c1bbabeb44095e44eaa', 'is_public': True, 'deleted_at': None, 'id': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', 'size': 18350080}, 'instance_type': {'root_gb': 1, 'name': 'm1.tiny', 'ephemeral_gb': 0, 'memory_mb': 512, 'vcpus': 1, 'swap': 0, 'rxtx_factor': 1.0, 'flavorid': '1', 'vcpu_weight': None, 'id': 2}, 'instance_properties': {'vm_state': u'building', 'internal_id': None, 'availability_zone': None, 'terminated_at': None, 'ephemeral_gb': 0L, 'instance_type_id': 2L, 'user_data': u'', 'cleaned': 0L, 'vm_mode': None, 'deleted_at': None, 'reservation_id': u'r-k4rso6fz', 'id': 5L, 'security_groups': ['default'], 'disable_terminate': False, 'root_device_name': u'/dev/vda', 'user_id': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', 'uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', 'default_swap_device': None, 'hostname': u'vm-93808201693-0-clone', 'launched_on': None, 'display_description': u'vm-93808201693', 'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXnMsgpUaR5CxSyOI4ACBSp6/SsPHmJ7wXu+mk1NN9TwjlHJQ6ILXeDTSnrxPZPGdIOkok0aA+mhdJ5tr1vlH/pM7ONKP5wll6RHmteRpS0Tz69BG5hVnhwyWA/jBFRbu/0CzlCfI9264vbFKUbbHe33cGposQm9pMzj18LEGWz9o6/bgq1+dksN2S09s2+/pc8UpDjMKnMyMxMyoI6eh4tOTd5DE8o+MXxBre9SWmq/nVOrkSxPs9fZb0uxyF03hIFhYLETkhR/M6GrdScl7BrXY7AkanvlQimxvhDzvs2hoFf55t+Qs/FEE7I/cyIMxz46YrAgqsBjJXygWwQLXj Generated by Nova\n', 'deleted': 0L, 'config_drive': u'', 'power_state': 0L, 'default_ephemeral_device': None, 'progress': None, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'launched_at': None, 'scheduled_at': None, 'node': None, 'ramdisk_id': u'', 'access_ip_v6': None, 'access_ip_v4': None, 'kernel_id': u'', 'key_name': u'keypair-8197170298', 'updated_at': None, 'host': None, 'root_gb': 1L, 'display_name': u'vm-93808201693-0-clone', 'system_metadata': {u'instance_type_name': u'm1.tiny', u'images': u'5b5fecdb-1913-486b-b74c-8457388618ef,07141dca-6ef8-4da0-b7c3-37b331a89382', u'image_min_ram': u'0', u'instance_type_ephemeral_gb': u'0', u'clean_attempts': u'1', u'network_allocated': u'True', u'image_base_image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', u'gc_src_host': u'10.35.0.14', u'logical_volumes': u'', u'instance_type_vcpu_weight': None, u'instance_type_root_gb': u'1', u'instance_type_id': u'2', u'instance_type_rxtx_factor': u'1.0', u'gc_dst_host': u'10.35.0.15', u'instance_type_vcpus': u'1', u'image_disk_format': u'qcow2', u'instance_type_memory_mb': u'512', u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e', u'attached_networks': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'instance_type_flavorid': u'1', u'image_container_format': u'bare', u'image_min_disk': u'1', u'instance_type_swap': u'0'}, 'task_state': u'scheduling', 'shutdown_terminate': False, 'cell_name': None, 'ephemeral_key_uuid': None, 'locked': False, 'name': 'instance-00000005', 'created_at': datetime.datetime(2014, 7, 11, 22, 15, 7), 'locked_by': None, 'launch_index': 0L, 'memory_mb': 512L, 'vcpus': 1L, 'image_ref': u'8fb48209-3c53-4767-a4a4-a47d5f85db49', 'architecture': None, 'auto_disk_config': False, 'os_type': None, 'metadata': {u'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e'}}, 'security_group': ['default'], 'instance_uuids': [u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c']}, 'filter_properties': {'scheduler_hints': {}}}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange='nova', auto_delete=False, passive=False, arguments=None, type='topic', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 messaging messaging.py _publish NBK: _publish routing_key=u'cobalt.10.35.0.14' exchange='nova' message.body='{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'cobalt.10.35.0.14', immediate=False, exchange='nova') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 connection connection.py _debug [Kombu connection:0x7f51a033c150] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'2', 'redelivered': False, 'routing_key': u'cobalt.10.35.0.14', 'delivery_tag': 1, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>}))
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'2', 'redelivered': False, 'routing_key': u'cobalt.10.35.0.14', 'delivery_tag': 1, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>}
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 common common.py _safe_log received {u'_context_roles': [u'_member_', u'Admin', u'ProjectAdmin', u'admin'], u'_context_request_id': u'req-b85b7831-dba3-4585-b0b0-cdaef260cfa5', u'_context_quota_class': None, u'_context_instance_lock_checked': False, u'_context_project_name': u'admin', u'_context_service_catalog': [{u'endpoints': [{u'adminURL': u'http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4', u'region': u'RegionOne', u'internalURL': u'http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4', u'publicURL': u'http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4'}], u'endpoints_links': [], u'type': u'volume', u'name': u'Volume Service'}], u'args': {u'instance_uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'migration_network_info': None, u'migration_url': None, u'params': {u'key_name': None, u'guest': {}, u'availability_zone': None}}, u'_context_tenant': u'e57d3cef3061401a900a9751b1fb0bc4', u'_context_user': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', u'_context_auth_token': '<SANITIZED>', u'method': u'launch_instance', u'_context_is_admin': True, u'version': u'2.0', u'_context_project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'_context_timestamp': u'2014-07-11T22:15:06.868281', u'_unique_id': u'f18e9d0c5adf452fa9da21bc8000cbb7', u'_context_read_deleted': u'no', u'_context_user_id': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', u'_context_user_name': u'admin', u'_context_remote_address': u'10.35.0.15'}
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 common common.py _safe_log unpacked context: {'tenant': u'e57d3cef3061401a900a9751b1fb0bc4', 'project_name': u'admin', 'user_id': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', 'roles': [u'_member_', u'Admin', u'ProjectAdmin', u'admin'], 'timestamp': u'2014-07-11T22:15:06.868281', 'auth_token': '<SANITIZED>', 'remote_address': u'10.35.0.15', 'quota_class': None, 'is_admin': True, 'user': u'6bb7c6d04a8b43dc83cf9e851b92b3fb', 'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4', u'region': u'RegionOne', u'internalURL': u'http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4', u'publicURL': u'http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4'}], u'type': u'volume', u'name': u'Volume Service'}], 'request_id': u'req-b85b7831-dba3-4585-b0b0-cdaef260cfa5', 'instance_lock_checked': False, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'user_name': u'admin', 'read_deleted': u'no'}
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=1.130 sec retry=1 got_ret=True exceptions=['connection_errors: [Errno 104] Connection reset by peer']
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_context_project_name\\": \\"admin\\", \\"_context_quota_class\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"_context_user_name\\": \\"admin\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"args\\": {\\"instance_uuid\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"migration_network_info\\": null, \\"migration_url\\": null, \\"params\\": {\\"guest\\": {}, \\"availability_zone\\": null, \\"key_name\\": null}}, \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_unique_id\\": \\"f18e9d0c5adf452fa9da21bc8000cbb7\\", \\"_context_is_admin\\": true, \\"version\\": \\"2.0\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_read_deleted\\": \\"no\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"method\\": \\"launch_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue cobalt -> <Exchange nova(topic) bound to chan:1> -> cobalt bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fddb2bf7bd0>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>, 'kwargs': {'exclusive': False, 'name': 'cobalt', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'cobalt', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>}}), TopicConsumer({'queue': <Queue cobalt.10.35.0.14 -> <Exchange nova(topic) bound to chan:1> -> cobalt.10.35.0.14 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fddb2bf7bd0>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>, 'kwargs': {'exclusive': False, 'name': 'cobalt.10.35.0.14', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'cobalt.10.35.0.14', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>}}), FanoutConsumer({'queue': <Queue cobalt_fanout_3476e93f95d740879d60eeb9f11d64a0 -> <Exchange cobalt_fanout(fanout) bound to chan:1> -> cobalt bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fddb2bf7bd0>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>, 'kwargs': {'exclusive': False, 'name': 'cobalt_fanout_3476e93f95d740879d60eeb9f11d64a0', 'exchange': <unbound Exchange cobalt_fanout(fanout)>, 'durable': True, 'routing_key': 'cobalt', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb345bb90>}})]
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590155599536 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'launch_instance' message={u'args': {u'instance_uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'migration_network_info': None, u'migration_url': None, u'params': {u'key_name': None, u'guest': {}, u'availability_zone': None}}, u'version': u'2.0', u'method': u'launch_instance'}
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 quota quota.py commit Committed reservations ['775379dc-0371-4d58-ac83-3c514fcc9189', 'f564fea1-1934-428a-8d79-ecce60d4e7b3', '72d91a0c-ff19-46df-abe4-8ca0443d6f2e'] commit /usr/lib64/python2.7/site-packages/nova/quota.py:1297
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 instance instance.py obj_load_attr Lazy-loading `metadata' on Instance uuid 7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c obj_load_attr /usr/lib64/python2.7/site-packages/nova/objects/instance.py:519
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 manager manager.py wrapped_fn launch_instance called: {u'instance_uuid': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'migration_network_info': None, u'migration_url': None, u'params': {u'key_name': None, u'guest': {}, u'availability_zone': None}, 'instance': <nova.objects.instance.Instance object at 0x7fddb360ce50>}
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 manager manager.py wrapped_fn Locking instance 7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c (fn:launch_instance)
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 manager manager.py _lock_instance Acquiring lock for instance 7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 manager manager.py _lock_instance Acquired lock for instance 7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c (me: 140590168431664, refcount=1)
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 instance instance.py obj_load_attr Lazy-loading `system_metadata' on Instance uuid 7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c obj_load_attr /usr/lib64/python2.7/site-packages/nova/objects/instance.py:519
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 hooks hooks.py _call_hooks Calling hooks for pre_launch action
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 hooks hooks.py _call_hooks Done calling hooks for pre_launch action
Jul 11 22:15:09 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:06 +0000] "POST /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/adb82c12-7ea3-40e1-b485-ccf1e91d3b8e/action HTTP/1.1" 200 1027 "" "python-novaclient"
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 log log.py write 172.31.0.13,10.35.0.15 "POST /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/adb82c12-7ea3-40e1-b485-ccf1e91d3b8e/action HTTP/1.1" status: 200 len: 1209 time: 2.4563382
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7fca7469f650>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7fca7469f650>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7fca74debc10>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 iso8601 iso8601.py parse_date Parsed 2014-07-11T22:15:07.000000 into {'tz_sign': None, 'second_fraction': '000000', 'hour': '22', 'daydash': '11', 'tz_hour': None, 'month': None, 'timezone': None, 'second': '07', 'tz_minute': None, 'year': '2014', 'separator': 'T', 'monthdash': '07', 'day': None, 'minute': '15'} with default timezone <iso8601.iso8601.Utc object at 0x7fddaf297b90>
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 iso8601 iso8601.py parse_date Parsed 2014-07-11T22:15:07.000000 into {'tz_sign': None, 'second_fraction': '000000', 'hour': '22', 'daydash': '11', 'tz_hour': None, 'month': None, 'timezone': None, 'second': '07', 'tz_minute': None, 'year': '2014', 'separator': 'T', 'monthdash': '07', 'day': None, 'minute': '15'} with default timezone <iso8601.iso8601.Utc object at 0x7fddaf297b90>
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 iso8601 iso8601.py parse_date Parsed 2014-07-11T22:15:07.000000 into {'tz_sign': None, 'second_fraction': '000000', 'hour': '22', 'daydash': '11', 'tz_hour': None, 'month': None, 'timezone': None, 'second': '07', 'tz_minute': None, 'year': '2014', 'separator': 'T', 'monthdash': '07', 'day': None, 'minute': '15'} with default timezone <iso8601.iso8601.Utc object at 0x7fddaf297b90>
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 iso8601 iso8601.py parse_date Parsed 2014-07-11T22:15:07.000000 into {'tz_sign': None, 'second_fraction': '000000', 'hour': '22', 'daydash': '11', 'tz_hour': None, 'month': None, 'timezone': None, 'second': '07', 'tz_minute': None, 'year': '2014', 'separator': 'T', 'monthdash': '07', 'day': None, 'minute': '15'} with default timezone <iso8601.iso8601.Utc object at 0x7fddaf297b90>
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 iso8601 iso8601.py parse_date Parsed 2014-07-11T22:15:07.000000 into {'tz_sign': None, 'second_fraction': '000000', 'hour': '22', 'daydash': '11', 'tz_hour': None, 'month': None, 'timezone': None, 'second': '07', 'tz_minute': None, 'year': '2014', 'separator': 'T', 'monthdash': '07', 'day': None, 'minute': '15'} with default timezone <iso8601.iso8601.Utc object at 0x7fddaf297b90>
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 instance instance.py obj_load_attr Lazy-loading `system_metadata' on Instance uuid adb82c12-7ea3-40e1-b485-ccf1e91d3b8e
Jul 11 22:15:09 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:09 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1632 "" "python-novaclient"
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1806 time: 0.1210771
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 middleware middleware.py __call__ Matched POST /e57d3cef3061401a900a9751b1fb0bc4/servers/adb82c12-7ea3-40e1-b485-ccf1e91d3b8e/action __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)/action', defaults: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 middleware middleware.py __call__ Match dict: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 wsgi wsgi.py _process_stack Action: 'action', body: {"gc_list_launched": null} _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:940
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 wsgi wsgi.py _process_stack Calling method '<bound method CobaltServerControllerExtension._dep_list_launched_instances of <cobalt.nova.osapi.cobalt_extension.CobaltServerControllerExtension object at 0x7f519ef94210>>' (Content-type='application/json', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 api api.py get_all Searching by: {'deleted': False, 'metadata': {'launched_from': u'adb82c12-7ea3-40e1-b485-ccf1e91d3b8e'}} get_all /usr/lib64/python2.7/site-packages/nova/compute/api.py:1839
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 manager manager.py _instance_network_info Making call to network for launching instance=instance-00000005
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 client client.py call NBK: rpc client call method='allocate_for_instance' target=<Target topic=network, version=1.9, server=10.35.0.14> timeout=60 rpc_response_timeout=60 kwargs={'vpn': False, 'requested_networks': [[u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', None]], 'instance_id': '7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', 'host': u'10.35.0.14', 'rxtx_factor': 1.0, 'macs': None, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4'}
!!! client sending 8fd1d3cb00f54d70bb6cdbc62700a0de, connection reset, retry
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 amqpdriver amqpdriver.py _send MSG_ID is 8fd1d3cb00f54d70bb6cdbc62700a0de
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 amqp amqp.py _add_unique_id UNIQUE_ID is f61079f76c13469999f336d901118c05.
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange='nova', auto_delete=False, passive=False, arguments=None, type='topic', durable=True)
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _error_callback Failed to publish message to topic 'network.10.35.0.14': [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 698, in ensure
ret = method(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 831, in _publish
publisher = cls(self.conf, self.channel, topic, **kwargs)
File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 443, in __init__
**options)
File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 390, in __init__
self.reconnect(channel)
File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 398, in reconnect
routing_key=self.routing_key)
File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 85, in __init__
self.revive(self._channel)
File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 224, in revive
self.declare()
File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 105, in declare
self.exchange.declare()
File "/usr/lib64/python2.7/site-packages/kombu/entity.py", line 172, in declare
ret = self.channel.exchange_declare(**args)
File "/usr/lib64/python2.7/site-packages/kombu/utils/debug.py", line 57, in __wrapped
ret = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/amqp/channel.py", line 613, in exchange_declare
self._send_method((40, 10), args)
File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/lib64/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/lib64/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 307, in sendall
tail = self.send(data, flags)
File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 293, in send
total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py ensure NBK: ensure reconnecting...
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _connect Reconnecting to AMQP server on 10.35.0.3:5672
Jul 11 22:15:09 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _connect Delaying reconnect for 1.0 seconds...
Jul 11 22:15:09 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:09 +0000] "POST /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/adb82c12-7ea3-40e1-b485-ccf1e91d3b8e/action HTTP/1.1" 200 1083 "" "python-novaclient"
Jul 11 22:15:09 10.35.0.13 nova-api 3620 139988495936240 log log.py write 172.31.0.13,10.35.0.15 "POST /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/adb82c12-7ea3-40e1-b485-ccf1e91d3b8e/action HTTP/1.1" status: 200 len: 1265 time: 0.2890389
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7fca7469f650>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7fca7469f650>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7fca74debc10>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:15:09 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:09 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1622 "" "python-novaclient"
Jul 11 22:15:09 10.35.0.14 nova-api 9768 140507510200912 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1796 time: 0.1009960
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f3f9eddf7d0>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f3f9eddf7d0>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7f3f9e939c50>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:15:09 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:09 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1622 "" "python-novaclient"
Jul 11 22:15:09 10.35.0.15 nova-api 4610 139911229646960 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1796 time: 0.1129191
!!! client reconnecting to rabbit to send
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb5d0] closed
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb5d0] establishing connection...
Jul 11 22:15:10 10.35.0.15 rabbitmq.log: =INFO REPORT==== 11-Jul-2014::22:15:10 ===
Jul 11 22:15:10 10.35.0.15 rabbitmq.log: accepting AMQP connection <0.1483.0> (10.35.0.14:63286 -> 10.35.0.3:5672)
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _start Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit@node-10-35-0-15', u'platform': u'Erlang/OTP', u'version': u'3.3.4'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _open_ok Open OK!
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb5d0] connection established: <kombu.transport.pyamqp.Connection object at 0x7fddb32ca710>
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb5d0] create channel
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 channel channel.py __init__ using channel_id: 1
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 channel channel.py _open_ok Channel open
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _connect Connected to AMQP server on 10.35.0.3:5672
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange='nova', auto_delete=False, passive=False, arguments=None, type='topic', durable=True)
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {'ttl': 60000}, {'delivery_mode': 2})
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', headers={'ttl': 60000}, properties={'delivery_mode': 2}
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 messaging messaging.py _publish NBK: _publish routing_key=u'network.10.35.0.14' exchange='nova' message.body='{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'network.10.35.0.14', immediate=False, exchange='nova')
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.030 sec retry=1 got_ret=True exceptions=['connection_errors: [Errno 104] Connection reset by peer']
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] close()
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 channel channel.py _do_close Closed channel #1
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb5d0] create channel
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 channel channel.py __init__ using channel_id: 1
!!! server receives message
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'2', 'redelivered': False, 'routing_key': u'network.10.35.0.14', 'delivery_tag': 3, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 channel channel.py _open_ok Channel open
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'2', 'redelivered': False, 'routing_key': u'network.10.35.0.14', 'delivery_tag': 3, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>}
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='8fd1d3cb00f54d70bb6cdbc62700a0de' timeout=60 self.incoming=[]
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
!!! client is consuming, connection reset, waits to reconnect
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_7889b54f29e94b08935e5fcda635faec -> <Exchange reply_7889b54f29e94b08935e5fcda635faec(direct) bound to chan:1> -> reply_7889b54f29e94b08935e5fcda635faec bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7fddb2c9fb50>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb2c9ff50>, 'kwargs': {'exclusive': False, 'name': 'reply_7889b54f29e94b08935e5fcda635faec', 'exchange': <unbound Exchange reply_7889b54f29e94b08935e5fcda635faec(direct)>, 'durable': True, 'routing_key': 'reply_7889b54f29e94b08935e5fcda635faec', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb2c9ff50>}})]
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _error_callback Failed to consume message from queue: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 698, in ensure
ret = method(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 810, in _consume
ret = self.connection.drain_events(timeout=timeout)
File "/usr/lib64/python2.7/site-packages/kombu/connection.py", line 279, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/usr/lib64/python2.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
return connection.drain_events(**kwargs)
File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
return amqp_method(channel, args)
File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 523, in _close
self._x_close_ok()
File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
self._send_method((10, 51))
File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/lib64/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/lib64/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 307, in sendall
tail = self.send(data, flags)
File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 293, in send
total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py ensure NBK: ensure reconnecting...
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _connect Reconnecting to AMQP server on 10.35.0.3:5672
Jul 11 22:15:10 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _connect Delaying reconnect for 1.0 seconds...
!!! server continues, acknowledges
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.020 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_context_roles\\": [\\"_member_\\", \\"Admin\\", \\"ProjectAdmin\\", \\"admin\\"], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-b85b7831-dba3-4585-b0b0-cdaef260cfa5\\", \\"_context_service_catalog\\": [{\\"endpoints\\": [{\\"adminURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"region\\": \\"RegionOne\\", \\"internalURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\", \\"publicURL\\": \\"http://10.35.2.6:8776/v1/e57d3cef3061401a900a9751b1fb0bc4\\"}], \\"endpoints_links\\": [], \\"type\\": \\"volume\\", \\"name\\": \\"Volume Service\\"}], \\"args\\": {\\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"macs\\": null, \\"instance_id\\": \\"7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c\\", \\"host\\": \\"10.35.0.14\\", \\"rxtx_factor\\": 1.0, \\"requested_networks\\": [[\\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", null]], \\"vpn\\": false}, \\"_unique_id\\": \\"f61079f76c13469999f336d901118c05\\", \\"_context_user\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_user_id\\": \\"6bb7c6d04a8b43dc83cf9e851b92b3fb\\", \\"_context_project_name\\": \\"admin\\", \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7889b54f29e94b08935e5fcda635faec\\", \\"_context_auth_token\\": \\"925d73112a9f441e9f5599509efc94ee\\", \\"_context_tenant\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:06.868281\\", \\"_context_user_name\\": \\"admin\\", \\"method\\": \\"allocate_for_instance\\", \\"_context_remote_address\\": \\"10.35.0.15\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue network -> <Exchange nova(topic) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fdfd65c2c90>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>, 'kwargs': {'exclusive': False, 'name': 'network', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>}}), TopicConsumer({'queue': <Queue network.10.35.0.14 -> <Exchange nova(topic) bound to chan:1> -> network.10.35.0.14 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fdfd65c2c90>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>, 'kwargs': {'exclusive': False, 'name': 'network.10.35.0.14', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network.10.35.0.14', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>}}), FanoutConsumer({'queue': <Queue network_fanout_083e9a4049044be0a851eac8ef12d2bc -> <Exchange network_fanout(fanout) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fdfd65c2c90>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>, 'kwargs': {'exclusive': False, 'name': 'network_fanout_083e9a4049044be0a851eac8ef12d2bc', 'exchange': <unbound Exchange network_fanout(fanout)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fdfd7291e90>}})]
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599308161744 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'allocate_for_instance' message={u'args': {u'vpn': False, u'requested_networks': [[u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', None]], u'instance_id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'host': u'10.35.0.14', u'rxtx_factor': 1.0, u'macs': None, u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4'}, u'version': u'1.9', u'method': u'allocate_for_instance'}
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 manager manager.py allocate_for_instance network allocations allocate_for_instance /usr/lib64/python2.7/site-packages/nova/network/manager.py:494
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 manager manager.py allocate_for_instance networks retrieved for instance: |[{'bridge': u'cloud', 'multi_host': True, 'should_create_bridge': True, 'tenant_id': None, 'label': u'private', 'id': 'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce'}]| allocate_for_instance /usr/lib64/python2.7/site-packages/nova/network/manager.py:501
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 quota quota.py reserve Created reservations ['23a15b1d-8d31-49bd-bbb0-40602d24a4c8'] reserve /usr/lib64/python2.7/site-packages/nova/quota.py:1272
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py apply Skipping apply due to lack of new rules apply /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:457
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py apply Skipping apply due to lack of new rules apply /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:457
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "lock_bridge" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Attempting to grab file lock "lock_bridge" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got file lock "lock_bridge" at /var/lock/nova/nova-lock_bridge lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "ensure_bridge" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Released file lock "lock_bridge" at /var/lock/nova/nova-lock_bridge lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "ensure_bridge" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py apply Skipping apply due to lack of new rules apply /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:457
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "lock_gateway" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Attempting to grab file lock "lock_gateway" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got file lock "lock_gateway" at /var/lock/nova/nova-lock_gateway lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "initialize_gateway_device" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Running cmd (subprocess): sysctl -n net.ipv4.ip_forward execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Running cmd (subprocess): ip addr show dev cloud scope global execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:10 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip route replace 10.35.3.0/24 dev cloud table cloud execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:10 10.35.0.14 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip route replace 10.35.3.0/24 dev cloud table cloud
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip route replace 0.0.0.0/0 via 10.35.3.1 dev cloud table cloud execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:11 10.35.0.14 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip route replace 0.0.0.0/0 via 10.35.3.1 dev cloud table cloud
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Released file lock "lock_gateway" at /var/lock/nova/nova-lock_gateway lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "initialize_gateway_device" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "dnsmasq_start" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "restart_dhcp" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Running cmd (subprocess): cat /proc/19343/cmdline execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kill -HUP 19343 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:11 10.35.0.14 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf kill -HUP 19343
Jul 11 22:15:11 10.35.0.14 dnsmasq-dhcp[19343]: read /var/lib/dnsmasq/nova-cloud.conf
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py add_rule Skipping duplicate iptables rule addition add_rule /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:296
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 linux_net linux_net.py apply Skipping apply due to lack of new rules apply /usr/lib64/python2.7/site-packages/nova/network/linux_net.py:457
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "restart_dhcp" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 quota quota.py commit Committed reservations [u'23a15b1d-8d31-49bd-bbb0-40602d24a4c8'] commit /usr/lib64/python2.7/site-packages/nova/quota.py:1297
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_7889b54f29e94b08935e5fcda635faec', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
!!! server sends result of msgid 8fd1d3cb00f54d70bb6cdbc62700a0de
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"fd824fcdeb454289a38edfeed2c502f6\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.8\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:c4:94:c3\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"b3277985-f0d9-425a-acbf-755610a3dbb8\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"fd824fcdeb454289a38edfeed2c502f6\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.8\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:c4:94:c3\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"b3277985-f0d9-425a-acbf-755610a3dbb8\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\"}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 messaging messaging.py _publish NBK: _publish routing_key=u'reply_7889b54f29e94b08935e5fcda635faec' exchange=u'reply_7889b54f29e94b08935e5fcda635faec' message.body='{"oslo.message": "{\\"_unique_id\\": \\"fd824fcdeb454289a38edfeed2c502f6\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.8\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:c4:94:c3\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"b3277985-f0d9-425a-acbf-755610a3dbb8\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"fd824fcdeb454289a38edfeed2c502f6\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.8\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.7\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:c4:94:c3\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"b3277985-f0d9-425a-acbf-755610a3dbb8\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_7889b54f29e94b08935e5fcda635faec', immediate=False, exchange=u'reply_7889b54f29e94b08935e5fcda635faec') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_7889b54f29e94b08935e5fcda635faec', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"9144561bc404478d97f7f46c1f0ecf0a\\", \\"failure\\": null, \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"9144561bc404478d97f7f46c1f0ecf0a\\", \\"failure\\": null, \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 messaging messaging.py _publish NBK: _publish routing_key=u'reply_7889b54f29e94b08935e5fcda635faec' exchange=u'reply_7889b54f29e94b08935e5fcda635faec' message.body='{"oslo.message": "{\\"_unique_id\\": \\"9144561bc404478d97f7f46c1f0ecf0a\\", \\"failure\\": null, \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"9144561bc404478d97f7f46c1f0ecf0a\\", \\"failure\\": null, \\"_msg_id\\": \\"8fd1d3cb00f54d70bb6cdbc62700a0de\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_7889b54f29e94b08935e5fcda635faec', immediate=False, exchange=u'reply_7889b54f29e94b08935e5fcda635faec') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 connection connection.py _debug [Kombu connection:0x7fdfd65c7490] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:11 10.35.0.14 nova-network 9862 140599325779120 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'allocate_for_instance' took=0.500 got_reply=True exceptions=[] message={u'args': {u'vpn': False, u'requested_networks': [[u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', None]], u'instance_id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', u'host': u'10.35.0.14', u'rxtx_factor': 1.0, u'macs': None, u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4'}, u'version': u'1.9', u'method': u'allocate_for_instance'}
!!! client reconnect to consume
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2c9fb10] closed
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb590] establishing connection...
Jul 11 22:15:11 10.35.0.15 rabbitmq.log: =INFO REPORT==== 11-Jul-2014::22:15:11 ===
Jul 11 22:15:11 10.35.0.15 rabbitmq.log: accepting AMQP connection <0.1509.0> (10.35.0.14:63287 -> 10.35.0.3:5672)
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _start Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit@node-10-35-0-15', u'platform': u'Erlang/OTP', u'version': u'3.3.4'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _open_ok Open OK!
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb590] connection established: <kombu.transport.pyamqp.Connection object at 0x7fddb31f5850>
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 connection connection.py _debug [Kombu connection:0x7fddb2feb590] create channel
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 channel channel.py __init__ using channel_id: 1
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 channel channel.py _open_ok Channel open
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange='reply_7889b54f29e94b08935e5fcda635faec', auto_delete=False, passive=False, arguments=None, type='direct', durable=True)
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] queue_declare(exclusive=False, nowait=False, auto_delete=False, queue='reply_7889b54f29e94b08935e5fcda635faec', arguments={}, passive=False, durable=True)
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] queue_bind(queue='reply_7889b54f29e94b08935e5fcda635faec', arguments=None, nowait=False, routing_key='reply_7889b54f29e94b08935e5fcda635faec', exchange='reply_7889b54f29e94b08935e5fcda635faec')
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _connect Connected to AMQP server on 10.35.0.3:5672
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=True self.consumers=[DirectConsumer({'queue': <Queue reply_7889b54f29e94b08935e5fcda635faec -> <Exchange reply_7889b54f29e94b08935e5fcda635faec(direct) bound to chan:1> -> reply_7889b54f29e94b08935e5fcda635faec bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7fddb2c9fb50>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb3839290>, 'kwargs': {'exclusive': False, 'name': 'reply_7889b54f29e94b08935e5fcda635faec', 'exchange': <unbound Exchange reply_7889b54f29e94b08935e5fcda635faec(direct)>, 'durable': True, 'routing_key': 'reply_7889b54f29e94b08935e5fcda635faec', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb3839290>}})]
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _consume NBK: queues_tail.consume queues_tail=DirectConsumer({'queue': <Queue reply_7889b54f29e94b08935e5fcda635faec -> <Exchange reply_7889b54f29e94b08935e5fcda635faec(direct) bound to chan:1> -> reply_7889b54f29e94b08935e5fcda635faec bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7fddb2c9fb50>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb3839290>, 'kwargs': {'exclusive': False, 'name': 'reply_7889b54f29e94b08935e5fcda635faec', 'exchange': <unbound Exchange reply_7889b54f29e94b08935e5fcda635faec(direct)>, 'durable': True, 'routing_key': 'reply_7889b54f29e94b08935e5fcda635faec', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fddb3839290>}})
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 debug debug.py __wrapped [Kombu channel:1] basic_consume(queue='reply_7889b54f29e94b08935e5fcda635faec', consumer_tag='1', nowait=False, no_ack=False, callback=<function _callback at 0x7fddb38648c0>)
Jul 11 22:15:11 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
!!! client should receive reply here
Jul 11 22:15:11 10.35.0.13 nova-consoleauth 5887 140479887808144 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 periodic_task periodic_task.py run_periodic_tasks Running periodic task CobaltManager._refresh_host
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 periodic_task periodic_task.py run_periodic_tasks Running periodic task CobaltManager._clean
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 vmsconn vmsconn.py _clean_lvm_symlinks Starting to clean symlinks in /dev/nova
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/1537482d-c1f5-4d6a-aadc-67e8a635af8f_disk
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/ea07a714-0252-486d-8806-8d03efde026c_disk
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/libvirt_save
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/novadisk
Jul 11 22:15:12 10.35.0.15 cobalt-compute: 4496 140425978537776 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds
Jul 11 22:15:13 10.35.0.14 nova-dhcpbridge 22385 140505010029584 dhcpbridge dhcpbridge.py main Called 'old' for mac 'fa:16:3e:fd:d5:79' with ip '10.35.3.6' main /usr/lib64/python2.7/site-packages/nova/cmd/dhcpbridge.py:109
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Got semaphore "compute_resources" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Got semaphore / lock "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Auditing locally available compute resources
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 driver driver.py update_status Updating host stats update_status /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:5260
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:15 10.35.0.13 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 driver driver.py get_instance_disk_info skipping disk for instance-00000001 as it does not have a path get_instance_disk_info /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:4765
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Got semaphore "compute_resources" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Got semaphore / lock "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Auditing locally available compute resources
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 driver driver.py update_status Updating host stats update_status /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:5260
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:15 10.35.0.15 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free ram (MB): 45445 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:409
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free disk (GB): 1619 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:410
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free VCPUs: 3 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:415
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: assignable PCI devices: [] _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:422
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Free ram (MB): 47217
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Free disk (GB): 1861
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Free VCPUS: 3
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 driver driver.py get_instance_disk_info skipping disk for instance-00000002 as it does not have a path get_instance_disk_info /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:4765
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 driver driver.py get_instance_disk_info skipping disk for instance-00000003 as it does not have a path get_instance_disk_info /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:4765
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _sync_compute_node Compute_service record updated for 10.35.0.13:node-10-35-0-13
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Semaphore / lock released "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 manager manager.py _reclaim_queued_deletes CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib64/python2.7/site-packages/nova/compute/manager.py:5388
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._heal_instance_info_cache run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 manager manager.py _heal_instance_info_cache Starting heal instance info cache _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4813
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 manager manager.py _heal_instance_info_cache Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4817
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 instance instance.py obj_load_attr Lazy-loading `system_metadata' on Instance uuid a3712753-7081-4700-9041-752640884141 obj_load_attr /usr/lib64/python2.7/site-packages/nova/objects/instance.py:519
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free ram (MB): 44694 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:409
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free disk (GB): 1618 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:410
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free VCPUs: 2 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:415
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: assignable PCI devices: [] _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:422
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 client client.py call NBK: rpc client call method='get_instance_nw_info' target=<Target topic=network, version=1.9> timeout=60 rpc_response_timeout=60 kwargs={'instance_id': 'a3712753-7081-4700-9041-752640884141', 'host': u'10.35.0.13', 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'rxtx_factor': 1.0}
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange='nova', auto_delete=False, passive=False, arguments=None, type='topic', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {'ttl': 60000}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', headers={'ttl': 60000}, properties={'delivery_mode': 2}
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 messaging messaging.py _publish NBK: _publish routing_key='network' exchange='nova' message.body='{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='network', immediate=False, exchange='nova') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 connection connection.py _debug [Kombu connection:0x7f0b8b1d4ed0] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'network', 'delivery_tag': 3, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'network', 'delivery_tag': 3, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>}
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='4a53f50db4e3417ab26a1f77914d68d9' timeout=60 self.incoming=[]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_7c54947630034c8d91b504e7f2a869f8 -> <Exchange reply_7c54947630034c8d91b504e7f2a869f8(direct) bound to chan:1> -> reply_7c54947630034c8d91b504e7f2a869f8 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7f0b8a84d7d0>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>, 'kwargs': {'exclusive': False, 'name': 'reply_7c54947630034c8d91b504e7f2a869f8', 'exchange': <unbound Exchange reply_7c54947630034c8d91b504e7f2a869f8(direct)>, 'durable': True, 'routing_key': 'reply_7c54947630034c8d91b504e7f2a869f8', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>}})]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.160 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-52ced663-f60e-4c83-b6a9-8c95a44965e0\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"a3712753-7081-4700-9041-752640884141\\", \\"host\\": \\"10.35.0.13\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"99b7f5df4dfa473e8d3163e1f99bf149\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_7c54947630034c8d91b504e7f2a869f8\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.295954\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue network -> <Exchange nova(topic) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fd70fafb290>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>, 'kwargs': {'exclusive': False, 'name': 'network', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>}}), TopicConsumer({'queue': <Queue network.10.35.0.15 -> <Exchange nova(topic) bound to chan:1> -> network.10.35.0.15 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fd70fafb290>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>, 'kwargs': {'exclusive': False, 'name': 'network.10.35.0.15', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network.10.35.0.15', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>}}), FanoutConsumer({'queue': <Queue network_fanout_0f7fa9d64d854eafaae3312b56b40764 -> <Exchange network_fanout(fanout) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7fd70fafb290>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>, 'kwargs': {'exclusive': False, 'name': 'network_fanout_0f7fa9d64d854eafaae3312b56b40764', 'exchange': <unbound Exchange network_fanout(fanout)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7fd710488190>}})]
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561649929264 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'get_instance_nw_info' message={u'args': {u'instance_id': u'a3712753-7081-4700-9041-752640884141', u'host': u'10.35.0.13', u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'rxtx_factor': 1.0}, u'version': u'1.9', u'method': u'get_instance_nw_info'}
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Free ram (MB): 46705
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Free disk (GB): 1860
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Free VCPUS: 2
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _sync_compute_node Compute_service record updated for 10.35.0.15:node-10-35-0-15
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Semaphore / lock released "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 manager manager.py _reclaim_queued_deletes CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib64/python2.7/site-packages/nova/compute/manager.py:5388
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._heal_instance_info_cache run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 manager manager.py _heal_instance_info_cache Starting heal instance info cache _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4813
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_7c54947630034c8d91b504e7f2a869f8', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 messaging messaging.py _publish NBK: _publish routing_key=u'reply_7c54947630034c8d91b504e7f2a869f8' exchange=u'reply_7c54947630034c8d91b504e7f2a869f8' message.body='{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_7c54947630034c8d91b504e7f2a869f8', immediate=False, exchange=u'reply_7c54947630034c8d91b504e7f2a869f8') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_7c54947630034c8d91b504e7f2a869f8', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_7c54947630034c8d91b504e7f2a869f8', 'delivery_tag': 3, 'exchange': u'reply_7c54947630034c8d91b504e7f2a869f8'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_7c54947630034c8d91b504e7f2a869f8', 'delivery_tag': 3, 'exchange': u'reply_7c54947630034c8d91b504e7f2a869f8'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>}
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 messaging messaging.py _publish NBK: _publish routing_key=u'reply_7c54947630034c8d91b504e7f2a869f8' exchange=u'reply_7c54947630034c8d91b504e7f2a869f8' message.body='{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_7c54947630034c8d91b504e7f2a869f8', immediate=False, exchange=u'reply_7c54947630034c8d91b504e7f2a869f8') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"fd1a05f76cb3462cbfbdca4df291e473\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [], \\"address\\": \\"10.35.3.2\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.3\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:41:32:40\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"45bf2e8e-da71-46f9-9079-e2c173040c67\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='4a53f50db4e3417ab26a1f77914d68d9' timeout=60 self.incoming=[{u'_unique_id': u'fd1a05f76cb3462cbfbdca4df291e473', u'failure': None, u'result': [{u'ovs_interfaceid': None, u'network': {u'bridge': u'cloud', u'label': u'private', u'meta': {u'tenant_id': None, u'multi_host': True, u'should_create_bridge': True}, u'id': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'subnets': [{u'ips': [{u'meta': {}, u'type': u'fixed', u'version': 4, u'address': u'10.35.3.2', u'floating_ips': []}], u'version': 4, u'meta': {u'dhcp_server': u'10.35.3.3'}, u'dns': [{u'meta': {}, u'type': u'dns', u'version': 4, u'address': u'8.8.4.4'}], u'routes': [], u'cidr': u'10.35.3.0/24', u'gateway': {u'meta': {}, u'type': u'gateway', u'version': 4, u'address': u'10.35.3.1'}}, {u'ips': [], u'version': None, u'meta': {u'dhcp_server': u'10.35.3.3'}, u'dns': [], u'routes': [], u'cidr': None, u'gateway': {u'meta': {}, u'type': u'gateway', u'version': None, u'address': None}}]}, u'devname': None, u'qbh_params': None, u'meta': {}, u'details': {}, u'address': u'fa:16:3e:41:32:40', u'active': False, u'type': u'bridge', u'id': u'45bf2e8e-da71-46f9-9079-e2c173040c67', u'qbg_params': None}], u'_msg_id': u'4a53f50db4e3417ab26a1f77914d68d9'}]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='4a53f50db4e3417ab26a1f77914d68d9' got incoming_msg_id=u'4a53f50db4e3417ab26a1f77914d68d9' message_data={u'_unique_id': u'fd1a05f76cb3462cbfbdca4df291e473', u'failure': None, u'result': [{u'ovs_interfaceid': None, u'network': {u'bridge': u'cloud', u'label': u'private', u'meta': {u'tenant_id': None, u'multi_host': True, u'should_create_bridge': True}, u'id': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'subnets': [{u'ips': [{u'meta': {}, u'type': u'fixed', u'version': 4, u'address': u'10.35.3.2', u'floating_ips': []}], u'version': 4, u'meta': {u'dhcp_server': u'10.35.3.3'}, u'dns': [{u'meta': {}, u'type': u'dns', u'version': 4, u'address': u'8.8.4.4'}], u'routes': [], u'cidr': u'10.35.3.0/24', u'gateway': {u'meta': {}, u'type': u'gateway', u'version': 4, u'address': u'10.35.3.1'}}, {u'ips': [], u'version': None, u'meta': {u'dhcp_server': u'10.35.3.3'}, u'dns': [], u'routes': [], u'cidr': None, u'gateway': {u'meta': {}, u'type': u'gateway', u'version': None, u'address': None}}]}, u'devname': None, u'qbh_params': None, u'meta': {}, u'details': {}, u'address': u'fa:16:3e:41:32:40', u'active': False, u'type': u'bridge', u'id': u'45bf2e8e-da71-46f9-9079-e2c173040c67', u'qbg_params': None}]}
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='4a53f50db4e3417ab26a1f77914d68d9' timeout=60 self.incoming=[]
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 connection connection.py _debug [Kombu connection:0x7fd70f392e10] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_7c54947630034c8d91b504e7f2a869f8 -> <Exchange reply_7c54947630034c8d91b504e7f2a869f8(direct) bound to chan:1> -> reply_7c54947630034c8d91b504e7f2a869f8 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7f0b8a84d7d0>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>, 'kwargs': {'exclusive': False, 'name': 'reply_7c54947630034c8d91b504e7f2a869f8', 'exchange': <unbound Exchange reply_7c54947630034c8d91b504e7f2a869f8(direct)>, 'durable': True, 'routing_key': 'reply_7c54947630034c8d91b504e7f2a869f8', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>}})]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_7c54947630034c8d91b504e7f2a869f8', 'delivery_tag': 4, 'exchange': u'reply_7c54947630034c8d91b504e7f2a869f8'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_7c54947630034c8d91b504e7f2a869f8', 'delivery_tag': 4, 'exchange': u'reply_7c54947630034c8d91b504e7f2a869f8'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0b8b5edc10>}
Jul 11 22:15:15 10.35.0.15 nova-network 4639 140561638496848 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'get_instance_nw_info' took=0.130 got_reply=True exceptions=[] message={u'args': {u'instance_id': u'a3712753-7081-4700-9041-752640884141', u'host': u'10.35.0.13', u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'rxtx_factor': 1.0}, u'version': u'1.9', u'method': u'get_instance_nw_info'}
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"35a98f1ec32440d08ee3b12add78fa1e\\", \\"failure\\": null, \\"_msg_id\\": \\"4a53f50db4e3417ab26a1f77914d68d9\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='4a53f50db4e3417ab26a1f77914d68d9' timeout=60 self.incoming=[{u'_unique_id': u'35a98f1ec32440d08ee3b12add78fa1e', u'failure': None, u'_msg_id': u'4a53f50db4e3417ab26a1f77914d68d9', u'result': None, u'ending': True}]
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='4a53f50db4e3417ab26a1f77914d68d9' got incoming_msg_id=u'4a53f50db4e3417ab26a1f77914d68d9' message_data={u'_unique_id': u'35a98f1ec32440d08ee3b12add78fa1e', u'failure': None, u'result': None, u'ending': True}
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 client client.py call NBK: rpc client call method='get_instance_nw_info' target=<Target topic=network, version=1.9> took=0.010 timeout=60 got_result=True exceptions=[] kwargs={'instance_id': 'a3712753-7081-4700-9041-752640884141', 'host': u'10.35.0.13', 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'rxtx_factor': 1.0}
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 api api.py update_instance_cache_with_nw_info Updating cache with info: [VIF({'ovs_interfaceid': None, 'network': Network({'bridge': u'cloud', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': u'fixed', 'floating_ips': [], 'address': u'10.35.3.2'})], 'version': 4, 'meta': {u'dhcp_server': u'10.35.3.3'}, 'dns': [IP({'meta': {}, 'version': 4, 'type': u'dns', 'address': u'8.8.4.4'})], 'routes': [], 'cidr': u'10.35.3.0/24', 'gateway': IP({'meta': {}, 'version': 4, 'type': u'gateway', 'address': u'10.35.3.1'})}), Subnet({'ips': [], 'version': None, 'meta': {u'dhcp_server': u'10.35.3.3'}, 'dns': [], 'routes': [], 'cidr': None, 'gateway': IP({'meta': {}, 'version': None, 'type': u'gateway', 'address': None})})], 'meta': {u'tenant_id': None, u'multi_host': True, u'should_create_bridge': True}, 'id': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', 'label': u'private'}), 'devname': None, 'qbh_params': None, 'meta': {}, 'details': {}, 'address': u'fa:16:3e:41:32:40', 'active': False, 'type': u'bridge', 'id': u'45bf2e8e-da71-46f9-9079-e2c173040c67', 'qbg_params': None})] update_instance_cache_with_nw_info /usr/lib64/python2.7/site-packages/nova/network/api.py:75
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Got semaphore "compute_resources" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Got semaphore / lock "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Auditing locally available compute resources
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 driver driver.py update_status Updating host stats update_status /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:5260
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 client client.py call NBK: rpc client call method='get_instance_nw_info' target=<Target topic=network, version=1.9> timeout=60 rpc_response_timeout=60 kwargs={'instance_id': '1537482d-c1f5-4d6a-aadc-67e8a635af8f', 'host': u'10.35.0.15', 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'rxtx_factor': 1.0}
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange='nova', auto_delete=False, passive=False, arguments=None, type='topic', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {'ttl': 60000}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', headers={'ttl': 60000}, properties={'delivery_mode': 2}
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 messaging messaging.py _publish NBK: _publish routing_key='network' exchange='nova' message.body='{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='network', immediate=False, exchange='nova') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 manager manager.py _heal_instance_info_cache Updated the network info_cache for instance _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4874
Jul 11 22:15:15 10.35.0.13 nova-compute 3510 139687513837808 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 23.01 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 connection connection.py _debug [Kombu connection:0x7ff69f879450] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'network', 'delivery_tag': 3, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'network', 'delivery_tag': 3, 'exchange': u'nova'}, 'properties': {'priority': 0, 'application_headers': {u'ttl': 60000}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>}
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='75d7bd17f8734ab5b0ea935741c2e5cb' timeout=60 self.incoming=[]
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_dea5d1a1b25346b6989765d9b2651d3a -> <Exchange reply_dea5d1a1b25346b6989765d9b2651d3a(direct) bound to chan:1> -> reply_dea5d1a1b25346b6989765d9b2651d3a bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7ff69f78d610>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>, 'kwargs': {'exclusive': False, 'name': 'reply_dea5d1a1b25346b6989765d9b2651d3a', 'exchange': <unbound Exchange reply_dea5d1a1b25346b6989765d9b2651d3a(direct)>, 'durable': True, 'routing_key': 'reply_dea5d1a1b25346b6989765d9b2651d3a', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>}})]
Jul 11 22:15:15 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.140 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_context_roles\\": [], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"_context_quota_class\\": null, \\"_context_request_id\\": \\"req-7a1258aa-28c2-4b38-ad33-5942b5d6430d\\", \\"_context_service_catalog\\": [], \\"args\\": {\\"instance_id\\": \\"1537482d-c1f5-4d6a-aadc-67e8a635af8f\\", \\"host\\": \\"10.35.0.15\\", \\"project_id\\": \\"e57d3cef3061401a900a9751b1fb0bc4\\", \\"rxtx_factor\\": 1.0}, \\"_unique_id\\": \\"2680a0c147cc46a6807e3b1dc5131add\\", \\"_context_user\\": null, \\"_context_user_id\\": null, \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_reply_q\\": \\"reply_dea5d1a1b25346b6989765d9b2651d3a\\", \\"_context_auth_token\\": null, \\"_context_tenant\\": null, \\"_context_instance_lock_checked\\": false, \\"_context_is_admin\\": true, \\"version\\": \\"1.9\\", \\"_context_project_id\\": null, \\"_context_timestamp\\": \\"2014-07-11T22:15:15.456143\\", \\"_context_user_name\\": null, \\"method\\": \\"get_instance_nw_info\\", \\"_context_remote_address\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue network -> <Exchange nova(topic) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f8facb31910>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>, 'kwargs': {'exclusive': False, 'name': 'network', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>}}), TopicConsumer({'queue': <Queue network.10.35.0.13 -> <Exchange nova(topic) bound to chan:1> -> network.10.35.0.13 bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f8facb31910>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>, 'kwargs': {'exclusive': False, 'name': 'network.10.35.0.13', 'exchange': <unbound Exchange nova(topic)>, 'durable': True, 'routing_key': 'network.10.35.0.13', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>}}), FanoutConsumer({'queue': <Queue network_fanout_f195b85e22574d37a95bd0ca797d114a -> <Exchange network_fanout(fanout) bound to chan:1> -> network bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f8facb31910>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>, 'kwargs': {'exclusive': False, 'name': 'network_fanout_f195b85e22574d37a95bd0ca797d114a', 'exchange': <unbound Exchange network_fanout(fanout)>, 'durable': True, 'routing_key': 'network', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f8fad665810>}})]
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031860752 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'get_instance_nw_info' message={u'args': {u'instance_id': u'1537482d-c1f5-4d6a-aadc-67e8a635af8f', u'host': u'10.35.0.15', u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'rxtx_factor': 1.0}, u'version': u'1.9', u'method': u'get_instance_nw_info'}
Jul 11 22:15:15 10.35.0.14 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova
Jul 11 22:15:15 10.35.0.14 nova-compute 9462 139687021030288 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 lockutils lockutils.py lock Got semaphore "get_dhcp" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 lockutils lockutils.py inner Got semaphore / lock "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 lockutils lockutils.py inner Semaphore / lock released "_get_dhcp_ip" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_dea5d1a1b25346b6989765d9b2651d3a', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 messaging messaging.py _publish NBK: _publish routing_key=u'reply_dea5d1a1b25346b6989765d9b2651d3a' exchange=u'reply_dea5d1a1b25346b6989765d9b2651d3a' message.body='{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:15 10.35.0.13 nova-network 3642 140255031857712 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_dea5d1a1b25346b6989765d9b2651d3a', immediate=False, exchange=u'reply_dea5d1a1b25346b6989765d9b2651d3a') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 debug debug.py __wrapped [Kombu channel:1] exchange_declare(nowait=False, exchange=u'reply_dea5d1a1b25346b6989765d9b2651d3a', auto_delete=False, passive=False, arguments=None, type='direct', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_dea5d1a1b25346b6989765d9b2651d3a', 'delivery_tag': 3, 'exchange': u'reply_dea5d1a1b25346b6989765d9b2651d3a'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_dea5d1a1b25346b6989765d9b2651d3a', 'delivery_tag': 3, 'exchange': u'reply_dea5d1a1b25346b6989765d9b2651d3a'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>}
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 debug debug.py __wrapped [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 pyamqp pyamqp.py prepare_message NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 messaging messaging.py _publish NBK: _publish routing_key=u'reply_dea5d1a1b25346b6989765d9b2651d3a' exchange=u'reply_dea5d1a1b25346b6989765d9b2651d3a' message.body='{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 debug debug.py __wrapped [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key=u'reply_dea5d1a1b25346b6989765d9b2651d3a', immediate=False, exchange=u'reply_dea5d1a1b25346b6989765d9b2651d3a') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"36c716e614e84e2191b5035440c28c82\\", \\"failure\\": null, \\"result\\": [{\\"ovs_interfaceid\\": null, \\"network\\": {\\"bridge\\": \\"cloud\\", \\"subnets\\": [{\\"ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"fixed\\", \\"floating_ips\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"floating\\", \\"address\\": \\"10.35.4.2\\"}], \\"address\\": \\"10.35.3.6\\"}], \\"version\\": 4, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [{\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"dns\\", \\"address\\": \\"8.8.4.4\\"}], \\"routes\\": [], \\"cidr\\": \\"10.35.3.0/24\\", \\"gateway\\": {\\"meta\\": {}, \\"version\\": 4, \\"type\\": \\"gateway\\", \\"address\\": \\"10.35.3.1\\"}}, {\\"ips\\": [], \\"version\\": null, \\"meta\\": {\\"dhcp_server\\": \\"10.35.3.5\\"}, \\"dns\\": [], \\"routes\\": [], \\"cidr\\": null, \\"gateway\\": {\\"meta\\": {}, \\"version\\": null, \\"type\\": \\"gateway\\", \\"address\\": null}}], \\"meta\\": {\\"tenant_id\\": null, \\"multi_host\\": true, \\"should_create_bridge\\": true}, \\"id\\": \\"fd3212a8-c4e6-4f8a-a83c-ba632264d1ce\\", \\"label\\": \\"private\\"}, \\"devname\\": null, \\"qbh_params\\": null, \\"meta\\": {}, \\"details\\": {}, \\"address\\": \\"fa:16:3e:fd:d5:79\\", \\"active\\": false, \\"type\\": \\"bridge\\", \\"id\\": \\"0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b\\", \\"qbg_params\\": null}], \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\"}", "oslo.version": "2.0"}'
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 debug debug.py __wrapped [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='75d7bd17f8734ab5b0ea935741c2e5cb' timeout=60 self.incoming=[{u'_unique_id': u'36c716e614e84e2191b5035440c28c82', u'failure': None, u'result': [{u'ovs_interfaceid': None, u'network': {u'bridge': u'cloud', u'label': u'private', u'meta': {u'tenant_id': None, u'multi_host': True, u'should_create_bridge': True}, u'id': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'subnets': [{u'ips': [{u'meta': {}, u'type': u'fixed', u'version': 4, u'address': u'10.35.3.6', u'floating_ips': [{u'meta': {}, u'type': u'floating', u'version': 4, u'address': u'10.35.4.2'}]}], u'version': 4, u'meta': {u'dhcp_server': u'10.35.3.5'}, u'dns': [{u'meta': {}, u'type': u'dns', u'version': 4, u'address': u'8.8.4.4'}], u'routes': [], u'cidr': u'10.35.3.0/24', u'gateway': {u'meta': {}, u'type': u'gateway', u'version': 4, u'address': u'10.35.3.1'}}, {u'ips': [], u'version': None, u'meta': {u'dhcp_server': u'10.35.3.5'}, u'dns': [], u'routes': [], u'cidr': None, u'gateway': {u'meta': {}, u'type': u'gateway', u'version': None, u'address': None}}]}, u'devname': None, u'qbh_params': None, u'meta': {}, u'details': {}, u'address': u'fa:16:3e:fd:d5:79', u'active': False, u'type': u'bridge', u'id': u'0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b', u'qbg_params': None}], u'_msg_id': u'75d7bd17f8734ab5b0ea935741c2e5cb'}]
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='75d7bd17f8734ab5b0ea935741c2e5cb' got incoming_msg_id=u'75d7bd17f8734ab5b0ea935741c2e5cb' message_data={u'_unique_id': u'36c716e614e84e2191b5035440c28c82', u'failure': None, u'result': [{u'ovs_interfaceid': None, u'network': {u'bridge': u'cloud', u'label': u'private', u'meta': {u'tenant_id': None, u'multi_host': True, u'should_create_bridge': True}, u'id': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', u'subnets': [{u'ips': [{u'meta': {}, u'type': u'fixed', u'version': 4, u'address': u'10.35.3.6', u'floating_ips': [{u'meta': {}, u'type': u'floating', u'version': 4, u'address': u'10.35.4.2'}]}], u'version': 4, u'meta': {u'dhcp_server': u'10.35.3.5'}, u'dns': [{u'meta': {}, u'type': u'dns', u'version': 4, u'address': u'8.8.4.4'}], u'routes': [], u'cidr': u'10.35.3.0/24', u'gateway': {u'meta': {}, u'type': u'gateway', u'version': 4, u'address': u'10.35.3.1'}}, {u'ips': [], u'version': None, u'meta': {u'dhcp_server': u'10.35.3.5'}, u'dns': [], u'routes': [], u'cidr': None, u'gateway': {u'meta': {}, u'type': u'gateway', u'version': None, u'address': None}}]}, u'devname': None, u'qbh_params': None, u'meta': {}, u'details': {}, u'address': u'fa:16:3e:fd:d5:79', u'active': False, u'type': u'bridge', u'id': u'0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b', u'qbg_params': None}]}
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='75d7bd17f8734ab5b0ea935741c2e5cb' timeout=60 self.incoming=[]
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 connection connection.py _debug [Kombu connection:0x7f8facb50390] create channel _debug /usr/lib64/python2.7/site-packages/kombu/connection.py:235
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py _consume NBK: _consume self.do_consume=False self.consumers=[DirectConsumer({'queue': <Queue reply_dea5d1a1b25346b6989765d9b2651d3a -> <Exchange reply_dea5d1a1b25346b6989765d9b2651d3a(direct) bound to chan:1> -> reply_dea5d1a1b25346b6989765d9b2651d3a bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.ReplyWaiter object at 0x7ff69f78d610>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>, 'kwargs': {'exclusive': False, 'name': 'reply_dea5d1a1b25346b6989765d9b2651d3a', 'exchange': <unbound Exchange reply_dea5d1a1b25346b6989765d9b2651d3a(direct)>, 'durable': True, 'routing_key': 'reply_dea5d1a1b25346b6989765d9b2651d3a', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>}})]
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events timeout=60
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_dea5d1a1b25346b6989765d9b2651d3a', 'delivery_tag': 4, 'exchange': u'reply_dea5d1a1b25346b6989765d9b2651d3a'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 pyamqp pyamqp.py message_to_python NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'reply_dea5d1a1b25346b6989765d9b2651d3a', 'delivery_tag': 4, 'exchange': u'reply_dea5d1a1b25346b6989765d9b2651d3a'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7ff69fea09d0>}
Jul 11 22:15:16 10.35.0.13 nova-network 3642 140255031857712 dispatcher dispatcher.py _dispatch_and_reply NBK: rpc server _dispatch_and_reply method=u'get_instance_nw_info' took=0.140 got_reply=True exceptions=[] message={u'args': {u'instance_id': u'1537482d-c1f5-4d6a-aadc-67e8a635af8f', u'host': u'10.35.0.15', u'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', u'rxtx_factor': 1.0}, u'version': u'1.9', u'method': u'get_instance_nw_info'}
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 debug debug.py __wrapped message_to_python.body=u'{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py acknowledge NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_unique_id\\": \\"8b011c46b7dc4a7093b0d3c218931561\\", \\"failure\\": null, \\"_msg_id\\": \\"75d7bd17f8734ab5b0ea935741c2e5cb\\", \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py _consume NBK: _consume connection.drain_events ret=None
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='75d7bd17f8734ab5b0ea935741c2e5cb' timeout=60 self.incoming=[{u'_unique_id': u'8b011c46b7dc4a7093b0d3c218931561', u'failure': None, u'_msg_id': u'75d7bd17f8734ab5b0ea935741c2e5cb', u'result': None, u'ending': True}]
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 amqpdriver amqpdriver.py _poll_connection NBK: _poll_connection msg_id='75d7bd17f8734ab5b0ea935741c2e5cb' got incoming_msg_id=u'75d7bd17f8734ab5b0ea935741c2e5cb' message_data={u'_unique_id': u'8b011c46b7dc4a7093b0d3c218931561', u'failure': None, u'result': None, u'ending': True}
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 client client.py call NBK: rpc client call method='get_instance_nw_info' target=<Target topic=network, version=1.9> took=0.010 timeout=60 got_result=True exceptions=[] kwargs={'instance_id': '1537482d-c1f5-4d6a-aadc-67e8a635af8f', 'host': u'10.35.0.15', 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'rxtx_factor': 1.0}
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 api api.py update_instance_cache_with_nw_info Updating cache with info: [VIF({'ovs_interfaceid': None, 'network': Network({'bridge': u'cloud', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': u'fixed', 'floating_ips': [IP({'meta': {}, 'version': 4, 'type': u'floating', 'address': u'10.35.4.2'})], 'address': u'10.35.3.6'})], 'version': 4, 'meta': {u'dhcp_server': u'10.35.3.5'}, 'dns': [IP({'meta': {}, 'version': 4, 'type': u'dns', 'address': u'8.8.4.4'})], 'routes': [], 'cidr': u'10.35.3.0/24', 'gateway': IP({'meta': {}, 'version': 4, 'type': u'gateway', 'address': u'10.35.3.1'})}), Subnet({'ips': [], 'version': None, 'meta': {u'dhcp_server': u'10.35.3.5'}, 'dns': [], 'routes': [], 'cidr': None, 'gateway': IP({'meta': {}, 'version': None, 'type': u'gateway', 'address': None})})], 'meta': {u'tenant_id': None, u'multi_host': True, u'should_create_bridge': True}, 'id': u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', 'label': u'private'}), 'devname': None, 'qbh_params': None, 'meta': {}, 'details': {}, 'address': u'fa:16:3e:fd:d5:79', 'active': False, 'type': u'bridge', 'id': u'0870f9c0-8e2d-4e73-abc3-e3cef4cd9e7b', 'qbg_params': None})] update_instance_cache_with_nw_info /usr/lib64/python2.7/site-packages/nova/network/api.py:75
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 manager manager.py _heal_instance_info_cache Updated the network info_cache for instance _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4874
Jul 11 22:15:16 10.35.0.15 nova-compute 4490 140697163517616 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 23.11 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free ram (MB): 45578 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:409
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free disk (GB): 1620 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:410
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free VCPUs: 4 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:415
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: assignable PCI devices: [] _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:422
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Free ram (MB): 47217
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Free disk (GB): 1861
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Free VCPUS: 3
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _sync_compute_node Compute_service record updated for 10.35.0.14:node-10-35-0-14
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Semaphore / lock released "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 manager manager.py _reclaim_queued_deletes CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib64/python2.7/site-packages/nova/compute/manager.py:5388
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._heal_instance_info_cache run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 manager manager.py _heal_instance_info_cache Starting heal instance info cache _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4813
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 manager manager.py _heal_instance_info_cache Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4817
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 manager manager.py _heal_instance_info_cache Skipping network cache update for instance because it is Building. _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4827
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 manager manager.py _heal_instance_info_cache Didn't find any instances for network info cache update. _heal_instance_info_cache /usr/lib64/python2.7/site-packages/nova/compute/manager.py:4879
Jul 11 22:15:16 10.35.0.14 nova-compute 9462 139687021030288 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 22.13 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7fca7469f650>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7fca7469f650>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:15:19 10.35.0.14 nova-api 9764 140507508318768 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7fca74debc10>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:15:20 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:20 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1746 "" "python-novaclient"
Jul 11 22:15:20 10.35.0.14 nova-api 9764 140507508318768 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1920 time: 0.1227169
Jul 11 22:15:21 10.35.0.13 cobalt-compute: 3516 139996110631152 periodic_task periodic_task.py run_periodic_tasks Running periodic task CobaltManager._refresh_host
Jul 11 22:15:22 10.35.0.13 cobalt-compute: 3516 139996110631152 periodic_task periodic_task.py run_periodic_tasks Running periodic task CobaltManager._clean
Jul 11 22:15:22 10.35.0.13 cobalt-compute: 3516 139996110631152 vmsconn vmsconn.py _clean_lvm_symlinks Starting to clean symlinks in /dev/nova
Jul 11 22:15:22 10.35.0.13 cobalt-compute: 3516 139996110631152 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/a3712753-7081-4700-9041-752640884141_disk
Jul 11 22:15:22 10.35.0.13 cobalt-compute: 3516 139996110631152 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/libvirt_save
Jul 11 22:15:22 10.35.0.13 cobalt-compute: 3516 139996110631152 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/novadisk
Jul 11 22:15:22 10.35.0.13 cobalt-compute: 3516 139996110631152 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.309 3935 DEBUG kombu.channel [-] [Kombu channel:1] exchange_declare(nowait=False, exchange='cinder-scheduler_fanout', auto_delete=False, passive=False, arguments=None, type='fanout', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.310 3935 DEBUG kombu.channel [-] [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.311 3935 ERROR kombu.transport.pyamqp [-] NBK: prepare_message body='{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.311 3935 ERROR kombu.messaging [-] NBK: _publish routing_key='' exchange='cinder-scheduler_fanout' message.body='{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.311 3935 DEBUG kombu.channel [-] [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='', immediate=False, exchange='cinder-scheduler_fanout') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.312 3935 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.312 3935 DEBUG kombu.channel [-] [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.313 7616 DEBUG kombu.channel [-] [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'3', 'redelivered': False, 'routing_key': u'', 'delivery_tag': 3, 'exchange': u'cinder-scheduler_fanout'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.313 7616 ERROR kombu.transport.pyamqp [-] NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'3', 'redelivered': False, 'routing_key': u'', 'delivery_tag': 3, 'exchange': u'cinder-scheduler_fanout'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.313 7616 DEBUG kombu.channel [-] message_to_python.body=u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:22 10.35.0.15 cinder-volume 2014-07-11 22:15:22.314 3935 INFO cinder.volume.manager [-] Updating volume status
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.315 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: _consume connection.drain_events ret=None
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.315 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.040 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.315 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-2a7ff039-a4af-4a93-b649-89886cff5880\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:22.308805\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-slow\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPHSLOW\\", \\"free_capacity_gb\\": 1282, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"820d81e7763245a6a5c8c44f70da2c5a\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.316 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue cinder-scheduler -> <Exchange openstack(topic) bound to chan:1> -> cinder-scheduler bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f0f62445e50>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>, 'kwargs': {'exclusive': False, 'name': 'cinder-scheduler', 'exchange': <unbound Exchange openstack(topic)>, 'durable': True, 'routing_key': 'cinder-scheduler', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}}), TopicConsumer({'queue': <Queue cinder-scheduler.cinder -> <Exchange openstack(topic) bound to chan:1> -> cinder-scheduler.cinder bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f0f62445e50>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>, 'kwargs': {'exclusive': False, 'name': 'cinder-scheduler.cinder', 'exchange': <unbound Exchange openstack(topic)>, 'durable': True, 'routing_key': 'cinder-scheduler.cinder', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}}), FanoutConsumer({'queue': <Queue cinder-scheduler_fanout_f87a8a1c1ff5465288c10b5bae42db6e -> <Exchange cinder-scheduler_fanout(fanout) bound to chan:1> -> cinder-scheduler bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f0f62445e50>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>, 'kwargs': {'exclusive': False, 'name': 'cinder-scheduler_fanout_f87a8a1c1ff5465288c10b5bae42db6e', 'exchange': <unbound Exchange cinder-scheduler_fanout(fanout)>, 'durable': True, 'routing_key': 'cinder-scheduler', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}})]
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.316 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.316 7616 ERROR oslo.messaging.rpc.dispatcher [-] NBK: rpc server _dispatch_and_reply method=u'update_service_capabilities' message={u'args': {u'service_name': u'volume', u'host': u'cinder@rbd-slow', u'capabilities': {u'allocated_capacity_gb': 0, u'volume_backend_name': u'CEPHSLOW', u'free_capacity_gb': 1282, u'driver_version': u'1.1.0', u'total_capacity_gb': 1282, u'reserved_percentage': 0, u'vendor_name': u'Open Source', u'storage_protocol': u'ceph'}}, u'version': u'1.0', u'method': u'update_service_capabilities'}
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.317 7616 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, 'user_identity': u'- - - - -'}
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.318 7616 DEBUG kombu.channel [-] [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"1aac1828469d4830ba26b7610bb0e988\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.318 7616 ERROR kombu.transport.pyamqp [-] NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"1aac1828469d4830ba26b7610bb0e988\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.318 7616 ERROR kombu.messaging [-] NBK: _publish routing_key='' exchange='' message.body='{"oslo.message": "{\\"_unique_id\\": \\"1aac1828469d4830ba26b7610bb0e988\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.319 7616 DEBUG kombu.channel [-] [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"1aac1828469d4830ba26b7610bb0e988\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='', immediate=False, exchange='') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.319 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.320 7616 DEBUG kombu.channel [-] [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"f582895c3a2f405080160e2de6effb35\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.320 7616 ERROR kombu.transport.pyamqp [-] NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"f582895c3a2f405080160e2de6effb35\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.320 7616 ERROR kombu.messaging [-] NBK: _publish routing_key='' exchange='' message.body='{"oslo.message": "{\\"_unique_id\\": \\"f582895c3a2f405080160e2de6effb35\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.320 7616 DEBUG kombu.channel [-] [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"f582895c3a2f405080160e2de6effb35\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='', immediate=False, exchange='') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.321 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.321 7616 DEBUG kombu.channel [-] [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:22 10.35.0.15 cinder-scheduler 2014-07-11 22:15:22.323 7616 ERROR oslo.messaging.rpc.dispatcher [-] NBK: rpc server _dispatch_and_reply method=u'update_service_capabilities' took=0.000 got_reply=True exceptions=[] message={u'args': {u'service_name': u'volume', u'host': u'cinder@rbd-slow', u'capabilities': {u'allocated_capacity_gb': 0, u'volume_backend_name': u'CEPHSLOW', u'free_capacity_gb': 1282, u'driver_version': u'1.1.0', u'total_capacity_gb': 1282, u'reserved_percentage': 0, u'vendor_name': u'Open Source', u'storage_protocol': u'ceph'}}, u'version': u'1.0', u'method': u'update_service_capabilities'}
Jul 11 22:15:25 10.35.0.15 nova-network 4639 140561649928944 periodic_task periodic_task.py run_periodic_tasks Running periodic task FlatDHCPManager._disassociate_stale_fixed_ips run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:25 10.35.0.15 nova-network.log: /usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py:435: Warning: Incorrect datetime value: '2014-07-11 22:05:25+00:00' for column 'updated_at' at row 1
Jul 11 22:15:25 10.35.0.15 nova-network.log: cursor.execute(statement, parameters)
Jul 11 22:15:25 10.35.0.15 nova-network 4639 140561649928944 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7f519c3d4ed0>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:15:30 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:30 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1746 "" "python-novaclient"
Jul 11 22:15:30 10.35.0.13 nova-api 3619 139988495936240 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1920 time: 0.1068110
Jul 11 22:15:31 10.35.0.14 nova-network 9862 140599317968048 periodic_task periodic_task.py run_periodic_tasks Running periodic task FlatDHCPManager._disassociate_stale_fixed_ips run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:31 10.35.0.14 nova-network.log: /usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py:435: Warning: Incorrect datetime value: '2014-07-11 22:05:31+00:00' for column 'updated_at' at row 1
Jul 11 22:15:31 10.35.0.14 nova-network.log: cursor.execute(statement, parameters)
Jul 11 22:15:31 10.35.0.14 nova-network 9862 140599317968048 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:32 10.35.0.15 nova-cert 4487 140128677854896 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Got semaphore "compute_resources" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Got semaphore / lock "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Auditing locally available compute resources
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 driver driver.py update_status Updating host stats update_status /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:5260
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:38 10.35.0.14 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free ram (MB): 45575 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:409
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free disk (GB): 1620 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:410
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free VCPUs: 4 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:415
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: assignable PCI devices: [] _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:422
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Free ram (MB): 47217
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Free disk (GB): 1861
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 log log.py audit Free VCPUS: 3
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 resource_tracker resource_tracker.py _sync_compute_node Compute_service record updated for 10.35.0.14:node-10-35-0-14
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Semaphore / lock released "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 manager manager.py _reclaim_queued_deletes CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib64/python2.7/site-packages/nova/compute/manager.py:5388
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._run_image_cache_manager_pass run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Got semaphore "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Attempting to grab file lock "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Got file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Got semaphore / lock "do_register_storage_use" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Released file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Semaphore / lock released "do_register_storage_use" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Got semaphore "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Attempting to grab file lock "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Got file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Got semaphore / lock "do_get_storage_users" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py lock Released file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 lockutils lockutils.py inner Semaphore / lock released "do_get_storage_users" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _age_and_verify_cached_images Verify base images _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:525
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _age_and_verify_cached_images Image id yields fingerprint da39a3ee5e6b4b0d3255bfef95601890afd80709 _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:531
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _age_and_verify_cached_images Image id 8fb48209-3c53-4767-a4a4-a47d5f85db49 yields fingerprint 65d9d391ace10c6b3835b05a5346944fa3c9791c _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:531
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): checking
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): in use: on this node 1 local, 0 on other nodes sharing this instance storage
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): image is in use _handle_base_image /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:519
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf chown 43 /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:38 10.35.0.14 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf chown 43 /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _age_and_verify_cached_images Active base files: /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 imagecache imagecache.py _age_and_verify_cached_images Verification complete _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:567
Jul 11 22:15:38 10.35.0.14 nova-compute 9462 139687021030288 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 34.98 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Got semaphore "compute_resources" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Got semaphore / lock "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Auditing locally available compute resources
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 driver driver.py update_status Updating host stats update_status /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:5260
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:38 10.35.0.13 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:38 10.35.0.13 nova-compute 3510 139687513837808 driver driver.py get_instance_disk_info skipping disk for instance-00000001 as it does not have a path get_instance_disk_info /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:4765
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free ram (MB): 45444 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:409
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free disk (GB): 1619 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:410
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free VCPUs: 3 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:415
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: assignable PCI devices: [] _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:422
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Got semaphore "compute_resources" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Got semaphore / lock "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Auditing locally available compute resources
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 driver driver.py update_status Updating host stats update_status /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:5260
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:39 10.35.0.15 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free nova
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Free ram (MB): 47217
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Free disk (GB): 1861
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 log log.py audit Free VCPUS: 3
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 resource_tracker resource_tracker.py _sync_compute_node Compute_service record updated for 10.35.0.13:node-10-35-0-13
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Semaphore / lock released "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 manager manager.py _reclaim_queued_deletes CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib64/python2.7/site-packages/nova/compute/manager.py:5388
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._run_image_cache_manager_pass run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Got semaphore "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Attempting to grab file lock "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Got file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Got semaphore / lock "do_register_storage_use" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 driver driver.py get_instance_disk_info skipping disk for instance-00000002 as it does not have a path get_instance_disk_info /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:4765
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 driver driver.py get_instance_disk_info skipping disk for instance-00000003 as it does not have a path get_instance_disk_info /usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py:4765
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Released file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Semaphore / lock released "do_register_storage_use" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Got semaphore "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Attempting to grab file lock "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Got file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Got semaphore / lock "do_get_storage_users" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py lock Released file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 lockutils lockutils.py inner Semaphore / lock released "do_get_storage_users" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _age_and_verify_cached_images Verify base images _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:525
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _age_and_verify_cached_images Image id yields fingerprint da39a3ee5e6b4b0d3255bfef95601890afd80709 _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:531
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _age_and_verify_cached_images Image id 8fb48209-3c53-4767-a4a4-a47d5f85db49 yields fingerprint 65d9d391ace10c6b3835b05a5346944fa3c9791c _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:531
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): checking
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): in use: on this node 1 local, 0 on other nodes sharing this instance storage
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): image is in use _handle_base_image /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:519
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf chown 43 /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:39 10.35.0.13 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf chown 43 /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _list_backing_images a3712753-7081-4700-9041-752640884141 is a valid instance name _list_backing_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:292
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _age_and_verify_cached_images Active base files: /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 imagecache imagecache.py _age_and_verify_cached_images Verification complete _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:567
Jul 11 22:15:39 10.35.0.13 nova-compute 3510 139687513837808 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 33.59 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free ram (MB): 44692 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:409
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free disk (GB): 1618 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:410
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: free VCPUs: 2 _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:415
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _report_hypervisor_resource_view Hypervisor: assignable PCI devices: [] _report_hypervisor_resource_view /usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py:422
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Free ram (MB): 46705
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Free disk (GB): 1860
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 log log.py audit Free VCPUS: 2
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 resource_tracker resource_tracker.py _sync_compute_node Compute_service record updated for 10.35.0.15:node-10-35-0-15
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Semaphore / lock released "update_available_resource" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 manager manager.py _reclaim_queued_deletes CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib64/python2.7/site-packages/nova/compute/manager.py:5388
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 periodic_task periodic_task.py run_periodic_tasks Running periodic task ComputeManager._run_image_cache_manager_pass run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Got semaphore "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Attempting to grab file lock "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Got file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Got semaphore / lock "do_register_storage_use" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:39 10.35.0.14 cobalt-compute: 9469 140590155598896 periodic_task periodic_task.py run_periodic_tasks Running periodic task CobaltManager._refresh_host
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Released file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Semaphore / lock released "do_register_storage_use" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Got semaphore "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:168
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Attempting to grab file lock "storage-registry-lock" lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:178
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Got file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:206
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Got semaphore / lock "do_get_storage_users" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:248
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py lock Released file lock "storage-registry-lock" at /mnt/novadisk/nova/instances/locks/nova-storage-registry-lock lock /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:210
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 lockutils lockutils.py inner Semaphore / lock released "do_get_storage_users" inner /usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py:252
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _age_and_verify_cached_images Verify base images _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:525
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _age_and_verify_cached_images Image id yields fingerprint da39a3ee5e6b4b0d3255bfef95601890afd80709 _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:531
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _age_and_verify_cached_images Image id 8fb48209-3c53-4767-a4a4-a47d5f85db49 yields fingerprint 65d9d391ace10c6b3835b05a5346944fa3c9791c _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:531
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): checking
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): in use: on this node 2 local, 0 on other nodes sharing this instance storage
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _handle_base_image image 8fb48209-3c53-4767-a4a4-a47d5f85db49 at (/mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c): image is in use _handle_base_image /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:519
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 processutils processutils.py execute Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf chown 43 /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:154
Jul 11 22:15:39 10.35.0.14 cobalt-compute: 9469 140590155598896 periodic_task periodic_task.py run_periodic_tasks Running periodic task CobaltManager._clean
Jul 11 22:15:39 10.35.0.14 cobalt-compute: 9469 140590155598896 vmsconn vmsconn.py _clean_lvm_symlinks Starting to clean symlinks in /dev/nova
Jul 11 22:15:39 10.35.0.14 cobalt-compute: 9469 140590155598896 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/libvirt_save
Jul 11 22:15:39 10.35.0.14 cobalt-compute: 9469 140590155598896 vmsconn vmsconn.py _clean_lvm_symlinks Link is still active /dev/nova/novadisk
Jul 11 22:15:39 10.35.0.14 cobalt-compute: 9469 140590155598896 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds
Jul 11 22:15:39 10.35.0.15 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf chown 43 /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 processutils processutils.py execute Result was 0 execute /usr/lib64/python2.7/site-packages/nova/openstack/common/processutils.py:187
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _list_backing_images ea07a714-0252-486d-8806-8d03efde026c is a valid instance name _list_backing_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:292
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _list_backing_images 1537482d-c1f5-4d6a-aadc-67e8a635af8f is a valid instance name _list_backing_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:292
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _age_and_verify_cached_images Active base files: /mnt/novadisk/nova/instances/_base/65d9d391ace10c6b3835b05a5346944fa3c9791c
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 imagecache imagecache.py _age_and_verify_cached_images Verification complete _age_and_verify_cached_images /usr/lib64/python2.7/site-packages/nova/virt/libvirt/imagecache.py:567
Jul 11 22:15:39 10.35.0.15 nova-compute 4490 140697163517616 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 33.53 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7f519c3d4ed0>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:15:40 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:15:40 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1746 "" "python-novaclient"
Jul 11 22:15:40 10.35.0.13 nova-api 3619 139988495936240 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1920 time: 0.1009729
Jul 11 22:15:40 10.35.0.13 nova-cert 3508 140123336157776 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:44 10.35.0.13 nova-network 3642 140255031865008 periodic_task periodic_task.py run_periodic_tasks Running periodic task FlatDHCPManager._disassociate_stale_fixed_ips run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:44 10.35.0.13 nova-network.log: /usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py:435: Warning: Incorrect datetime value: '2014-07-11 22:05:44+00:00' for column 'updated_at' at row 1
Jul 11 22:15:44 10.35.0.13 nova-network.log: cursor.execute(statement, parameters)
Jul 11 22:15:44 10.35.0.13 nova-network 3642 140255031865008 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:47 10.35.0.14 nova-cert 9459 140153095137232 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:47 10.35.0.13 nova-scheduler 3514 139794942292688 periodic_task periodic_task.py run_periodic_tasks Running periodic task SchedulerManager._expire_reservations run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:47 10.35.0.13 nova-scheduler 3514 139794942292688 periodic_task periodic_task.py run_periodic_tasks Running periodic task SchedulerManager._run_periodic_tasks run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:47 10.35.0.13 nova-scheduler 3514 139794942292688 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:48 10.35.0.15 nova-scheduler 4494 140686411265200 periodic_task periodic_task.py run_periodic_tasks Running periodic task SchedulerManager._expire_reservations run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:48 10.35.0.15 nova-scheduler 4494 140686411265200 periodic_task periodic_task.py run_periodic_tasks Running periodic task SchedulerManager._run_periodic_tasks run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:15:48 10.35.0.15 nova-scheduler 4494 140686411265200 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.352 3934 DEBUG kombu.channel [-] [Kombu channel:1] exchange_declare(nowait=False, exchange='cinder-scheduler_fanout', auto_delete=False, passive=False, arguments=None, type='fanout', durable=True) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.353 3934 DEBUG kombu.channel [-] [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.353 3934 ERROR kombu.transport.pyamqp [-] NBK: prepare_message body='{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.353 3934 ERROR kombu.messaging [-] NBK: _publish routing_key='' exchange='cinder-scheduler_fanout' message.body='{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.353 3934 DEBUG kombu.channel [-] [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='', immediate=False, exchange='cinder-scheduler_fanout') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.354 3934 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.354 3934 DEBUG kombu.channel [-] [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.355 7616 DEBUG kombu.channel [-] [Kombu channel:1] message_to_python(Message({'body': u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'3', 'redelivered': False, 'routing_key': u'', 'delivery_tag': 4, 'exchange': u'cinder-scheduler_fanout'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>})) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.355 7616 ERROR kombu.transport.pyamqp [-] NBK: message_to_python raw_message.__dict__={'body': u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}', 'delivery_info': {'consumer_tag': u'3', 'redelivered': False, 'routing_key': u'', 'delivery_tag': 4, 'exchange': u'cinder-scheduler_fanout'}, 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'utf-8', 'content_type': u'application/json'}, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.355 7616 DEBUG kombu.channel [-] message_to_python.body=u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}' __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:59
Jul 11 22:15:58 10.35.0.15 cinder-volume 2014-07-11 22:15:58.356 3934 INFO cinder.volume.manager [-] Updating volume status
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.357 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: _consume connection.drain_events ret=None
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.357 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.030 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.357 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: acknowledge self._raw_message.body=u'{"oslo.message": "{\\"_context_domain\\": null, \\"_context_request_id\\": \\"req-ba2439df-3354-445c-89e9-02b3c9cd9594\\", \\"_context_quota_class\\": null, \\"_context_service_catalog\\": [], \\"_context_auth_token\\": null, \\"_context_user_id\\": null, \\"_context_is_admin\\": true, \\"version\\": \\"1.0\\", \\"_context_timestamp\\": \\"2014-07-11T22:15:58.350960\\", \\"_context_project_domain\\": null, \\"_context_user\\": null, \\"method\\": \\"update_service_capabilities\\", \\"_context_remote_address\\": null, \\"_context_roles\\": [\\"admin\\"], \\"args\\": {\\"service_name\\": \\"volume\\", \\"host\\": \\"cinder@rbd-fast\\", \\"capabilities\\": {\\"allocated_capacity_gb\\": 0, \\"volume_backend_name\\": \\"CEPH\\", \\"free_capacity_gb\\": 1281, \\"driver_version\\": \\"1.1.0\\", \\"total_capacity_gb\\": 1282, \\"reserved_percentage\\": 0, \\"vendor_name\\": \\"Open Source\\", \\"storage_protocol\\": \\"ceph\\"}}, \\"_unique_id\\": \\"71ca368e80314d33afb59b98b70bb382\\", \\"_context_project_name\\": null, \\"_context_read_deleted\\": \\"no\\", \\"_context_user_identity\\": \\"- - - - -\\", \\"_context_tenant\\": null, \\"_context_project_id\\": null, \\"_context_user_domain\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.358 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: _consume self.do_consume=False self.consumers=[TopicConsumer({'queue': <Queue cinder-scheduler -> <Exchange openstack(topic) bound to chan:1> -> cinder-scheduler bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f0f62445e50>, 'tag': '1', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>, 'kwargs': {'exclusive': False, 'name': 'cinder-scheduler', 'exchange': <unbound Exchange openstack(topic)>, 'durable': True, 'routing_key': 'cinder-scheduler', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}}), TopicConsumer({'queue': <Queue cinder-scheduler.cinder -> <Exchange openstack(topic) bound to chan:1> -> cinder-scheduler.cinder bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f0f62445e50>, 'tag': '2', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>, 'kwargs': {'exclusive': False, 'name': 'cinder-scheduler.cinder', 'exchange': <unbound Exchange openstack(topic)>, 'durable': True, 'routing_key': 'cinder-scheduler.cinder', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}}), FanoutConsumer({'queue': <Queue cinder-scheduler_fanout_f87a8a1c1ff5465288c10b5bae42db6e -> <Exchange cinder-scheduler_fanout(fanout) bound to chan:1> -> cinder-scheduler bound to chan:1>, 'callback': <oslo.messaging._drivers.amqpdriver.AMQPListener object at 0x7f0f62445e50>, 'tag': '3', 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>, 'kwargs': {'exclusive': False, 'name': 'cinder-scheduler_fanout_f87a8a1c1ff5465288c10b5bae42db6e', 'exchange': <unbound Exchange cinder-scheduler_fanout(fanout)>, 'durable': True, 'routing_key': 'cinder-scheduler', 'queue_arguments': {}, 'auto_delete': False, 'channel': <kombu.transport.pyamqp.Channel object at 0x7f0f62445d50>}})]
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.358 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: _consume connection.drain_events timeout=None
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.358 7616 ERROR oslo.messaging.rpc.dispatcher [-] NBK: rpc server _dispatch_and_reply method=u'update_service_capabilities' message={u'args': {u'service_name': u'volume', u'host': u'cinder@rbd-fast', u'capabilities': {u'allocated_capacity_gb': 0, u'volume_backend_name': u'CEPH', u'free_capacity_gb': 1281, u'driver_version': u'1.1.0', u'total_capacity_gb': 1282, u'reserved_percentage': 0, u'vendor_name': u'Open Source', u'storage_protocol': u'ceph'}}, u'version': u'1.0', u'method': u'update_service_capabilities'}
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.359 7616 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, 'user_identity': u'- - - - -'}
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.360 7616 DEBUG kombu.channel [-] [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"0e043c179d814737a61e839c0b8f6d6f\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.360 7616 ERROR kombu.transport.pyamqp [-] NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"0e043c179d814737a61e839c0b8f6d6f\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.360 7616 ERROR kombu.messaging [-] NBK: _publish routing_key='' exchange='' message.body='{"oslo.message": "{\\"_unique_id\\": \\"0e043c179d814737a61e839c0b8f6d6f\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}'
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.361 7616 DEBUG kombu.channel [-] [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"0e043c179d814737a61e839c0b8f6d6f\\", \\"failure\\": null, \\"result\\": null}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='', immediate=False, exchange='') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.361 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.000 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.362 7616 DEBUG kombu.channel [-] [Kombu channel:1] prepare_message('{"oslo.message": "{\\"_unique_id\\": \\"b2616269dad94df6a4ed5ee59e2dbb12\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 0, 'application/json', 'utf-8', {}, {'delivery_mode': 2}) __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.362 7616 ERROR kombu.transport.pyamqp [-] NBK: prepare_message body='{"oslo.message": "{\\"_unique_id\\": \\"b2616269dad94df6a4ed5ee59e2dbb12\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', headers={}, properties={'delivery_mode': 2}
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.362 7616 ERROR kombu.messaging [-] NBK: _publish routing_key='' exchange='' message.body='{"oslo.message": "{\\"_unique_id\\": \\"b2616269dad94df6a4ed5ee59e2dbb12\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}'
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.363 7616 DEBUG kombu.channel [-] [Kombu channel:1] _basic_publish(Message({'body': '{"oslo.message": "{\\"_unique_id\\": \\"b2616269dad94df6a4ed5ee59e2dbb12\\", \\"failure\\": null, \\"result\\": null, \\"ending\\": true}", "oslo.version": "2.0"}', 'properties': {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': 'utf-8', 'content_type': 'application/json'}, 'channel': None}), mandatory=False, routing_key='', immediate=False, exchange='') __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.363 7616 ERROR oslo.messaging._drivers.impl_rabbit [-] NBK: ensure method=oslo.messaging._drivers.impl_rabbit._publish took=0.010 sec retry=0 got_ret=True exceptions=[]
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.363 7616 DEBUG kombu.channel [-] [Kombu channel:1] close() __wrapped /usr/lib64/python2.7/site-packages/kombu/utils/debug.py:56
Jul 11 22:15:58 10.35.0.15 cinder-scheduler 2014-07-11 22:15:58.365 7616 ERROR oslo.messaging.rpc.dispatcher [-] NBK: rpc server _dispatch_and_reply method=u'update_service_capabilities' took=0.010 got_reply=True exceptions=[] message={u'args': {u'service_name': u'volume', u'host': u'cinder@rbd-fast', u'capabilities': {u'allocated_capacity_gb': 0, u'volume_backend_name': u'CEPH', u'free_capacity_gb': 1281, u'driver_version': u'1.1.0', u'total_capacity_gb': 1282, u'reserved_percentage': 0, u'vendor_name': u'Open Source', u'storage_protocol': u'ceph'}}, u'version': u'1.0', u'method': u'update_service_capabilities'}
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f519cefc8d0>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7f519c3d4ed0>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:16:00 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:16:00 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1746 "" "python-novaclient"
Jul 11 22:16:00 10.35.0.13 nova-api 3619 139988495936240 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1920 time: 0.2649701
Jul 11 22:16:05 10.35.0.14 nova-scheduler 9467 140353575370832 periodic_task periodic_task.py run_periodic_tasks Running periodic task SchedulerManager._expire_reservations run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:16:05 10.35.0.14 nova-scheduler 9467 140353575370832 periodic_task periodic_task.py run_periodic_tasks Running periodic task SchedulerManager._run_periodic_tasks run_periodic_tasks /usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py:178
Jul 11 22:16:05 10.35.0.14 nova-scheduler 9467 140353575370832 loopingcall loopingcall.py _inner Dynamic looping call sleeping for 60.00 seconds _inner /usr/lib64/python2.7/site-packages/nova/openstack/common/loopingcall.py:132
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py __call__ Authenticating user token __call__ /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:602
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py _remove_auth_headers Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:661
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py _cache_get Returning cached token _cache_get /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1010
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 auth_token auth_token.py _build_user_headers Received request from user: 6bb7c6d04a8b43dc83cf9e851b92b3fb with project_id : e57d3cef3061401a900a9751b1fb0bc4 and roles: _member_,Admin,ProjectAdmin _build_user_headers /usr/lib64/python2.7/site-packages/keystoneclient/middleware/auth_token.py:909
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 middleware middleware.py __call__ Matched GET /e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:100
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 middleware middleware.py __call__ Route path: '/{project_id}/servers/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f3f9eddf7d0>} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:102
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 middleware middleware.py __call__ Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f3f9eddf7d0>, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4', 'id': u'7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c'} __call__ /usr/lib64/python2.7/site-packages/routes/middleware.py:103
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 wsgi wsgi.py _process_stack Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7f3f9e939c50>>' (Content-type='None', Accept='application/json') _process_stack /usr/lib64/python2.7/site-packages/nova/api/openstack/wsgi.py:945
Jul 11 22:16:10 10.35.0.15 pound: 10.35.2.8:8774 172.31.0.13 - - [11/Jul/2014:22:16:10 +0000] "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" 200 1746 "" "python-novaclient"
Jul 11 22:16:10 10.35.0.15 nova-api 4610 139911229646960 log log.py write 172.31.0.13,10.35.0.15 "GET /v2/e57d3cef3061401a900a9751b1fb0bc4/servers/7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c HTTP/1.1" status: 200 len: 1920 time: 0.1037478
!!! client times out
Jul 11 22:16:11 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py _error_callback Timed out waiting for RPC response: timed out
Jul 11 22:16:11 10.35.0.14 cobalt-compute: 9469 140590168431664 impl_rabbit impl_rabbit.py ensure NBK: ensure method=oslo.messaging._drivers.impl_rabbit._consume took=0.100 sec retry=1 got_ret=False exceptions=['connection_errors: [Errno 104] Connection reset by peer', 'connection_errors: timed out']
Jul 11 22:16:11 10.35.0.14 cobalt-compute: 9469 140590168431664 amqpdriver amqpdriver.py _poll_connection NBK: conn.consume timed out msg_id='8fd1d3cb00f54d70bb6cdbc62700a0de'
Jul 11 22:16:11 10.35.0.14 cobalt-compute: 9469 140590168431664 client client.py call NBK: rpc client call method='allocate_for_instance' target=<Target topic=network, version=1.9, server=10.35.0.14> took=0.130 timeout=60 got_result=False exceptions=[MessagingTimeout('Timed out waiting for a reply to message ID 8fd1d3cb00f54d70bb6cdbc62700a0de',)] kwargs={'vpn': False, 'requested_networks': [[u'fd3212a8-c4e6-4f8a-a83c-ba632264d1ce', None]], 'instance_id': '7e4c93c5-83d6-40e3-baa2-ce3f8e26cd6c', 'host': u'10.35.0.14', 'rxtx_factor': 1.0, 'macs': None, 'project_id': u'e57d3cef3061401a900a9751b1fb0bc4'}
!!! client timed out for 8fd1d3cb00f54d70bb6cdbc62700a0de
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment