Skip to content

Instantly share code, notes, and snippets.

@smalleni

smalleni/gist:a90133782d4f903ef995339293a45b8f Secret

Last active March 7, 2017 21:26
Show Gist options
  • Save smalleni/a90133782d4f903ef995339293a45b8f to your computer and use it in GitHub Desktop.
Save smalleni/a90133782d4f903ef995339293a45b8f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
2017-03-07 20:22:17Z [overcloud.ComputeHostsDeployment.30]: CREATE_IN_PROGRESS state changed [245/1919]
2017-03-07 20:22:19Z [overcloud.ComputeHostsDeployment.10]: CREATE_IN_PROGRESS state changed
2017-03-07 20:22:23Z [overcloud.ComputeHostsDeployment.12]: CREATE_IN_PROGRESS state changed
2017-03-07 20:22:30Z [overcloud.ControllerHostsDeployment.1]: SIGNAL_IN_PROGRESS Signal: deployment 2fc950da-ce5a-44b4-83c7-778d8d902d06 succeeded
2017-03-07 20:22:31Z [overcloud.ComputeHostsDeployment.47]: CREATE_IN_PROGRESS state changed
2017-03-07 20:22:33Z [overcloud.ControllerHostsDeployment.1]: CREATE_COMPLETE state changed
2017-03-07 20:22:40Z [overcloud.ComputeHostsDeployment.22]: CREATE_IN_PROGRESS state changed
2017-03-07 20:23:00Z [overcloud.ComputeHostsDeployment.43]: CREATE_IN_PROGRESS state changed
2017-03-07 20:23:32Z [overcloud.ComputeHostsDeployment.46]: CREATE_IN_PROGRESS state changed
2017-03-07 20:24:04Z [overcloud.ComputeHostsDeployment.31]: CREATE_IN_PROGRESS state changed
2017-03-07 20:24:16Z [overcloud.ControllerHostsDeployment.0]: SIGNAL_IN_PROGRESS Signal: deployment 86d37495-8f12-4dd0-9a2f-d7235ca70aa8 succeeded
2017-03-07 20:24:17Z [overcloud.ControllerHostsDeployment.0]: CREATE_COMPLETE state changed
2017-03-07 20:24:35Z [overcloud.ControllerHostsDeployment.2]: SIGNAL_IN_PROGRESS Signal: deployment e0da3769-e221-4c01-aa13-7f15845e5e5a succeeded
2017-03-07 20:24:36Z [overcloud.ControllerHostsDeployment.2]: CREATE_COMPLETE state changed
2017-03-07 20:24:36Z [overcloud.ControllerHostsDeployment]: CREATE_COMPLETE Stack CREATE completed successfully
2017-03-07 20:24:37Z [overcloud.ControllerHostsDeployment]: CREATE_COMPLETE state changed
2017-03-07 20:24:41Z [overcloud.ComputeHostsDeployment.4]: CREATE_IN_PROGRESS state changed
2017-03-07 20:25:18Z [overcloud.ComputeHostsDeployment.15]: CREATE_IN_PROGRESS state changed
2017-03-07 20:25:55Z [overcloud.ComputeHostsDeployment.33]: CREATE_IN_PROGRESS state changed
2017-03-07 20:26:36Z [overcloud.ComputeHostsDeployment.25]: CREATE_IN_PROGRESS state changed
2017-03-07 20:27:16Z [overcloud.ComputeHostsDeployment.44]: CREATE_IN_PROGRESS state changed
2017-03-07 20:28:00Z [overcloud.ComputeHostsDeployment.5]: CREATE_IN_PROGRESS state changed
2017-03-07 20:28:49Z [overcloud.ComputeHostsDeployment.13]: CREATE_IN_PROGRESS state changed
2017-03-07 20:29:40Z [overcloud.ComputeHostsDeployment.3]: CREATE_IN_PROGRESS state changed
^[2017-03-07 20:30:33Z [overcloud.ComputeHostsDeployment.34]: CREATE_IN_PROGRESS state changed
2017-03-07 20:31:30Z [overcloud.ComputeHostsDeployment.19]: CREATE_IN_PROGRESS state changed
2017-03-07 20:32:27Z [overcloud.ComputeHostsDeployment.48]: CREATE_IN_PROGRESS state changed
2017-03-07 20:33:46Z [overcloud.ComputeHostsDeployment.1]: CREATE_IN_PROGRESS state changed
2017-03-07 20:35:58Z [overcloud.ComputeHostsDeployment.37]: CREATE_IN_PROGRESS state changed
2017-03-07 20:38:30Z [overcloud.ComputeHostsDeployment.49]: CREATE_IN_PROGRESS state changed
^[2017-03-07 20:41:15Z [overcloud.ComputeHostsDeployment.38]: CREATE_IN_PROGRESS state changed
^[^[^[^[^[^[^[^[^[^[^[^[2017-03-07 20:44:18Z [overcloud.ComputeHostsDeployment.26]: CREATE_IN_PROGRESS state changed
2017-03-07 20:47:52Z [overcloud.ComputeHostsDeployment.20]: CREATE_IN_PROGRESS state changed
2017-03-07 20:53:06Z [overcloud.ComputeHostsDeployment.45]: SIGNAL_IN_PROGRESS Signal: deployment fe5668f2-4e1d-4944-9f6a-3448c4b712fb succeeded
2017-03-07 20:53:11Z [overcloud.ComputeHostsDeployment.45]: CREATE_COMPLETE state changed
^[ERROR: Timed out waiting for a reply to message ID 14aee5a7c8b3452dab6cfb146d2febb7
real 99m29.356s
user 0m9.784s
sys 0m1.655s
Tue Mar 7 21:06:11 UTC 2017
[root@c02-h10-r620 stack]# grep -nr 14aee5a7c8b3452dab6cfb146d2febb7 /var/log
/var/log/heat/heat-engine.log:776793:2017-03-07 20:56:11.222 25131 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: 14aee5a7c8b3452dab6cfb146d2febb7 reply to reply_20afe0fef7b047c68a0ee7e1ba0a62e3 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:194
/var/log/heat/heat-engine.log:777841:2017-03-07 21:06:53.522 25131 DEBUG oslo_messaging._drivers.amqpdriver [req-a579de0b-3b9c-4e87-a20a-f257a8a27d53 67d943cc3ddd43dc83a85de09dd6cd3c 6d1dd3559ead444fbcc5edbc75fb0f7b - - -] sending reply msg_id: 14aee5a7c8b3452dab6cfb146d2febb7 reply queue: reply_20afe0fef7b047c68a0ee7e1ba0a62e3 time elapsed: 642.298795432s _send_reply /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:73
/var/log/heat/heat-api.log:23357:2017-03-07 20:56:11.219 25138 DEBUG oslo_messaging._drivers.amqpdriver [req-a579de0b-3b9c-4e87-a20a-f257a8a27d53 67d943cc3ddd43dc83a85de09dd6cd3c 6d1dd3559ead444fbcc5edbc75fb0f7b - default default] CALL msg_id: 14aee5a7c8b3452dab6cfb146d2febb7 exchange 'heat' topic 'engine' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
/var/log/heat/heat-api.log:23358:2017-03-07 21:06:11.221 25138 ERROR heat.common.wsgi [req-a579de0b-3b9c-4e87-a20a-f257a8a27d53 67d943cc3ddd43dc83a85de09dd6cd3c 6d1dd3559ead444fbcc5edbc75fb0f7b - default default] Unexpected error occurred serving API: Timed out waiting for a reply to message ID 14aee5a7c8b3452dab6cfb146d2febb7
/var/log/heat/heat-api.log:23386:2017-03-07 21:06:11.221 25138 ERROR heat.common.wsgi MessagingTimeout: Timed out waiting for a reply to message ID 14aee5a7c8b3452dab6cfb146d2febb7
/var/log/heat/heat-api.log:23388:2017-03-07 21:06:11.230 25138 DEBUG heat.common.serializers [req-a579de0b-3b9c-4e87-a20a-f257a8a27d53 67d943cc3ddd43dc83a85de09dd6cd3c 6d1dd3559ead444fbcc5edbc75fb0f7b - default default] JSON response : {"explanation": "The server has either erred or is incapable of performing the requested operation.", "code": 500, "error": {"message": "Timed out waiting for a reply to message ID 14aee5a7c8b3452dab6cfb146d2febb7", "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/heat/api/middleware/fault.py\", line 162, in process_request\n return req.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1299, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1263, in call_application\n app_iter = application(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/oslo_middleware/base.py\", line 126, in __call__\n response = req.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1299, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1263, in call_application\n app_iter = application(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/heat/common/wsgi.py\", line 584, in __call__\n response = req.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1299, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1263, in call_application\n app_iter = application(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/osprofiler/web.py\", line 108, in __call__\n return request.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1299, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1263, in call_application\n app_iter = application(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/heat/common/wsgi.py\", line 584, in __call__\n response = req.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1299, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1263, in call_application\n app_iter = application(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py\", line 323, in __call__\n response = req.get_response(self._app)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1299, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1263, in call_application\n app_iter = application(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/heat/common/wsgi.py\", line 584, in __call__\n response = req.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1299, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1263, in call_application\n app_iter = application(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 144, in __call__\n return resp(environ, start_response)\n File \"/usr/lib/python2.7/site-packages/routes/middleware.py\", line 131, in __call__\n response = self.app(environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 144, in __call__\n return resp(environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/heat/common/wsgi.py\", line 874, in __call__\n raise translate_exception(err, request.best_match_language())\nMessagingTimeout: Timed out waiting for a reply to message ID 14aee5a7c8b3452dab6cfb146d2febb7\n", "type": "MessagingTimeout"}, "title": "Internal Server Error"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:40
/var/log/heat/heat-api.log:23390:2017-03-07 21:06:53.533 25138 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 14aee5a7c8b3452dab6cfb146d2febb7 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
/var/log/heat/heat-api.log:23391:2017-03-07 21:06:53.534 25138 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 14aee5a7c8b3452dab6cfb146d2febb7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment