Skip to content

Instantly share code, notes, and snippets.

@a1git

a1git/gist:9abf881940097b607b880ed3d905e816 Secret

Created August 13, 2019 09:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save a1git/9abf881940097b607b880ed3d905e816 to your computer and use it in GitHub Desktop.
Save a1git/9abf881940097b607b880ed3d905e816 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
root@c2-cinder-api-container-ac86f1fd:/openstack/venvs/cinder-18.1.9# source bin/activate
(cinder-18.1.9) root@c2-cinder-api-container-ac86f1fd:/openstack/venvs/cinder-18.1.9# cinder-manage --debug db online_data_migrations --max_count 1
Error attempting to run shared_targets_online_data_migration
+--------------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+--------------------------------------------+--------------+-----------+
| attachment_specs_online_data_migration | 0 | 0 |
| backup_service_online_migration | 0 | 0 |
| service_uuids_online_data_migration | 0 | 0 |
| shared_targets_online_data_migration | 0 | 0 |
| volume_service_uuids_online_data_migration | 0 | 0 |
+--------------------------------------------+--------------+-----------+
Some migrations failed unexpectedly. Check log for details.
(cinder-18.1.9) root@c2-cinder-api-container-ac86f1fd:/openstack/venvs/cinder-18.1.9# journalctl --since "1 hour ago" | grep cinder.cmd.manage
Aug 13 05:00:18 c2-cinder-api-container-ac86f1fd cinder-manage[89976]: 2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage [req-5471eec7-5fe6-4954-a6fc-7eab61c5f530 - - - - -] Error attempting to run shared_targets_online_data_migration: MessagingTimeout: Timed out waiting for a reply to message ID d35ffa51455f49b2a85d852832b657b2
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage Traceback (most recent call last):
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/cmd/manage.py", line 351, in _run_migration
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage found, done = migration_meth(ctxt, count)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/cmd/manage.py", line 128, in shared_targets_online_data_migration
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage non_shared_hosts, total_vols_to_update = _get_non_shared_target_hosts(ctxt)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/cmd/manage.py", line 111, in _get_non_shared_target_hosts
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage capabilities = rpcapi.get_capabilities(ctxt, service.host, True)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 341, in get_capabilities
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage return cctxt.call(ctxt, 'get_capabilities', discover=discover)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage retry=self.retry)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage retry=retry)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage call_monitor_timeout, retry=retry)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage call_monitor_timeout)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage message = self.waiters.get(msg_id, timeout=timeout)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage 'to message ID %s' % msg_id)
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage MessagingTimeout: Timed out waiting for a reply to message ID d35ffa51455f49b2a85d852832b657b2
2019-08-13 05:00:18.648 89976 ERROR cinder.cmd.manage
@a1git
Copy link
Author

a1git commented Aug 13, 2019

root@c3-utility-container-3938e142:~# openstack volume service list
+------------------+------------------------------------------------+------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+------------------------------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | c2-cinder-api-container-ac86f1fd | nova | enabled | up | 2019-08-13T09:06:22.000000 |
| cinder-scheduler | c1-cinder-api-container-d6fb5a8f | nova | enabled | up | 2019-08-13T09:06:23.000000 |
| cinder-scheduler | c3-cinder-api-container-a16fa64b | nova | enabled | up | 2019-08-13T09:06:22.000000 |
| cinder-volume | c2-cinder-volumes-container-59b55633@ssd_iscsi | nova | enabled | down | 2019-03-28T20:54:34.000000 |
| cinder-volume | c2-cinder-volumes-container-59b55633@ssd_nfs | nova | enabled | down | 2019-03-28T20:54:34.000000 |
| cinder-volume | c1-cinder-volumes-container-a96c1c74@ssd_iscsi | nova | enabled | down | 2019-03-28T20:54:27.000000 |
| cinder-volume | c1-cinder-volumes-container-a96c1c74@ssd_nfs | nova | enabled | down | 2019-03-28T20:54:26.000000 |
| cinder-volume | c3-cinder-volumes-container-15e0cc90@ssd_iscsi | nova | enabled | down | 2019-03-28T20:54:30.000000 |
| cinder-volume | c3-cinder-volumes-container-15e0cc90@ssd_nfs | nova | enabled | down | 2019-03-28T20:54:29.000000 |
| cinder-volume | c2@ssd_iscsi | nova | enabled | up | 2019-08-13T09:06:19.000000 |
| cinder-volume | c2@ssd_nfs | nova | enabled | up | 2019-08-13T09:06:22.000000 |
| cinder-volume | c1@ssd_iscsi | nova | enabled | up | 2019-08-13T09:06:16.000000 |
| cinder-volume | c1@ssd_nfs | nova | enabled | up | 2019-08-13T09:06:17.000000 |
| cinder-volume | c3@ssd_iscsi | nova | enabled | up | 2019-08-13T09:06:24.000000 |
| cinder-volume | c3@ssd_nfs | nova | enabled | up | 2019-08-13T09:06:18.000000 |
+------------------+------------------------------------------------+------+---------+-------+----------------------------+

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment