openstack-ansible rebuild rabbitmq cluster

openstack-ansible rebuild rabbitmq cluster recovery

========> This is what i did to rebuild RabbitMQ cluster for Openstack-ansible <===============

1. stop all services on 3 rabbitmq containers, if it fail then kill -9 but clean up all the existing process. 
2. yum remove rabbitmq-server  (remove package)
3. rm -rf /var/lib/rabbitmq/mnesia/*   (remove all dirty files) 
4. ssh ostack-osa (ansible host)
5. cd /opt/openstack-ansible/playbooks/
6. openstack-ansible rabbitmq-install.yml -e rabbitmq_upgrade=true    (It will install fresh new cluster)
7. After successfully installation verify cluster (rabbitmqctl cluster_status) 
8. You need to re-create queue for each services (nova, neutron, glance, heat)
        a. find each service password from ostack-osa:/etc/openstack_deploy/user_secrets.yml   ( neutron_rabbitmq_password: c85a92859de407369c8d161b228d918bb)
        b. go to any rabbitmq container and run following command one by one for each services
                1. rabbitmqctl add_user neutron <neutron_rabbitmq_password>
                2. rabbitmqctl add_vhost /neutron
                3. rabbitmqctl set_permissions neutron -p /neutron ".*" ".*" ".*"
9. After creating all queues you can verify them with following command
       a. rabbitmqctl list_queues -p /nova
       b. rabbitmqctl list_queues -p /neutron
       c. rabbitmqctl list_queues -p /heat
       d. rabbitmqctl list_queues -p /glance
10. You are done here, now you can just create test VM and delete to verify functionality.