gistfile1.txt

Current recommended steps to start cluster are:
 
Start all supporting nodes, if available - Quantum, Cinder, Swift - all except controllers and computes.
Then, go to controller nodes.
If you know exactly, which controller was shut down last and you are sure this controller was shut down gracefully - go to step 6 and start this controller first with new Galera cluster creation. If you don't, or somebody really yanked a global power cord - start all controllers, better without big pauses between each controller start.
It should help RabbitMQ to assemble to cluster automatically.

Hint: You can use mcollective remote exec plugin to speed up clusterwide operations (use with caution: it will run issued remote commands in random order!), f.e.:
fuel-master-node# mco rpc -v execute_shell_command execute cmd="mysql -uroot -e \"show global status like 'wsrep%'\" | grep -e size -e status -e state; cat /var/lib/mysql/grastate.dat"
Start checking from MySQL/Galera.

Essential info about Galera cluster restart available here:
http://www.codership.com/wiki/doku.php?id=mysql_galera_restart

All operations listed below require root access rights!

     2. Run mysql command on every controller - you should get MySQL prompt. If no MySQL prompt appears - make one single attempt to restart mysql service on that controller with service mysql restart command (In case of pacemaker OCF controlled resources, use crm_resource -r p_mysql --force-start).
 
            3. Run   show status like "wsrep%";    command on every controller with started up MySQL.
Check wsrep_local_state_comment variable. If it is Synced - it means this node of your Galera cluster operates somehow. Most probably it should be in Initialized state, that means Galera node is up, but waiting for master node to sync with.

    4. Check wsrep_local_state_uuid and wsrep_cluster_state_uuid variables. If both has the same value - current node is synchronized with the cluster.
Your goal is to find the greatest uuid value. Controller with the greatest uuid value has the most recent data replica. Since the replica may be inconsistent due to unfinished transaction - it is great idea to back up MySQL DB on every controller before restoring Galera cluster operation.

    5. Check /var/lib/mysql/grastate.dat on every controller. This file contains the same wsrep_local_state_uuid as you checked in previous step (4). If it is greater than wsrep_cluster_state_uuid (checked in previous step) it indicates non-finished transaction.  In most cases you will not be able to recover such transaction, so, simply run rm -f /var/lib/grastate.dat on every controller with non-starting MySQL. But leave this file on controllers with MySQL started!
There is one more important parameter - seqno. 
If it looks like  seqno: -1 it mean this node either crashed or is in inconsistent state. Avoid starting new cluster from such nodes!
   
   6. As soon as you have selected the node to start Galera cluster - perform the following commands:
Move /var/lib/mysql/grastate.dat to some other place (for the case of possible issues, described below) from every node, except that one you have selected to start up new cluster.
Run service mysql stop (for OCF case use crm_resource -r p_mysql --force-stop) to shut down mysql.
Run service mysql start --wsrep-cluster-address=gcomm:// to start up new cluster. The remained Galera nodes should join new cluster automatically. For OCF scripts case, you should manually edit /etc/mysql/conf.d/wsrep.cnf and temporary comment existing wsrep-cluster-address setting, add wsrep-cluster-address="gcomm://", and after that, use crm_resource -r p_mysql --force-start to launch mysql
Wait some time until Galera cluster synchronization finishes. 
Stop or kill mysql and start it again on the nodes, where mysql  failed to start in step #2. All such nodes should join cluster automatically.
Run show status like "wsrep%"; command to check cluster status.
Check wsrep_cluster_size value - it shows the number of the nodes currently joined cluster.
Check wsrep_incoming_addresses value - addresses:ports of currently joined nodes should be enlisted here.
    7. There is known issue with Galera cluster. If you omitted to move /var/lib/mysql/grastate.dat and start new cluster with all grastate.dat instances in place - Galera will attempt to determine automatically which node shut down last.
In case it find out that you have started cluster not from the node with the most recent data replica - all the nodes, joining cluster will stop operate to prevent data corruption.
In such case - try to stop all Galera nodes (kill may be required for stalled ones) and try to find out the node with most recent replica again. Saving grastate.dat files is recommended for such case.
Please note - in order to start up serving requests Galera cluster should achieve quorum. By default quorum is one half plus 1 operating nodes relatively to total cluster node number, but it depends on your cluster settings.
As soon as you get operational Galera cluster you may proceed to the next step - assembling RabbitMQ cluster.

Note: for OCF scripts case, p_mysql might have had a broken state after direct crm_resource --force manipulations, thus crm resource cleanup p_mysql should be issued for all cluster nodes.