network partition testing

tower_testing.md

When all 3 nodes come up, there are no network partitions

vagrant@tower1 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbitmq@tower1 ...
[{nodes,[{disc,[rabbitmq@tower1,rabbitmq@tower2,rabbitmq@tower3]}]},
 {running_nodes,[rabbitmq@tower2,rabbitmq@tower3,rabbitmq@tower1]},
 {cluster_name,<<"rabbitmq@tower2">>},
 {partitions,[]},
 {alarms,[{rabbitmq@tower2,[]},{rabbitmq@tower3,[]},{rabbitmq@tower1,[]}]}]

Simulate network connectivity loss, by shutting down the interface on tower3.

Run ``rabbitmqctl cluster_status" again on tower1. Command hangs for a few minutes then it returns an alarm stating that tower3 is down.

[vagrant@tower1 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbitmq@tower1 ...
[{nodes,[{disc,[rabbitmq@tower1,rabbitmq@tower2,rabbitmq@tower3]}]},
 {running_nodes,[rabbitmq@tower2,rabbitmq@tower3,rabbitmq@tower1]},
 {cluster_name,<<"rabbitmq@tower2">>},
 {partitions,[]},
 {alarms,[{rabbitmq@tower2,[]},
          {rabbitmq@tower3,[nodedown]},
          {rabbitmq@tower1,[]}]}]

Waited for a minute and rerun rabbitmqctl cluster_status on tower1. This time the command only hangs for a second, before returning a cluster list showing that tower3 is not longer listed in running_nodes.

[vagrant@tower1 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbitmq@tower1 ...
[{nodes,[{disc,[rabbitmq@tower1,rabbitmq@tower2,rabbitmq@tower3]}]},
 {running_nodes,[rabbitmq@tower2,rabbitmq@tower1]},
 {cluster_name,<<"rabbitmq@tower2">>},
 {partitions,[]},
 {alarms,[{rabbitmq@tower2,[]},{rabbitmq@tower1,[]}]}]

On tower3 the rabbitmqctl cluster_status command says the following. You have to wait a little while before the command returns a result.

vagrant@tower3 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbitmq@tower3 ...
[{nodes,[{disc,[rabbitmq@tower1,rabbitmq@tower2,rabbitmq@tower3]}]},
 {running_nodes,[rabbitmq@tower3]},
 {cluster_name,<<"rabbitmq@tower2">>},
 {partitions,[]},
 {alarms,[{rabbitmq@tower3,[]}]}]

• simulated network connectivity restoration by enabling the tower3 ethernet interface.

From tower3, I observe a network partition

[vagrant@tower3 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbitmq@tower3 ...
[{nodes,[{disc,[rabbitmq@tower1,rabbitmq@tower2,rabbitmq@tower3]}]},
 {running_nodes,[rabbitmq@tower3]},
 {cluster_name,<<"rabbitmq@tower2">>},
 {partitions,[{rabbitmq@tower3,[rabbitmq@tower1,rabbitmq@tower2]}]},
 {alarms,[{rabbitmq@tower3,[]}]}]

From tower1 I see network partition

[vagrant@tower1 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbitmq@tower1 ...
[{nodes,[{disc,[rabbitmq@tower1,rabbitmq@tower2,rabbitmq@tower3]}]},
 {running_nodes,[rabbitmq@tower2,rabbitmq@tower1]},
 {cluster_name,<<"rabbitmq@tower2">>},
 {partitions,[{rabbitmq@tower2,[rabbitmq@tower3]},
              {rabbitmq@tower1,[rabbitmq@tower3]}]},
 {alarms,[{rabbitmq@tower2,[]},{rabbitmq@tower1,[]}]}]