pause-minority-unexpected-behaviour

Three node cluster
  * ip-10-105-2-176
  * ip-10-105-14-231
  * ip-10-105-9-122

All traffic blocked between node ip-10-105-2-176 and node ip-10-105-14-231.
I did that using iptables on node ip-10-105-2-176 (DROP in INPUT/OUTPUT chains for traffic from/to ip-10-105-14-231).

########################
# RabbitMQ config file #
########################

[
  {rabbit, [
    {queue_master_locator, <<"min-masters">>},
    {cluster_partition_handling, pause_minority}
   ]},
  {kernel, [
    {inet_dist_listen_min, 9100},
    {inet_dist_listen_max, 9110},
    {inet_default_listen_options, [{nodelay,true},{sndbuf,65535},{recbuf,65535}]},
    {inet_default_connect_options, [{nodelay,true}]}
   ]}
].

##############################
# Node ip-10-105-14-231 logs #
##############################

=ERROR REPORT==== 28-Feb-2017::12:20:48 ===
** Node 'rabbit@ip-10-105-2-176' not responding **
** Removing (timedout) connection **

=INFO REPORT==== 28-Feb-2017::12:20:48 ===
rabbit on node 'rabbit@ip-10-105-2-176' down

=INFO REPORT==== 28-Feb-2017::12:20:48 ===
node 'rabbit@ip-10-105-2-176' down: net_tick_timeout

=ERROR REPORT==== 28-Feb-2017::12:20:48 ===
Partial partition detected:
 * We saw DOWN from rabbit@ip-10-105-2-176
 * We can still see rabbit@ip-10-105-9-122 which can see rabbit@ip-10-105-2-176
 * pause_minority mode enabled
We will therefore pause until the *entire* cluster recovers

=WARNING REPORT==== 28-Feb-2017::12:20:48 ===
Cluster minority/secondary status detected - awaiting recovery

=INFO REPORT==== 28-Feb-2017::12:20:48 ===
Stopping RabbitMQ

=INFO REPORT==== 28-Feb-2017::12:20:54 ===
stopped TCP Listener on [::]:5672

=INFO REPORT==== 28-Feb-2017::12:20:55 ===
Stopped RabbitMQ application

########################################
# Node ip-10-105-14-231 cluster status #
########################################

Cluster status of node 'rabbit@ip-10-105-14-231' ...
[{nodes,[{disc,['rabbit@ip-10-105-14-231','rabbit@ip-10-105-2-176',
                'rabbit@ip-10-105-9-122']}]},
 {alarms,[{'rabbit@ip-10-105-9-122',[]}]}]

#############################
# Node ip-10-105-2-176 logs #
#############################

=ERROR REPORT==== 28-Feb-2017::12:20:47 ===
** Node 'rabbit@ip-10-105-14-231' not responding **
** Removing (timedout) connection **

=INFO REPORT==== 28-Feb-2017::12:20:47 ===
rabbit on node 'rabbit@ip-10-105-14-231' down

=INFO REPORT==== 28-Feb-2017::12:20:54 ===
node 'rabbit@ip-10-105-14-231' down: net_tick_timeout

=ERROR REPORT==== 28-Feb-2017::12:20:54 ===
Partial partition detected:
 * We saw DOWN from rabbit@ip-10-105-14-231
 * We can still see rabbit@ip-10-105-9-122 which can see rabbit@ip-10-105-14-231
 * pause_minority mode enabled
We will therefore pause until the *entire* cluster recovers

=WARNING REPORT==== 28-Feb-2017::12:20:54 ===
Cluster minority/secondary status detected - awaiting recovery

=INFO REPORT==== 28-Feb-2017::12:20:54 ===
Stopping RabbitMQ

=INFO REPORT==== 28-Feb-2017::12:20:54 ===
stopped TCP Listener on [::]:5672

=INFO REPORT==== 28-Feb-2017::12:20:54 ===
Stopped RabbitMQ application

#######################################
# Node ip-10-105-2-176 cluster status #
#######################################

Cluster status of node 'rabbit@ip-10-105-2-176' ...
[{nodes,[{disc,['rabbit@ip-10-105-14-231','rabbit@ip-10-105-2-176',
                'rabbit@ip-10-105-9-122']}]},
 {alarms,[{'rabbit@ip-10-105-9-122',[]}]}]

#############################
# Node ip-10-105-9-122 logs #
#############################

=WARNING REPORT==== 28-Feb-2017::12:20:48 ===
Received a 'DOWN' message from 'rabbit@ip-10-105-2-176' but still can communicate with it

=WARNING REPORT==== 28-Feb-2017::12:20:54 ===
Received a 'DOWN' message from 'rabbit@ip-10-105-14-231' but still can communicate with it

#######################################
# Node ip-10-105-9-122 cluster status #
#######################################

Cluster status of node 'rabbit@ip-10-105-9-122' ...
[{nodes,[{disc,['rabbit@ip-10-105-14-231','rabbit@ip-10-105-2-176',
                'rabbit@ip-10-105-9-122']}]},
 {running_nodes,['rabbit@ip-10-105-9-122']},
 {cluster_name,<<"rabbit@ip-10-105-14-231.eu-west-1.compute.internal">>},
 {partitions,[]},
 {alarms,[{'rabbit@ip-10-105-9-122',[]}]}]