Logstash + RabbitMQ HA attempt

Steps.md

1. Configure RabbitMQ server clustering per https://www.rabbitmq.com/clustering.html.
2. Configure at least one logstash agent to output to RabbitMQ. See config example:

output {
  rabbitmq {
    vhost => 'logstash'
    exchange => 'my_exchange'
    exchange_type => 'direct'
    host => 'logstashmq.example.com'
    key => 'my_key'
    user => 'logstash'
    password => 'super_secret'
    persistent => true
    port => 5671
    ssl => true
    verify_ssl => true
  }
}

3. Configure a different logstash agent with a RabbitMQ input. This agent will make 1 connection to each server node. Ultimately, cross-connected logstash agents on the input side is ideal because any single RabbitMQ node can fail at the same time as any single Logstash agent and messages would continue to flow. See example configuration:

input {
  rabbitmq {
    arguments => { "x-ha-policy" => "all" }
    user => 'logstash'
    password => 'super_secret'
    durable => true
    exchange => 'my_exchange'
    host => 'logstashmq1.example.com'
    key => 'my_key'
    port => 5671
    ssl => true
    queue => 'logstash'
    verify_ssl => true
    vhost => 'logstash'
  }
  
  rabbitmq {
    arguments => { "x-ha-policy" => "all" }
    user => 'logstash'
    password => 'super_secret'
    durable => true
    exchange => 'my_exchange'
    host => 'logstashmq2.example.com'
    key => 'my_key'
    port => 5671
    ssl => true
    queue => 'logstash'
    verify_ssl => true
    vhost => 'logstash'
  }  
}

4. Start up both of the logstash agents (the 1 with inputs and 1 with output).
5. In the RabbitMQ dashboard in the "Connections" tab you should see 2 connections from your Logstash agent - 1 to each server node.
6. In the "Queues" tab, click on the "logstash" queue. In the "Consumers" section of this page you should see 2 consumers.
7. In the "Details" section of the page note which RabbitMQ server is listed as the "Node" which is the primary for this queue. (3rd column over you will see Virtual Host, Node, and Mirrors - The one you want is the "Node" server.
8. Stop RabbitMQ on the "Node" server. Reload the dashboard (or pull up the dashboard for the mirrored server node). Back in the Queues -> logstash -> Consumers section notice that no consumers are listed.
9. On the "Connections" tab, notice that the logstash agent is still connected to the new primary node. The connection is still alive to the server, but the consumer is not active on the queue. I believe this is because RabbitMQ "cancelled" all consumers. Consumers are then responsible for handling the cancellation and calling "consume" on the queue again. At this point I would presume that all replicated/queued messages will be delivered to the consumer.

how is the output HA here though?

You need HAProxy ....