Skip to content

Instantly share code, notes, and snippets.

@cap10morgan

cap10morgan/gist:8cddfde9eca7e2ed657e Secret

Created April 14, 2015 15:39
Show Gist options
  • Save cap10morgan/8cddfde9eca7e2ed657e to your computer and use it in GitHub Desktop.
Save cap10morgan/8cddfde9eca7e2ed657e to your computer and use it in GitHub Desktop.
Download ZIP
rabbitmq-autocluster-consul re-join cluster failure
Raw
gistfile1.txt
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Copyright (C) 2007-2014 GoPivotal, Inc.
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Licensed under the MPL. See http://www.rabbitmq.com/
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:04 ===
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: node : rabbit@ip-10-36-116-78
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: home dir : /root
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: config file(s) : /etc/rabbitmq/rabbitmq.config
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: cookie hash : Rb12DTFU4RRfqgMYl0J6LQ==
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: log : tty
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: sasl log : /var/log/rabbitmq/rabbit@ip-10-36-116-78-sasl.log
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-36-116-78
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Consul Cluster Initializing: Cluster name: <<"rabbit@rabbitmq">>
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Query Body: "{\"Name\":\"rabbitmq:autocluster\", \"Tags\": [\"rabbit@rabbitmq\"]}"
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Cluster name: <<"rabbit@rabbitmq">>
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Node registered
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Cluster name: <<"rabbit@rabbitmq">>
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Query URL: "http://10.36.116.78:8500/v1/catalog/service/rabbitmq:autocluster?tag=rabbit@rabbitmq"
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Cluster name: <<"rabbit@rabbitmq">>
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Joining existing cluster.
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:04 ===
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: application: mnesia
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:04 ===
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: Resetting Rabbit
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:04 ===
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: application: mnesia
Apr 14 15:33:04 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: mnesia
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Already member of cluster: ['rabbit@ip-10-36-136-99','rabbit@ip-10-36-126-31',
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: 'rabbit@ip-10-36-116-78']
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Limiting to approx 1048476 file handles (943626 sockets)
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: mnesia
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Priority queues enabled, real BQ is rabbit_variable_queue
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Memory limit set to 6017MB of 15043MB total.
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Disk free limit set to 50MB
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: msg_store_transient: using rabbit_msg_store_ets_index to provide index
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: msg_store_persistent: using rabbit_msg_store_ets_index to provide index
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =WARNING REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: msg_store_persistent: rebuilding indices from scratch
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Adding vhost '/'
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Creating user 'guest'
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Setting user tags for user 'guest' to [administrator]
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: started TCP Listener on [::]:5672
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Management plugin started. Port: 15672
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: rabbitmq_management
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: {{shutdown,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {failed_to_start_child,rabbit_mgmt_sup,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {'EXIT',
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {{shutdown,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [{{already_started,<22806.25166.0>},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {child,undefined,rabbit_mgmt_db,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {rabbit_mgmt_db,start_link,[]},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: permanent,4294967295,worker,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [rabbit_mgmt_db]}}]},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {gen_server2,call,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [<0.530.0>,{init,<0.528.0>},infinity]}}}}},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {rabbit_mgmt_app,start,[normal,[]]}}
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: rabbitmq_web_dispatch
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: webmachine
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: mochiweb
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: rabbitmq_autocluster_consul
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: rabbitmq_management_agent
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: stopped TCP Listener on [::]:5672
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: rabbit
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: os_mon
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: inets
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: mnesia
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: BOOT FAILED
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: ===========
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Error description:
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {could_not_start,rabbitmq_management,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {{shutdown,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {failed_to_start_child,rabbit_mgmt_sup,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {'EXIT',
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {{shutdown,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [{{already_started,<22806.25166.0>},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {child,undefined,rabbit_mgmt_db,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {rabbit_mgmt_db,start_link,[]},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: permanent,4294967295,worker,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [rabbit_mgmt_db]}}]},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {gen_server2,call,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [<0.530.0>,{init,<0.528.0>},infinity]}}}}},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {rabbit_mgmt_app,start,[normal,[]]}}}
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Log files (may contain more information):
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: tty
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: /var/log/rabbitmq/rabbit@ip-10-36-116-78-sasl.log
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: amqp_client
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: application: xmerl
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: exited: stopped
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: type: temporary
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: =INFO REPORT==== 14-Apr-2015::15:33:05 ===
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Error description:
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {could_not_start,rabbitmq_management,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {{shutdown,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {failed_to_start_child,rabbit_mgmt_sup,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {'EXIT',
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {{shutdown,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [{{already_started,<22806.25166.0>},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {child,undefined,rabbit_mgmt_db,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {rabbit_mgmt_db,start_link,[]},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: permanent,4294967295,worker,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [rabbit_mgmt_db]}}]},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {gen_server2,call,
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: [<0.530.0>,{init,<0.528.0>},infinity]}}}}},
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: {rabbit_mgmt_app,start,[normal,[]]}}}
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: Log files (may contain more information):
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: tty
Apr 14 15:33:05 ip-10-36-116-78 bash[30962]: /var/log/rabbitmq/rabbit@ip-10-36-116-78-sasl.log
Apr 14 15:33:06 ip-10-36-116-78 bash[30962]: {"init terminating in do_boot",{could_not_start,rabbitmq_management,{{shutdown,{failed_to_start_child,rabbit_mgmt_sup,{'EXIT',{{shutdown,[{{already_started,<22806.25166.0>},{child,undefined,rabbit_mgmt_db,{rabbit_mgmt_db,start_link,[]},permanent,4294967295,worker,[rabbit_mgmt_db]}}]},{gen_server2,call,[<0.530.0>,{init,<0.528.0>},infinity]}}}}},{rabbit_mgmt_app,start,[normal,[]]}}}}
Apr 14 15:33:07 ip-10-36-116-78 bash[30962]: Crash dump was written to: erl_crash.dump
Apr 14 15:33:07 ip-10-36-116-78 bash[30962]: init terminating in do_boot ()
Apr 14 15:33:07 ip-10-36-116-78 systemd[1]: rabbitmq@3.service: main process exited, code=exited, status=1/FAILURE
Apr 14 15:33:07 ip-10-36-116-78 docker[31150]: rabbitmq
Apr 14 15:33:08 ip-10-36-116-78 systemd[1]: Unit rabbitmq@3.service entered failed state.
Apr 14 15:33:08 ip-10-36-116-78 systemd[1]: rabbitmq@3.service failed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment