Skip to content

Instantly share code, notes, and snippets.

@carlhoerberg

carlhoerberg/rabbit@tiger01-sasl.log Secret

Last active December 18, 2015 02:48
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save carlhoerberg/ff6c6bd4f7639bf4b2f5 to your computer and use it in GitHub Desktop.
Save carlhoerberg/ff6c6bd4f7639bf4b2f5 to your computer and use it in GitHub Desktop.
Download ZIP
RabbitMQ crash log
Raw
rabbit@tiger01-sasl.log
=CRASH REPORT==== 5-Jun-2013::10:12:14 ===
crasher:
initial call: gen:init_it/6
pid: <0.4232.622>
registered_name: []
exception exit: {{badmatch,true},
[{rabbit_queue_index,init,2},
{rabbit_variable_queue,init,5},
{rabbit_mirror_queue_slave,init,1},
{gen_server2,init_it,6},
{proc_lib,init_p_do_apply,3}]}
in function gen_server2:init_it/6
ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.184.0>]
messages: []
links: [<0.1118.0>,<0.4233.622>]
dictionary: []
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 1088
neighbours:
neighbour: [{pid,<0.4233.622>},
{registered_name,[]},
{initial_call,{gen,init_it,
['Argument__1','Argument__2',
'Argument__3','Argument__4',
'Argument__5','Argument__6']}},
{current_function,{gen_server2,process_next_msg,1}},
{ancestors,[<0.4232.622>,rabbit_mirror_queue_slave_sup,
rabbit_sup,<0.184.0>]},
{messages,[]},
{links,[<0.4232.622>]},
{dictionary,[{random_seed,{22387,2880,18489}}]},
{trap_exit,false},
{status,waiting},
{heap_size,28657},
{stack_size,7},
{reductions,5360}]
=CRASH REPORT==== 5-Jun-2013::10:12:14 ===
crasher:
initial call: gen:init_it/6
pid: <0.4243.622>
registered_name: []
exception exit: {{badmatch,true},
[{rabbit_queue_index,init,2},
{rabbit_variable_queue,init,5},
{rabbit_mirror_queue_slave,init,1},
{gen_server2,init_it,6},
{proc_lib,init_p_do_apply,3}]}
in function gen_server2:init_it/6
ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.184.0>]
messages: []
links: [<0.1118.0>,<0.4244.622>]
dictionary: []
trap_exit: true
status: running
heap_size: 233
stack_size: 24
reductions: 1104
neighbours:
neighbour: [{pid,<0.4244.622>},
{registered_name,[]},
{initial_call,{gen,init_it,
['Argument__1','Argument__2',
'Argument__3','Argument__4',
'Argument__5','Argument__6']}},
{current_function,{gen_server2,process_next_msg,1}},
{ancestors,[<0.4243.622>,rabbit_mirror_queue_slave_sup,
rabbit_sup,<0.184.0>]},
{messages,[]},
{links,[<0.4243.622>]},
{dictionary,[{random_seed,{22387,2880,19185}}]},
{trap_exit,false},
{status,waiting},
{heap_size,987},
{stack_size,7},
{reductions,844}]
=CRASH REPORT==== 5-Jun-2013::10:13:26 ===
crasher:
initial call: gen:init_it/6
pid: <0.1698.292>
registered_name: []
exception exit: {bad_return_value,
{error,
{{badmatch,
[{<0.1699.292>,<0.1698.292>},
{<6868.7396.477>,<6868.7395.477>}]},
[{rabbit_mirror_queue_misc,
'-remove_from_queue/3-fun-1-',3},
{mnesia_tm,apply_fun,3},
{mnesia_tm,execute_transaction,5},
{rabbit_misc,
'-execute_mnesia_transaction/1-fun-0-',1},
{worker_pool_worker,handle_call,3},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}}}
in function gen_server2:terminate/3
ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.184.0>]
messages: []
links: [<0.1118.0>,<0.1699.292>]
dictionary: [{{credit_to,<6867.4505.631>},49},
{{xtype_to_module,fanout},rabbit_exchange_type_fanout},
{fhc_age_tree,{0,nil}},
{{credit_from,<0.260.0>},1963},
{credit_blocked,[]},
{{xtype_to_module,direct},rabbit_exchange_type_direct},
{{credit_to,<6867.21551.801>},49},
{{credit_to,<6867.20991.730>},49},
{guid,{{977707758,2293788077,3108534888,3378518352},1}}]
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 520494
neighbours:
neighbour: [{pid,<0.1699.292>},
{registered_name,[]},
{initial_call,{gen,init_it,
['Argument__1','Argument__2',
'Argument__3','Argument__4',
'Argument__5','Argument__6']}},
{current_function,{gen,do_call,4}},
{ancestors,[<0.1698.292>,rabbit_mirror_queue_slave_sup,
rabbit_sup,<0.184.0>]},
{messages,[{'$gen_cast',leave}]},
{links,[<0.1698.292>]},
{dictionary,[{random_seed,{21874,13233,26903}}]},
{trap_exit,false},
{status,waiting},
{heap_size,610},
{stack_size,28},
{reductions,55759}]
=CRASH REPORT==== 5-Jun-2013::10:14:45 ===
crasher:
initial call: gen:init_it/6
pid: <0.27491.545>
registered_name: []
exception exit: {badarith,[{rabbit_mgmt_db,rate,5},
{rabbit_mgmt_db,'-rates/5-lc$^0/1-0-',5},
{rabbit_mgmt_db,rates,5},
{rabbit_mgmt_db,handle_stats,6},
{rabbit_mgmt_db,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}
in function gen_server2:terminate/3
ancestors: [<0.3247.0>,rabbit_mgmt_sup,<0.1852.0>]
messages: []
links: [<0.3247.0>]
dictionary: [{delegate,delegate_14}]
trap_exit: false
status: running
heap_size: 317811
stack_size: 24
reductions: 749752413799
neighbours:
=CRASH REPORT==== 5-Jun-2013::10:14:45 ===
crasher:
initial call: gen:init_it/6
pid: <0.27491.545>
registered_name: []
exception exit: {badarith,[{rabbit_mgmt_db,rate,5},
{rabbit_mgmt_db,'-rates/5-lc$^0/1-0-',5},
{rabbit_mgmt_db,rates,5},
{rabbit_mgmt_db,handle_stats,6},
{rabbit_mgmt_db,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}
in function gen_server2:terminate/3
ancestors: [<0.3247.0>,rabbit_mgmt_sup,<0.1852.0>]
messages: []
links: [<0.3247.0>]
dictionary: [{delegate,delegate_14}]
trap_exit: false
status: running
heap_size: 317811
stack_size: 24
reductions: 749752413799
neighbours:
=SUPERVISOR REPORT==== 5-Jun-2013::10:14:45 ===
Supervisor: {<0.3247.0>,mirrored_supervisor}
Context: child_terminated
Reason: {badarith,[{rabbit_mgmt_db,rate,5},
{rabbit_mgmt_db,'-rates/5-lc$^0/1-0-',5},
{rabbit_mgmt_db,rates,5},
{rabbit_mgmt_db,handle_stats,6},
{rabbit_mgmt_db,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}
Offender: [{pid,<0.27491.545>},
{name,rabbit_mgmt_db},
{mfa,{rabbit_mgmt_db,start_link,[]}},
{restart_type,permanent},
{shutdown,4294967295},
{child_type,worker}]
=CRASH REPORT==== 5-Jun-2013::10:14:57 ===
crasher:
initial call: gen:init_it/6
pid: <0.1605.493>
registered_name: []
exception exit: {bad_return_value,
{error,
{{badmatch,
[{<0.1606.493>,<0.1605.493>},
{<6868.14464.1250>,<6868.14463.1250>}]},
[{rabbit_mirror_queue_misc,
'-remove_from_queue/3-fun-1-',3},
{mnesia_tm,apply_fun,3},
{mnesia_tm,execute_transaction,5},
{rabbit_misc,
'-execute_mnesia_transaction/1-fun-0-',1},
{worker_pool_worker,handle_call,3},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}}}
in function gen_server2:terminate/3
ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.184.0>]
messages: []
links: [<0.1118.0>,<0.1606.493>]
dictionary: [{{credit_from,<0.257.0>},1545},
{credit_blocked,[]},
{{xtype_to_module,direct},rabbit_exchange_type_direct},
{fhc_age_tree,{0,nil}},
{guid,{{938233082,3676877726,2686204268,2362756830},1}}]
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 1261029
neighbours:
neighbour: [{pid,<0.1606.493>},
{registered_name,[]},
{initial_call,{gen,init_it,
['Argument__1','Argument__2',
'Argument__3','Argument__4',
'Argument__5','Argument__6']}},
{current_function,{gen,do_call,4}},
{ancestors,[<0.1605.493>,rabbit_mirror_queue_slave_sup,
rabbit_sup,<0.184.0>]},
{messages,[{'$gen_cast',leave}]},
{links,[<0.1605.493>]},
{dictionary,[{random_seed,{22216,28590,17891}}]},
{trap_exit,false},
{status,waiting},
{heap_size,377},
{stack_size,28},
{reductions,334814}]
Raw
rabbit@tiger01.log
=ERROR REPORT==== 5-Jun-2013::08:43:13 ===
** Node rabbit@tiger02 not responding **
** Removing (timedout) connection **
=INFO REPORT==== 5-Jun-2013::08:43:27 ===
rabbit on node rabbit@tiger02 down
=INFO REPORT==== 5-Jun-2013::09:00:24 ===
Mirrored-queue (queue '***' in vhost '***'): Master <rabbit@tiger01.3.744.0> saw deaths of mirrors <rabbit@tiger02.3.3057.0>
... a lot of them ...
=ERROR REPORT==== 5-Jun-2013::10:13:26 ===
** Generic server <0.1698.292> terminating
** Last message in was {gm_deaths,[<6867.8088.485>]}
** When Server state == {state,
{amqqueue,
{resource,<<"***">>,queue,
<<"outbox.event.mail.reviewbootstrap.audit">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"outbox.poison">>}],
<6867.8083.485>,
[<6868.7395.477>],
[],
[{vhost,<<"***">>},
{name,<<"HA">>},
{pattern,<<".*">>},
{definition,[{<<"ha-mode">>,<<"all">>}]},
{priority,0}],
[{<6868.7396.477>,<6868.7395.477>},
{<6867.8088.485>,<6867.8083.485>}]},
<0.1699.292>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
37,
{0,nil},
undefined,
{0,nil},
{qistate,
"/var/lib/rabbitmq/mnesia/rabbit@tiger01/queues/3DLZ3FZKPMRPTI3Y7GNGA52EK",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
#Ref<0.0.341.154856>,0,262144,
#Fun<rabbit_variable_queue.2.41635915>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<104,190,144,130,243,38,187,190,65,216,116,253,
112,40,99,56>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,245835,
"/var/lib/rabbitmq/mnesia/rabbit@tiger01/msg_store_persistent"},
rabbit_msg_store_ets_index,
"/var/lib/rabbitmq/mnesia/rabbit@tiger01/msg_store_persistent",
<0.263.0>,249932,241738,254029,258126},
{client_msstate,msg_store_transient,
<<58,70,162,238,136,184,109,173,185,72,122,104,201,
96,25,80>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,225350,
"/var/lib/rabbitmq/mnesia/rabbit@tiger01/msg_store_transient"},
rabbit_msg_store_ets_index,
"/var/lib/rabbitmq/mnesia/rabbit@tiger01/msg_store_transient",
<0.258.0>,229447,221246,233544,237641}},
true,0,#Fun<rabbit_mirror_queue_slave.1.5393384>,0,
0,0,0,0,0,0,0,
{rates,
{{1370,421806,891086},0},
{{1370,421806,891086},0},
0.0,0.0,
{1370,421811,939005}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0,
{rates,
{{1370,421806,891086},0},
{{1370,421806,891086},0},
0.0,0.0,
{1370,421811,939005}}},
undefined,undefined,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
0}
** Reason for termination ==
** {bad_return_value,
{error,
{{badmatch,
[{<0.1699.292>,<0.1698.292>},
{<6868.7396.477>,<6868.7395.477>}]},
[{rabbit_mirror_queue_misc,'-remove_from_queue/3-fun-1-',3},
{mnesia_tm,apply_fun,3},
{mnesia_tm,execute_transaction,5},
{rabbit_misc,'-execute_mnesia_transaction/1-fun-0-',1},
{worker_pool_worker,handle_call,3},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}}}
=ERROR REPORT==== 5-Jun-2013::10:13:36 ===
error on AMQP connection <0.17077.622>: {ssl_upgrade_error,closed} (unknown POSIX error)
Raw
rabbit@tiger02-sasl.log
=CRASH REPORT==== 5-Jun-2013::09:04:27 ===
crasher:
initial call: application_master:init/4
pid: <0.1563.0>
registered_name: []
exception exit: {bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_vhost,rabbit_listener,
rabbit_durable_route,rabbit_semi_durable_route,
rabbit_route,rabbit_reverse_route,
rabbit_topic_trie_node,rabbit_topic_trie_edge,
rabbit_topic_trie_binding,rabbit_exchange,
rabbit_runtime_parameters,rabbit_queue]}}}}}}
in function application_master:init/4
ancestors: [<0.1562.0>]
messages: [{'EXIT',<0.1564.0>,normal}]
links: [<0.1562.0>,<0.7.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 246
neighbours:
Raw
rabbit@tiger02.log
=INFO REPORT==== 5-Jun-2013::09:03:53 ===
Starting RabbitMQ 3.0.4 on Erlang R14B04
=INFO REPORT==== 5-Jun-2013::09:03:54 ===
Limiting to approx 32668 file handles (29399 sockets)
=INFO REPORT==== 5-Jun-2013::09:04:26 ===
Timeout contacting cluster nodes: [rabbit@tiger01,rabbit@tiger03].
DIAGNOSTICS
===========
nodes in question: [rabbit@tiger01,rabbit@tiger03]
hosts, their running nodes and ports:
- tiger01: [{rabbit,41052}]
- tiger03: [{rabbit,55783}]
current node details:
- node name: rabbit@tiger02
- home dir: /var/lib/rabbitmq
- cookie hash: ***
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: rabbit
exited: {bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_vhost,rabbit_listener,
rabbit_durable_route,
rabbit_semi_durable_route,rabbit_route,
rabbit_reverse_route,rabbit_topic_trie_node,
rabbit_topic_trie_edge,
rabbit_topic_trie_binding,rabbit_exchange,
rabbit_runtime_parameters,
rabbit_queue]}}}}}}
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: os_mon
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: rabbitmq_web_dispatch
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: webmachine
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: mochiweb
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: xmerl
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: inets
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: mnesia
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
application: amqp_client
exited: stopped
type: temporary
=INFO REPORT==== 5-Jun-2013::09:04:27 ===
Error description:
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_vhost,rabbit_listener,
rabbit_durable_route,rabbit_semi_durable_route,
rabbit_route,rabbit_reverse_route,
rabbit_topic_trie_node,rabbit_topic_trie_edge,
rabbit_topic_trie_binding,rabbit_exchange,
rabbit_runtime_parameters,rabbit_queue]}}}}}}}
Log files (may contain more information):
/var/log/rabbitmq/rabbit@tiger02.log
/var/log/rabbitmq/rabbit@tiger02-sasl.log
Raw
startup_log
** Found 0 name clashes in code paths
+---+ +---+
| | | |
| | | |
| | | |
| +---+ +-------+
| |
| RabbitMQ +---+ |
| | | |
| v3.0.4 +---+ |
| |
+-------------------+
AMQP 0-9-1 / 0-9 / 0-8
Copyright (C) 2007-2013 VMware, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
node : rabbit@tiger02
app descriptor : /usr/lib/rabbitmq/lib/rabbitmq_server-3.0.4/sbin/../ebin/rabbit.app
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : ***
log : /var/log/rabbitmq/rabbit@tiger02.log
sasl log : /var/log/rabbitmq/rabbit@tiger02-sasl.log
database dir : /var/lib/rabbitmq/mnesia/rabbit@tiger02
erlang version : 5.8.5
-- rabbit boot start
starting file handle cache server ...done
starting worker pool ...done
starting database ...
BOOT FAILED
===========
Timeout contacting cluster nodes: [rabbit@tiger01,rabbit@tiger03].
DIAGNOSTICS
===========
nodes in question: [rabbit@tiger01,rabbit@tiger03]
hosts, their running nodes and ports:
- tiger01: [{rabbit,41052}]
- tiger03: [{rabbit,55783}]
current node details:
- node name: rabbit@tiger02
- home dir: /var/lib/rabbitmq
- cookie hash: ***
BOOT FAILED
===========
Error description:
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_vhost,rabbit_listener,
rabbit_durable_route,rabbit_semi_durable_route,
rabbit_route,rabbit_reverse_route,
rabbit_topic_trie_node,rabbit_topic_trie_edge,
rabbit_topic_trie_binding,rabbit_exchange,
rabbit_runtime_parameters,rabbit_queue]}}}}}}}
Log files (may contain more information):
/var/log/rabbitmq/rabbit@tiger02.log
/var/log/rabbitmq/rabbit@tiger02-sasl.log
{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{timeout_waiting_for_tables,[rabbit_user,rabbit_vhost,rabbit_listener,rabbit_durable_route,rabbit_semi_durable_route,rabbit_route,rabbit_reverse_route,rabbit_topic_trie_node,rabbit_topic_trie_edge,rabbit_topic_trie_binding,rabbit_exchange,rabbit_runtime_parameters,rabbit_queue]}}}}}}}}}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment