Skip to content

Instantly share code, notes, and snippets.

@dcorbacho
Created June 14, 2017 12:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dcorbacho/70ddf1ca887668de64e147c8dd39c490 to your computer and use it in GitHub Desktop.
Save dcorbacho/70ddf1ca887668de64e147c8dd39c490 to your computer and use it in GitHub Desktop.
2017-06-14 13:08:29.302 [info] <0.165.0> Configured peer discovery backend: rabbit_peer_discovery_etcd
2017-06-14 13:08:29.302 [info] <0.165.0> Will try to lock with peer discovery backend rabbit_peer_discovery_etcd
2017-06-14 13:08:29.303 [error] <0.164.0> CRASH REPORT Process <0.164.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(httpc_manager, {request,{request,undefined,<0.165.0>,0,http,{"localhost",2379},"/v2/keys/rabbitmq/default/start...",...}}, infinity) in application_master:init/4 line 134
@dcorbacho
Copy link
Author

dcorbacho commented Jun 14, 2017

Two different startup sequences (at random, no changes on the code/startup):

2017-06-14 12:51:13.830 [info] <0.3.0> Log file opened with Lager
2017-06-14 12:51:14.071 [info] <0.31.0> Application asn1 started on node rabbit4@mars
2017-06-14 12:51:14.071 [info] <0.31.0> Application crypto started on node rabbit4@mars
2017-06-14 12:51:14.071 [info] <0.31.0> Application jsx started on node rabbit4@mars
2017-06-14 12:51:14.073 [info] <0.31.0> Application os_mon started on node rabbit4@mars
2017-06-14 12:51:14.073 [info] <0.31.0> Application public_key started on node rabbit4@mars
2017-06-14 12:51:14.084 [info] <0.31.0> Application ssl started on node rabbit4@mars
2017-06-14 12:51:14.127 [info] <0.31.0> Application mnesia started on node rabbit4@mars
2017-06-14 12:51:14.130 [info] <0.31.0> Application ranch started on node rabbit4@mars
2017-06-14 12:51:14.130 [info] <0.31.0> Application ranch_proxy_protocol started on node rabbit4@mars
2017-06-14 12:51:14.130 [info] <0.31.0> Application xmerl started on node rabbit4@mars
2017-06-14 12:51:14.130 [info] <0.31.0> Application rabbit_common started on node rabbit4@mars
2017-06-14 12:51:14.133 [info] <0.165.0>
 Starting RabbitMQ 3.7.0.milestone15+6.g24cfe17.dirty on Erlang 19.0
 Copyright (C) 2007-2017 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/
2017-06-14 12:51:14.135 [info] <0.165.0>
 node           : rabbit4@mars
 home dir       : /Users/dparracorbacho
 config file(s) : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/test.config
 cookie hash    : iCekESKlAouYCMe/hLn89g==
 log(s)         : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/log/rabbit4.log
                : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/log/rabbit4_upgrade.log
 database dir   : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/mnesia/rabbit4
2017-06-14 12:51:15.147 [info] <0.185.0> Memory limit set to 5699 MiB (5976378572 bytes) of 14248 MiB (14940946432 bytes) total.
2017-06-14 12:51:15.199 [info] <0.188.0> Enabling free disk space monitoring
2017-06-14 12:51:15.199 [info] <0.188.0> Disk free limit set to 50MB
2017-06-14 12:51:15.223 [info] <0.195.0> Limiting to approx 4764 file handles (4285 sockets)
2017-06-14 12:51:15.223 [info] <0.196.0> FHC read buffering:  OFF
2017-06-14 12:51:15.223 [info] <0.196.0> FHC write buffering: ON
2017-06-14 12:51:15.224 [info] <0.165.0> Node database directory at /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/mnesia/rabbit4 is empty. Assuming we need to join an
existing cluster or initialise from scratch...
2017-06-14 12:51:15.225 [info] <0.165.0> Configured peer discovery backend: rabbit_peer_discovery_etcd
2017-06-14 12:51:15.225 [info] <0.165.0> Will try to lock with peer discovery backend rabbit_peer_discovery_etcd
2017-06-14 12:51:15.225 [error] <0.164.0> CRASH REPORT Process <0.164.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(httpc_manager, {request,{request,undefined,<0.165.0>,0,http,{"localhost",2379},"/v2/keys/rabbitmq/default/start...",...}}, infinity) in application_master:init/4 line 134
2017-06-14 12:51:15.225 [info] <0.31.0> Application rabbit exited with reason: no such process or port in call to gen_server:call(httpc_manager, {request,{request,undefined,<0.165.0>,0,http,{"localhost",2379},"/v2/keys/rabbitmq/default/start...",...}}, infinity)
2017-06-14 12:50:40.963 [info] <0.31.0> Application lager started on node rabbit4@mars
2017-06-14 12:50:40.964 [info] <0.3.0> Log file opened with Lager
2017-06-14 12:50:41.202 [info] <0.31.0> Application asn1 started on node rabbit4@mars
2017-06-14 12:50:41.236 [info] <0.31.0> Application inets started on node rabbit4@mars
2017-06-14 12:50:41.236 [info] <0.31.0> Application crypto started on node rabbit4@mars
2017-06-14 12:50:41.236 [info] <0.31.0> Application jsx started on node rabbit4@mars
2017-06-14 12:50:41.239 [info] <0.31.0> Application os_mon started on node rabbit4@mars
2017-06-14 12:50:41.239 [info] <0.31.0> Application public_key started on node rabbit4@mars
2017-06-14 12:50:41.252 [info] <0.31.0> Application ssl started on node rabbit4@mars
2017-06-14 12:50:41.312 [info] <0.31.0> Application mnesia started on node rabbit4@mars
2017-06-14 12:50:41.316 [info] <0.31.0> Application ranch started on node rabbit4@mars
2017-06-14 12:50:41.316 [info] <0.31.0> Application ranch_proxy_protocol started on node rabbit4@mars
2017-06-14 12:50:41.316 [info] <0.31.0> Application xmerl started on node rabbit4@mars
2017-06-14 12:50:41.316 [info] <0.31.0> Application rabbit_common started on node rabbit4@mars
2017-06-14 12:50:41.320 [info] <0.172.0>
 Starting RabbitMQ 3.7.0.milestone15+6.g24cfe17.dirty on Erlang 19.0
 Copyright (C) 2007-2017 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/
2017-06-14 12:50:41.322 [info] <0.172.0>
 node           : rabbit4@mars
 home dir       : /Users/dparracorbacho
 config file(s) : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/test.config
 cookie hash    : iCekESKlAouYCMe/hLn89g==
 log(s)         : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/log/rabbit4.log
                : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/log/rabbit4_upgrade.log
 database dir   : /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/mnesia/rabbit4
2017-06-14 12:50:42.336 [info] <0.180.0> Memory limit set to 5702 MiB (5979365376 bytes) of 14255 MiB (14948413440 bytes) total.
2017-06-14 12:50:42.383 [info] <0.183.0> Enabling free disk space monitoring
2017-06-14 12:50:42.384 [info] <0.183.0> Disk free limit set to 50MB
2017-06-14 12:50:42.406 [info] <0.185.0> Limiting to approx 4764 file handles (4285 sockets)
2017-06-14 12:50:42.407 [info] <0.186.0> FHC read buffering:  OFF
2017-06-14 12:50:42.407 [info] <0.186.0> FHC write buffering: ON
2017-06-14 12:50:42.408 [info] <0.172.0> Node database directory at /var/folders/34/x13f96mn2m7d6z683pkb8bhm0000gq/T/rabbitmq-test-instances/rabbit4/mnesia/rabbit4 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2017-06-14 12:50:42.408 [info] <0.172.0> Configured peer discovery backend: rabbit_peer_discovery_etcd
2017-06-14 12:50:42.408 [info] <0.172.0> Will try to lock with peer discovery backend rabbit_peer_discovery_etcd
2017-06-14 12:50:42.426 [info] <0.172.0> All discovered existing cluster peers: rabbit2@mars, rabbit3@mars
2017-06-14 12:50:42.426 [info] <0.172.0> Peer nodes we can cluster with: rabbit2@mars, rabbit3@mars

@dcorbacho
Copy link
Author

Cluster status of node rabbit6@mars ...
Error:
badarg
 Stacktrace [{erlang,'++',
                 [{badrpc,
                      {'EXIT',
                          {aborted,
                              {no_exists,
                                  [rabbit_runtime_parameters,cluster_name]}}}},
                  [{alarms,[]}]],
                 []},
             {'Elixir.RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand',run,2,
                 [{file,
                      "lib/rabbitmq/cli/ctl/commands/cluster_status_command.ex"},
                  {line,35}]},
             {'Elixir.RabbitMQCtl',maybe_run_command,3,
                 [{file,"lib/rabbitmqctl.ex"},{line,171}]},
             {'Elixir.RabbitMQCtl','-exec_command/2-fun-0-',5,
                 [{file,"lib/rabbitmqctl.ex"},{line,70}]},
             {'Elixir.RabbitMQCtl',main,1,
                 [{file,"lib/rabbitmqctl.ex"},{line,43}]},
             {'Elixir.Kernel.CLI','-exec_fun/2-fun-0-',3,
                 [{file,"lib/kernel/cli.ex"},{line,76}]}]

@dcorbacho
Copy link
Author

rabbit@neptuno doesn't exist

mars:scripts dparracorbacho$ ./rabbitmqctl join_cluster rabbit@neptuno
Clustering node rabbit@mars with rabbit@neptuno
Error: unable to connect to node 'rabbit@mars': nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@mars]

rabbit@mars:
  * connected to epmd (port 4369) on mars
  * node rabbit@mars up, 'rabbit' application not running
  * running applications on rabbit@mars: [lager,ranch_proxy_protocol,ranch,
                                          ssl,public_key,crypto,inets,xmerl,
                                          jsx,asn1,goldrush,compiler,
                                          syntax_tools,sasl,stdlib,kernel]
  * suggestion: start_app on rabbit@mars

current node details:
- node name: rabbitmqcli49@mars
- home dir: /Users/dparracorbacho
- cookie hash: iCekESKlAouYCMe/hLn89g==

@dcorbacho
Copy link
Author

Listing users ...
Error:
{:badrpc, {:EXIT, {:aborted, {:no_exists, [:rabbit_user, {:internal_user, :_, :_, :_, :_}]}}}}
Listing bindings for vhost /...
Error:
{:badrpc, {:EXIT, {:aborted, {:no_exists, [:rabbit_route, {:route, {:binding, {:resource, "/", :_, :_}, :_, {:resource, "/", :_, :_}, :_}, :_}]}}}}
Listing channels ...

@dcorbacho
Copy link
Author

mars:test dparracorbacho$ sh perform_many_ops.sh rabbit2 testuser
Cluster status of node rabbit2@mars ...
Error:
#{'__exception__' => true,
  '__struct__' => 'Elixir.Protocol.UndefinedError',
  description => <<>>,
  protocol => 'Elixir.Enumerable',
  value => {error,{corrupt_or_missing_cluster_files,{error,enoent},{error,enoent}}}}
 Stacktrace [{'Elixir.Enumerable','impl_for!',1,
                 [{file,"lib/enum.ex"},{line,1}]},
             {'Elixir.Enumerable',reduce,3,[{file,"lib/enum.ex"},{line,116}]},
             {'Elixir.Enum',map,2,[{file,"lib/enum.ex"},{line,1776}]},
             {'Elixir.RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand',run,2,
                 [{file,
                      "lib/rabbitmq/cli/ctl/commands/cluster_status_command.ex"},
                  {line,34}]},
             {'Elixir.RabbitMQCtl',maybe_run_command,3,
                 [{file,"lib/rabbitmqctl.ex"},{line,171}]},
             {'Elixir.RabbitMQCtl','-exec_command/2-fun-0-',5,
                 [{file,"lib/rabbitmqctl.ex"},{line,70}]},
             {'Elixir.RabbitMQCtl',main,1,
                 [{file,"lib/rabbitmqctl.ex"},{line,43}]},
             {'Elixir.Kernel.CLI','-exec_fun/2-fun-0-',3,
                 [{file,"lib/kernel/cli.ex"},{line,76}]}]

@dcorbacho
Copy link
Author

dcorbacho commented Jun 16, 2017

list_connections and list_queues hang forever when the node is restarted

@dcorbacho
Copy link
Author

Listing runtime parameters for vhost "/" ...
Error:
{:badrpc, {:EXIT, {:aborted, {:no_exists, [:rabbit_vhost, "/"]}}}}
Clearing runtime parameter "name" for component "federation-upstream" on vhost "/" ...
Error:
{:badrpc, {:EXIT, {:aborted, {:no_exists, [:rabbit_runtime_parameters, {"/", "federation-upstream", "name"}]}}}}

@dcorbacho
Copy link
Author

dcorbacho commented Jun 16, 2017

Checking health of node rabbit2@mars ...
Error:
{:badrpc, {:EXIT, {:aborted, {:no_exists, :rabbit_vhost}}}}
Adding user "testuser" ...
Error:
{:badrpc, {:EXIT, {:noproc, {:gen_server2, :call, [:worker_pool, {:next_free, #PID<8233.188.0>}, :infinity]}}}}
Changing password for user "testuser" ...
Error:
{:no_such_user, "testuser"}
Setting runtime parameter "federation-upstream" for component "name" to "{"uri" : "amqp://"}" ...
Error:
{:badrpc, {:EXIT, {:badarg, [{:ets, :lookup, [:rabbit_registry, {:runtime_parameter, :"federation-upstream"}], []}, {:rabbit_registry, :lookup_module, 2, [file: 'src/rabbit_registry.erl', line: 64]}, {:ra
bbit_runtime_parameters, :lookup_component, 1, [file: 'src/rabbit_runtime_parameters.erl', line: 405]}, {:rabbit_runtime_parameters, :set_any0, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 149]},
{:rabbit_runtime_parameters, :set_any, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 143]}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 187]}]}}}
Listing runtime parameters for vhost "/" ...

Clearing runtime parameter "name" for component "federation-upstream" on vhost "/" ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment