Created
May 30, 2013 17:49
-
-
Save jhhayden/5679708 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Intro: | |
I have 2 Rabbits running on 2 different ec2 instances in AWS | |
Rabbit1 has had a Shovel plugin installed in order to send messages to Rabbit2 | |
These Rabbits are being used for a Sensu monitoring network | |
The Sensu server is in the same region as Rabbit1 | |
When a message is sent by Sensu to Rabbit1 it fansout to all connected queues (maybe the wrong terminology here), including one called us-west-2 | |
The Shovel is supposed to pick up messages from us-west-2 and forward it to Rabbit2 | |
Later on I will add other queues to forward to other Rabbits | |
So, here we go | |
First, here is my minimal rabbit.config to configure the shovel in Rabbit1 | |
[ | |
{rabbitmq_shovel, | |
[{shovels, [ {test_shovel, | |
[ {sources, [{broker, "amqp://admin:mdnakar1@54.243.157.166:5672/sensu"}]} | |
, {destinations, [{broker, "amqp://admin:mdnakar1@54.245.126.50:5672/sensu"}]} | |
, {queue, <<"rabbitmq-us-west-2">>} | |
] | |
}]}] | |
} | |
]. | |
I have used the ip address for both source and destination because no variation of amqp://user:passwd@/sensu worked | |
After restarting Rabbit1, here are the log and sasl-log | |
log: | |
=INFO REPORT==== 30-May-2013::17:33:50 === | |
Server startup complete; 8 plugins started. | |
* amqp_client | |
* mochiweb | |
* rabbitmq_management | |
* rabbitmq_management_agent | |
* rabbitmq_shovel | |
* rabbitmq_shovel_management | |
* rabbitmq_web_dispatch | |
* webmachine | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.296.0> (10.141.151.194:59597 -> 10.40.215.251:5672) | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.299.0> (10.141.151.194:59598 -> 10.40.215.251:5672) | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.302.0> (10.76.219.221:60537 -> 10.40.215.251:5672) | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.305.0> (10.168.29.245:33249 -> 10.40.215.251:5672) | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.311.0> (10.151.25.234:50593 -> 10.40.215.251:5672) | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.317.0> (10.62.97.118:33139 -> 10.40.215.251:5672) | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.334.0> (10.141.151.194:59599 -> 10.40.215.251:5672) | |
=INFO REPORT==== 30-May-2013::17:33:51 === | |
accepting AMQP connection <0.373.0> (10.73.3.20:42261 -> 10.40.215.251:5672) | |
=ERROR REPORT==== 30-May-2013::17:34:53 === | |
** Generic server <0.227.0> terminating | |
** Last message in was {'$gen_cast',init} | |
** When Server state == {state,undefined,undefined,undefined,undefined, | |
test_shovel, | |
{shovel, | |
{endpoint, | |
[{amqp_params_network,<<"admin">>,<<"mdnakar1">>, | |
<<"sensu">>,"54.243.157.166",5672,0,0,0,infinity, | |
none, | |
[#Fun<amqp_uri.7.123484526>, | |
#Fun<amqp_uri.7.123484526>], | |
[],[]}], | |
[]}, | |
{endpoint, | |
[{amqp_params_network,<<"admin">>,<<"mdnakar1">>, | |
<<"sensu">>,"54.245.126.50",5672,0,0,0,infinity, | |
none, | |
[#Fun<amqp_uri.7.123484526>, | |
#Fun<amqp_uri.7.123484526>], | |
[],[]}], | |
[]}, | |
0,on_confirm, | |
#Fun<rabbit_shovel_config.15.100604638>, | |
#Fun<rabbit_shovel_config.15.100604638>, | |
<<"rabbitmq-us-west-2">>,5}, | |
undefined,undefined,undefined,undefined,undefined} | |
** Reason for termination == | |
** {{badmatch,{error,etimedout}}, | |
[{rabbit_shovel_worker,make_conn_and_chan,1}, | |
{rabbit_shovel_worker,handle_cast,2}, | |
{gen_server2,handle_msg,2}, | |
{proc_lib,init_p_do_apply,3}]} | |
As you can see, approx a minute after restart shovel times out | |
sasl log: | |
=SUPERVISOR REPORT==== 30-May-2013::17:33:41 === | |
Supervisor: {<0.224.0>,mirrored_supervisor} | |
Context: shutdown_error | |
Reason: {{badmatch,{error,etimedout}}, | |
[{rabbit_shovel_worker,make_conn_and_chan,1}, | |
{rabbit_shovel_worker,handle_cast,2}, | |
{gen_server2,handle_msg,2}, | |
{proc_lib,init_p_do_apply,3}]} | |
Offender: [{pid,<0.4491.0>}, | |
{name,test_shovel}, | |
{mfa, | |
{rabbit_shovel_worker,start_link, | |
[test_shovel, | |
[{sources, | |
[{broker, | |
"amqp://admin:mdnakar1@54.243.157.166:5672/sensu"}]}, | |
{destinations, | |
[{broker, | |
"amqp://admin:mdnakar1@54.245.126.50:5672/sensu"}]}, | |
{queue,<<"rabbitmq-us-west-2">>}, | |
{reconnect_delay,5}]]}}, | |
{restart_type,{permanent,5}}, | |
{shutdown,4294967295}, | |
{child_type,worker}] | |
=CRASH REPORT==== 30-May-2013::17:34:53 === | |
crasher: | |
initial call: gen:init_it/6 | |
pid: <0.227.0> | |
registered_name: [] | |
exception exit: {{badmatch,{error,etimedout}}, | |
[{rabbit_shovel_worker,make_conn_and_chan,1}, | |
{rabbit_shovel_worker,handle_cast,2}, | |
{gen_server2,handle_msg,2}, | |
{proc_lib,init_p_do_apply,3}]} | |
in function gen_server2:terminate/3 | |
ancestors: [<0.224.0>,test_shovel,rabbit_shovel_sup,<0.220.0>] | |
messages: [] | |
links: [<0.224.0>] | |
dictionary: [{random_seed,{22216,20311,28008}}] | |
trap_exit: true | |
status: running | |
heap_size: 987 | |
stack_size: 24 | |
reductions: 2672 | |
neighbours: | |
=CRASH REPORT==== 30-May-2013::17:35:56 === | |
crasher: | |
initial call: gen:init_it/6 | |
pid: <0.404.0> | |
registered_name: [] | |
exception exit: {{badmatch,{error,etimedout}}, | |
[{rabbit_shovel_worker,make_conn_and_chan,1}, | |
{rabbit_shovel_worker,handle_cast,2}, | |
{gen_server2,handle_msg,2}, | |
{proc_lib,init_p_do_apply,3}]} | |
in function gen_server2:terminate/3 | |
ancestors: [<0.224.0>,test_shovel,rabbit_shovel_sup,<0.220.0>] | |
messages: [] | |
links: [<0.224.0>] | |
dictionary: [{random_seed,{22216,840,5775}}] | |
trap_exit: true | |
status: running | |
heap_size: 987 | |
stack_size: 24 | |
reductions: 2672 | |
neighbours: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
New config file:
[
{rabbitmq_shovel,
[{shovels, [ {test_shovel,
[ {sources, [{broker, "amqp://sensu:mypass@/sensu"}]}
, {destinations, [{broker, "amqp://sensu:mypass@ec2-54-245-126-50.us-west-2.compute.amazonaws.com/sensu"}]}
, {queue, <<"rabbitmq-us-west-2">>}
]
}]}]
}
].
new log output:
=INFO REPORT==== 30-May-2013::21:31:00 ===
Server startup complete; 8 plugins started.
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.313.0> (10.141.151.194:59639 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.316.0> (10.141.151.194:59640 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.319.0> (10.141.151.194:59641 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.322.0> (10.168.29.245:38145 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.325.0> (10.76.219.221:32780 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.328.0> (10.62.97.118:33615 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.370.0> (10.151.25.234:38571 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.391.0> (10.73.3.20:47014 -> 10.40.215.251:5672)
=ERROR REPORT==== 30-May-2013::21:31:15 ===
** Generic server <0.408.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,
test_shovel,
{shovel,
{endpoint,
[{amqp_params_direct,<<"sensu">>,none,<<"sensu">>,
'rabbit@ip-10-40-215-251',none,[]}],
[]},
{endpoint,
[{amqp_params_network,<<"sensu">>,<<"mypass">>,
<<"sensu">>,
"ec2-54-245-126-50.us-west-2.compute.amazonaws.com",
undefined,0,0,0,infinity,none,
[#Fun<amqp_uri.7.123484526>,
#Fun<amqp_uri.7.123484526>],
[],[]}],
[]},
0,on_confirm,
#Fun<rabbit_shovel_config.15.100604638>,
#Fun<rabbit_shovel_config.15.100604638>,
<<"rabbitmq-us-west-2">>,5},
undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {{badmatch,{error,access_refused}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}
and new sasl-log
=CRASH REPORT==== 30-May-2013::21:34:00 ===
crasher:
initial call: gen:init_it/6
pid: <0.632.0>
registered_name: []
exception exit: {{badmatch,{error,access_refused}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}
in function gen_server2:terminate/3
ancestors: [<0.226.0>,test_shovel,rabbit_shovel_sup,<0.222.0>]
messages: []
links: [<0.226.0>]
dictionary: [{random_seed,{22216,13657,10317}}]
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 2670
neighbours: