Skip to content

Instantly share code, notes, and snippets.

@jhhayden

jhhayden/rabbitmq_shovel2

Created May 30, 2013 17:49
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jhhayden/5679708 to your computer and use it in GitHub Desktop.
Save jhhayden/5679708 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
rabbitmq_shovel2
Intro:
I have 2 Rabbits running on 2 different ec2 instances in AWS
Rabbit1 has had a Shovel plugin installed in order to send messages to Rabbit2
These Rabbits are being used for a Sensu monitoring network
The Sensu server is in the same region as Rabbit1
When a message is sent by Sensu to Rabbit1 it fansout to all connected queues (maybe the wrong terminology here), including one called us-west-2
The Shovel is supposed to pick up messages from us-west-2 and forward it to Rabbit2
Later on I will add other queues to forward to other Rabbits
So, here we go
First, here is my minimal rabbit.config to configure the shovel in Rabbit1
[
{rabbitmq_shovel,
[{shovels, [ {test_shovel,
[ {sources, [{broker, "amqp://admin:mdnakar1@54.243.157.166:5672/sensu"}]}
, {destinations, [{broker, "amqp://admin:mdnakar1@54.245.126.50:5672/sensu"}]}
, {queue, <<"rabbitmq-us-west-2">>}
]
}]}]
}
].
I have used the ip address for both source and destination because no variation of amqp://user:passwd@/sensu worked
After restarting Rabbit1, here are the log and sasl-log
log:
=INFO REPORT==== 30-May-2013::17:33:50 ===
Server startup complete; 8 plugins started.
* amqp_client
* mochiweb
* rabbitmq_management
* rabbitmq_management_agent
* rabbitmq_shovel
* rabbitmq_shovel_management
* rabbitmq_web_dispatch
* webmachine
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.296.0> (10.141.151.194:59597 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.299.0> (10.141.151.194:59598 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.302.0> (10.76.219.221:60537 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.305.0> (10.168.29.245:33249 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.311.0> (10.151.25.234:50593 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.317.0> (10.62.97.118:33139 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.334.0> (10.141.151.194:59599 -> 10.40.215.251:5672)
=INFO REPORT==== 30-May-2013::17:33:51 ===
accepting AMQP connection <0.373.0> (10.73.3.20:42261 -> 10.40.215.251:5672)
=ERROR REPORT==== 30-May-2013::17:34:53 ===
** Generic server <0.227.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,
test_shovel,
{shovel,
{endpoint,
[{amqp_params_network,<<"admin">>,<<"mdnakar1">>,
<<"sensu">>,"54.243.157.166",5672,0,0,0,infinity,
none,
[#Fun<amqp_uri.7.123484526>,
#Fun<amqp_uri.7.123484526>],
[],[]}],
[]},
{endpoint,
[{amqp_params_network,<<"admin">>,<<"mdnakar1">>,
<<"sensu">>,"54.245.126.50",5672,0,0,0,infinity,
none,
[#Fun<amqp_uri.7.123484526>,
#Fun<amqp_uri.7.123484526>],
[],[]}],
[]},
0,on_confirm,
#Fun<rabbit_shovel_config.15.100604638>,
#Fun<rabbit_shovel_config.15.100604638>,
<<"rabbitmq-us-west-2">>,5},
undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {{badmatch,{error,etimedout}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}
As you can see, approx a minute after restart shovel times out
sasl log:
=SUPERVISOR REPORT==== 30-May-2013::17:33:41 ===
Supervisor: {<0.224.0>,mirrored_supervisor}
Context: shutdown_error
Reason: {{badmatch,{error,etimedout}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}
Offender: [{pid,<0.4491.0>},
{name,test_shovel},
{mfa,
{rabbit_shovel_worker,start_link,
[test_shovel,
[{sources,
[{broker,
"amqp://admin:mdnakar1@54.243.157.166:5672/sensu"}]},
{destinations,
[{broker,
"amqp://admin:mdnakar1@54.245.126.50:5672/sensu"}]},
{queue,<<"rabbitmq-us-west-2">>},
{reconnect_delay,5}]]}},
{restart_type,{permanent,5}},
{shutdown,4294967295},
{child_type,worker}]
=CRASH REPORT==== 30-May-2013::17:34:53 ===
crasher:
initial call: gen:init_it/6
pid: <0.227.0>
registered_name: []
exception exit: {{badmatch,{error,etimedout}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}
in function gen_server2:terminate/3
ancestors: [<0.224.0>,test_shovel,rabbit_shovel_sup,<0.220.0>]
messages: []
links: [<0.224.0>]
dictionary: [{random_seed,{22216,20311,28008}}]
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 2672
neighbours:
=CRASH REPORT==== 30-May-2013::17:35:56 ===
crasher:
initial call: gen:init_it/6
pid: <0.404.0>
registered_name: []
exception exit: {{badmatch,{error,etimedout}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}
in function gen_server2:terminate/3
ancestors: [<0.224.0>,test_shovel,rabbit_shovel_sup,<0.220.0>]
messages: []
links: [<0.224.0>]
dictionary: [{random_seed,{22216,840,5775}}]
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 2672
neighbours:
@jhhayden
Copy link
Author

per hyperthunk:

I su - rabbitmq and ran erl -sname debug -remsh rabbit@ip-10-40-215-251
this is the results:

Erlang R14B04 (erts-5.8.5) [source] [64-bit] [rq:1] [async-threads:0] [kernel-poll:false]

Eshell V5.8.5 (abort with ^G)

@jhhayden
Copy link
Author

New config file:
[
{rabbitmq_shovel,
[{shovels, [ {test_shovel,
[ {sources, [{broker, "amqp://sensu:mypass@/sensu"}]}
, {destinations, [{broker, "amqp://sensu:mypass@ec2-54-245-126-50.us-west-2.compute.amazonaws.com/sensu"}]}
, {queue, <<"rabbitmq-us-west-2">>}
]
}]}]
}
].

new log output:
=INFO REPORT==== 30-May-2013::21:31:00 ===
Server startup complete; 8 plugins started.

  • amqp_client
  • mochiweb
  • rabbitmq_management
  • rabbitmq_management_agent
  • rabbitmq_shovel
  • rabbitmq_shovel_management
  • rabbitmq_web_dispatch
  • webmachine

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.313.0> (10.141.151.194:59639 -> 10.40.215.251:5672)

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.316.0> (10.141.151.194:59640 -> 10.40.215.251:5672)

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.319.0> (10.141.151.194:59641 -> 10.40.215.251:5672)

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.322.0> (10.168.29.245:38145 -> 10.40.215.251:5672)

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.325.0> (10.76.219.221:32780 -> 10.40.215.251:5672)

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.328.0> (10.62.97.118:33615 -> 10.40.215.251:5672)

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.370.0> (10.151.25.234:38571 -> 10.40.215.251:5672)

=INFO REPORT==== 30-May-2013::21:31:01 ===
accepting AMQP connection <0.391.0> (10.73.3.20:47014 -> 10.40.215.251:5672)

=ERROR REPORT==== 30-May-2013::21:31:15 ===
** Generic server <0.408.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,
test_shovel,
{shovel,
{endpoint,
[{amqp_params_direct,<<"sensu">>,none,<<"sensu">>,
'rabbit@ip-10-40-215-251',none,[]}],
[]},
{endpoint,
[{amqp_params_network,<<"sensu">>,<<"mypass">>,
<<"sensu">>,
"ec2-54-245-126-50.us-west-2.compute.amazonaws.com",
undefined,0,0,0,infinity,none,
[#Fun<amqp_uri.7.123484526>,
#Fun<amqp_uri.7.123484526>],
[],[]}],
[]},
0,on_confirm,
#Fun<rabbit_shovel_config.15.100604638>,
#Fun<rabbit_shovel_config.15.100604638>,
<<"rabbitmq-us-west-2">>,5},
undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {{badmatch,{error,access_refused}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}

and new sasl-log
=CRASH REPORT==== 30-May-2013::21:34:00 ===
crasher:
initial call: gen:init_it/6
pid: <0.632.0>
registered_name: []
exception exit: {{badmatch,{error,access_refused}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}
in function gen_server2:terminate/3
ancestors: [<0.226.0>,test_shovel,rabbit_shovel_sup,<0.222.0>]
messages: []
links: [<0.226.0>]
dictionary: [{random_seed,{22216,13657,10317}}]
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 2670
neighbours:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment