Skip to content

Instantly share code, notes, and snippets.

@mrballcb

mrballcb/description.txt

Last active August 20, 2019 18:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mrballcb/629cf122869baf1f02ae838534db9cc5 to your computer and use it in GitHub Desktop.
Save mrballcb/629cf122869baf1f02ae838534db9cc5 to your computer and use it in GitHub Desktop.
Download ZIP
RabbitMQ Federation Instability
Raw
description.txt
Cluster description:
Three 2 node RMQ-3.7.14 clusters, running in Kube, all inbound ports are exposed
via an internal ELB.
All users, permissions, policies, parameters (upstreams, sets) are defined in a
definitions.json file read when each cluster node starts.
The data volumes are EBS volumes, so is persistent across restarts.
Cluster locations:
us-east-1 (10.9.73.0/24)
eu-central-1 (10.10.17.0/24)
us-west-1 (10.11.12.0/24)
Each cluster is federated to the other 2 (forms a bi-directional triangle formation), and
is connecting to the others through their ELB. Federation docs refer to it as a "small
complete graph" and it looks like this:
https://www.rabbitmq.com/img/federation03.png
Each cluster has:
2 federation-upstreams in a federation-upstream-set A
2 federation-upstreams in a federation-upstream-set B
2 policies (1 for each federation-upstream-set, applied by a regex against exchange names)
Problem description:
Create an exchange, policy applies federation to it, and brings up federation
to the other two clusters. Is stable for a while (a day, a week sometimes).
Both us-east-1 and us-west-1 seem to be stable federated to each other long
term.
The eu-central-1 seems to be stable federated *to* us-east-1 and us-west-1.
Main Problem:
When it finally starts messing up, the federation link *to* eu-central-1 from
both US regions is constantly erroring and reconnecting, about every 45 seconds.
I don't have a successful way to recover other than completely deleting the
exchange from all clusters, then let our custom application recreate the
exchange.
IPs:
us-east-1 ELB rmq-fed-debug-us-east-1.dev.example.net:
10.9.73.4
10.9.73.114
10.9.73.211
us-east-1 cluster members:
rmq-debug-rabbitmq-ha-0 10.9.73.23
rmq-debug-rabbitmq-ha-1 10.9.73.113
eu-central-1 ELB rmq-fed-debug-eu-central-1.dev.example.net:
10.10.17.48
10.10.17.110
10.10.17.163
eu-central-1 cluster members:
rmq-debug-rabbitmq-ha-0 10.10.17.122
rmq-debug-rabbitmq-ha-1 10.10.17.169
us-west-1 ELB rmq-fed-debug-us-west-1.dev.example.net:
10.11.61.57
10.11.61.88
us-west-1 cluster members:
rmq-debug-rabbitmq-ha-0 10.11.61.24
rmq-debug-rabbitmq-ha-1 10.11.61.99
Non Debug logs for the recipient (10.10.17.122) of this federation attempt follow.
This rmq node is in eu-central-1.
The complaint seems to be spelled out in line 7, line 17, and line 33.
The exchange exists so I'm not quite sure what error it's describing.
It seems like merely it's complaining that the federation link is down, which is
accurate. Is there something else non-obvious?
1 2019-08-17 02:13:52.456 [info] <0.11875.1031> accepting AMQP connection <0.11875.1031> (10.10.17.122:29584 -> 100.96.12.4:5672)
2 2019-08-17 02:13:52.642 [info] <0.11875.1031> connection <0.11875.1031> (10.10.17.122:29584 -> 100.96.12.4:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
3 2019-08-17 02:13:54.126 [info] <0.12298.1031> accepting AMQP connection <0.12298.1031> (100.96.2.0:53420 -> 100.96.12.4:5672)
4 2019-08-17 02:14:23.508 [info] <0.12277.1031> accepting AMQP connection <0.12277.1031> (100.96.2.0:53470 -> 100.96.12.4:5672)
5 2019-08-17 02:14:23.695 [info] <0.12277.1031> connection <0.12277.1031> (100.96.2.0:53470 -> 100.96.12.4:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
6 2019-08-17 02:14:23.881 [error] <0.11731.1031> Channel error on connection <0.12277.1031> (100.96.2.0:53470 -> 100.96.12.4:5672, vhost: '/', user: 'FEDUSER'), channel 1:
7 operation exchange.declare caused a channel exception not_found: no exchange 'federation: bob-request-test-1 -> rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local B' in vhost '/'
8 2019-08-17 02:14:24.066 [info] <0.12277.1031> closing AMQP connection <0.12277.1031> (100.96.2.0:53470 -> 100.96.12.4:5672, vhost: '/', user: 'FEDUSER')
9 2019-08-17 02:14:24.732 [info] <0.11720.1031> closing AMQP connection <0.11720.1031> (10.10.17.122:29580 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited), vhost: '/', user: 'FEDUSER')
10 2019-08-17 02:14:25.433 [info] <0.11868.1031> closing AMQP connection <0.11868.1031> (10.10.17.122:29582 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited), vhost: '/', user: 'FEDUSER')
11 2019-08-17 02:14:30.613 [info] <0.12132.1031> accepting AMQP connection <0.12132.1031> (100.96.4.0:53264 -> 100.96.12.4:5672)
12 2019-08-17 02:14:30.771 [info] <0.12132.1031> Connection <0.12132.1031> (100.96.4.0:53264 -> 100.96.12.4:5672) has a client-provided name: Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited)
13 2019-08-17 02:14:30.928 [info] <0.12132.1031> connection <0.12132.1031> (100.96.4.0:53264 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited)): user 'FEDUSER' authenticated and granted access to vhost '/'
14 2019-08-17 02:15:02.246 [info] <0.12094.1031> accepting AMQP connection <0.12094.1031> (100.96.4.0:53294 -> 100.96.12.4:5672)
15 2019-08-17 02:15:02.433 [info] <0.12094.1031> connection <0.12094.1031> (100.96.4.0:53294 -> 100.96.12.4:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
16 2019-08-17 02:15:02.620 [error] <0.11832.1031> Channel error on connection <0.12094.1031> (100.96.4.0:53294 -> 100.96.12.4:5672, vhost: '/', user: 'FEDUSER'), channel 1:
17 operation exchange.declare caused a channel exception not_found: no exchange 'federation: bob-request-test-1 -> rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local A' in vhost '/'
18 2019-08-17 02:15:02.806 [info] <0.12094.1031> closing AMQP connection <0.12094.1031> (100.96.4.0:53294 -> 100.96.12.4:5672, vhost: '/', user: 'FEDUSER')
19 2019-08-17 02:15:04.184 [info] <0.12132.1031> closing AMQP connection <0.12132.1031> (100.96.4.0:53264 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited), vhost: '/', user: 'FEDUSER')
20 2019-08-17 02:15:09.381 [info] <0.12462.1031> accepting AMQP connection <0.12462.1031> (10.10.17.122:29672 -> 100.96.12.4:5672)
21 2019-08-17 02:15:09.538 [info] <0.12462.1031> Connection <0.12462.1031> (10.10.17.122:29672 -> 100.96.12.4:5672) has a client-provided name: Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited)
22 2019-08-17 02:15:09.694 [info] <0.12462.1031> connection <0.12462.1031> (10.10.17.122:29672 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited)): user 'FEDUSER' authenticated and granted access to vhost '/'
23 2019-08-17 02:15:11.589 [info] <0.12744.1031> accepting AMQP connection <0.12744.1031> (100.96.2.0:53534 -> 100.96.12.4:5672)
24 2019-08-17 02:15:11.904 [info] <0.12744.1031> connection <0.12744.1031> (100.96.2.0:53534 -> 100.96.12.4:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
25 2019-08-17 02:15:12.536 [info] <0.12744.1031> closing AMQP connection <0.12744.1031> (100.96.2.0:53534 -> 100.96.12.4:5672, vhost: '/', user: 'FEDUSER')
26 2019-08-17 02:15:42.929 [info] <0.12462.1031> closing AMQP connection <0.12462.1031> (10.10.17.122:29672 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited), vhost: '/', user: 'FEDUSER')
27 2019-08-17 02:15:48.109 [info] <0.12313.1031> accepting AMQP connection <0.12313.1031> (10.10.17.122:29710 -> 100.96.12.4:5672)
28 2019-08-17 02:15:48.266 [info] <0.12313.1031> Connection <0.12313.1031> (10.10.17.122:29710 -> 100.96.12.4:5672) has a client-provided name: Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited)
29 2019-08-17 02:15:48.423 [info] <0.12313.1031> connection <0.12313.1031> (10.10.17.122:29710 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited)): user 'FEDUSER' authenticated and granted access to vhost '/'
30 2019-08-17 02:16:19.770 [info] <0.12663.1031> accepting AMQP connection <0.12663.1031> (100.96.4.0:53396 -> 100.96.12.4:5672)
31 2019-08-17 02:16:19.958 [info] <0.12663.1031> connection <0.12663.1031> (100.96.4.0:53396 -> 100.96.12.4:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
32 2019-08-17 02:16:20.146 [error] <0.12901.1031> Channel error on connection <0.12663.1031> (100.96.4.0:53396 -> 100.96.12.4:5672, vhost: '/', user: 'FEDUSER'), channel 1:
33 operation exchange.declare caused a channel exception not_found: no exchange 'federation: bob-request-test-1 -> rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local A' in vhost '/'
34 2019-08-17 02:16:20.333 [info] <0.12663.1031> closing AMQP connection <0.12663.1031> (100.96.4.0:53396 -> 100.96.12.4:5672, vhost: '/', user: 'FEDUSER')
35 2019-08-17 02:16:21.702 [info] <0.12313.1031> closing AMQP connection <0.12313.1031> (10.10.17.122:29710 -> 100.96.12.4:5672 - Federation link (upstream: bridge_solicited_stream-eu-central-1, policy: federate-bob-solicited), vhost: '/', user: 'FEDUSER')
Debug level logs for the federation initiator in us-east-1:
1 2019-08-17 01:47:22.721 [debug] <0.418.0> Supervisor {<0.418.0>,rabbit_federation_link_sup} started rabbit_federation_exchange_link:start_link({{upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example....">>],...},...}) at pid <0.4106.1411>
2 2019-08-17 01:47:22.722 [debug] <0.3984.1411> Supervisor {<0.3984.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.4270.1411>
3 2019-08-17 01:47:22.722 [debug] <0.3984.1411> Supervisor {<0.3984.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.4270.1411>, {amqp_params_direct,none,none,<<"/">>,'rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local',...}) at pid <0.3682.1411>
4 2019-08-17 01:47:22.723 [debug] <0.4270.1411> Supervisor {<0.4270.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(direct, <0.3682.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.3...">>) at pid <0.3170.1411>
5 2019-08-17 01:47:22.723 [debug] <0.4270.1411> Supervisor {<0.4270.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.3682.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.3...">>, <0.3170.1411>) at pid <0.4123.1411>
6 2019-08-17 01:47:22.723 [debug] <0.4270.1411> Supervisor {<0.4270.1411>,amqp_connection_type_sup} started rabbit_queue_collector:start_link(<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.3...">>) at pid <0.4069.1411>
7 2019-08-17 01:47:22.723 [debug] <0.4449.1411> Supervisor {<0.4449.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.3124.1411>
8 2019-08-17 01:47:22.723 [debug] <0.4449.1411> Supervisor {<0.4449.1411>,amqp_channel_sup} started amqp_channel:start_link(direct, <0.3682.1411>, 1, <0.3124.1411>, {<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.4044.1411>
9 2019-08-17 01:47:22.724 [debug] <0.2746.1411> Supervisor {<0.2746.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.3174.1411>
10 2019-08-17 01:47:22.724 [debug] <0.2746.1411> Supervisor {<0.2746.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.4044.1411>, <0.4044.1411>, <0.3682.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.3...">>, rabbit_framing_amqp_0_9_1, {user,<<"none">>,[],[{rabbit_auth_backend_dummy,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.4069.1411>, <0.3174.1411>) at pid <0.4262.1411>
11 2019-08-17 01:47:22.724 [debug] <0.3513.1411> Supervisor {<0.3513.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.3512.1411>
12 2019-08-17 01:47:22.724 [debug] <0.3513.1411> Supervisor {<0.3513.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.3512.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-eu-ce...",...}) at pid <0.3678.1411>
13 2019-08-17 01:47:22.900 [debug] <0.3512.1411> Supervisor {<0.3512.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.3678.1411>, <<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>) at pid <0.3985.1411>
14 2019-08-17 01:47:22.900 [debug] <0.3512.1411> Supervisor {<0.3512.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.3678.1411>, <<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>, <0.3985.1411>) at pid <0.4134.1411>
15 2019-08-17 01:47:22.900 [debug] <0.3512.1411> Supervisor {<0.3512.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943639>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.3678.1411>, <<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>) at pid <0.3662.1411>
16 2019-08-17 01:47:22.900 [debug] <0.3512.1411> Supervisor {<0.3512.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943639>, <0.3678.1411>, <0.4134.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>) at pid <0.3623.1411>
17 2019-08-17 01:47:23.213 [debug] <0.3512.1411> Supervisor {<0.3512.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943639>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>}) at pid <0.3400.1411>
18 2019-08-17 01:47:23.213 [debug] <0.3512.1411> Supervisor {<0.3512.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943639>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>}) at pid <0.5040.1411>
19 2019-08-17 01:47:23.370 [debug] <0.3683.1411> Supervisor {<0.3683.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,1}) at pid <0.3729.1411>
20 2019-08-17 01:47:23.370 [debug] <0.3683.1411> Supervisor {<0.3683.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3678.1411>, 1, <0.3729.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,1}) at pid <0.4260.1411>
21 2019-08-17 01:47:23.370 [debug] <0.3683.1411> Supervisor {<0.3683.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943639>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.4260.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,1}) at pid <0.4436.1411>
22 2019-08-17 01:47:23.996 [debug] <0.3490.1411> Supervisor {<0.3490.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3445.1411>
23 2019-08-17 01:47:23.996 [debug] <0.3490.1411> Supervisor {<0.3490.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3678.1411>, 2, <0.3445.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3658.1411>
24 2019-08-17 01:47:23.996 [debug] <0.3490.1411> Supervisor {<0.3490.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943639>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.3658.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3600.1411>
25 2019-08-17 01:47:24.464 [debug] <0.3515.1411> Supervisor {<0.3515.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.1785.1411>
26 2019-08-17 01:47:24.465 [debug] <0.3515.1411> Supervisor {<0.3515.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3678.1411>, 2, <0.1785.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.4150.1411>
27 2019-08-17 01:47:24.465 [debug] <0.3515.1411> Supervisor {<0.3515.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943639>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4150.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3632.1411>
28 2019-08-17 01:47:24.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
29 2019-08-17 01:47:24.672 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
30 2019-08-17 01:47:24.940 [debug] <0.4866.1411> Supervisor {<0.4866.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.3734.1411>
31 2019-08-17 01:47:24.940 [debug] <0.4866.1411> Supervisor {<0.4866.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.3734.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-eu-ce...",...}) at pid <0.3810.1411>
32 2019-08-17 01:47:25.107 [debug] <0.3734.1411> Supervisor {<0.3734.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.3810.1411>, <<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>) at pid <0.3327.1411>
33 2019-08-17 01:47:25.108 [debug] <0.3734.1411> Supervisor {<0.3734.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.3810.1411>, <<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>, <0.3327.1411>) at pid <0.3319.1411>
34 2019-08-17 01:47:25.108 [debug] <0.3734.1411> Supervisor {<0.3734.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943777>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.3810.1411>, <<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>) at pid <0.2007.1411>
35 2019-08-17 01:47:25.108 [debug] <0.3734.1411> Supervisor {<0.3734.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943777>, <0.3810.1411>, <0.3319.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>) at pid <0.4274.1411>
36 2019-08-17 01:47:25.198 [debug] <0.4695.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
37 2019-08-17 01:47:25.290 [debug] <0.3693.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
38 2019-08-17 01:47:25.422 [debug] <0.3734.1411> Supervisor {<0.3734.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943777>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>}) at pid <0.3500.1411>
39 2019-08-17 01:47:25.422 [debug] <0.3734.1411> Supervisor {<0.3734.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943777>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>}) at pid <0.2476.1411>
40 2019-08-17 01:47:25.578 [debug] <0.4252.1411> Supervisor {<0.4252.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>,1}) at pid <0.4284.1411>
41 2019-08-17 01:47:25.578 [debug] <0.4252.1411> Supervisor {<0.4252.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3810.1411>, 1, <0.4284.1411>, {<<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>,1}) at pid <0.3360.1411>
42 2019-08-17 01:47:25.578 [debug] <0.4252.1411> Supervisor {<0.4252.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943777>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.3360.1411>, {<<"client 100.96.3.38:35231 -> 10.10.17.48:5672">>,1}) at pid <0.2975.1411>
43 2019-08-17 01:47:26.220 [debug] <0.4636.1411> Supervisor {<0.4636.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3057.1411>
44 2019-08-17 01:47:26.220 [debug] <0.4636.1411> Supervisor {<0.4636.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3678.1411>, 2, <0.3057.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3294.1411>
45 2019-08-17 01:47:26.220 [debug] <0.4636.1411> Supervisor {<0.4636.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943639>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.3294.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3345.1411>
46 2019-08-17 01:47:26.600 [debug] <0.4181.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
47 2019-08-17 01:47:26.701 [info] <0.4106.1411> Federation exchange 'bob-request-test-1' in vhost '/' connected to exchange 'bob-request-test-1' in vhost '/' on amqp://rmq-fed-debug-eu-central-1.dev.example.net
48 2019-08-17 01:47:28.771 [debug] <0.3813.1411> Supervisor {<0.3813.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.4497.1411>
49 2019-08-17 01:47:28.771 [debug] <0.3813.1411> Supervisor {<0.3813.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.4497.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-us-ea...",...}) at pid <0.4364.1411>
50 2019-08-17 01:47:28.864 [debug] <0.4497.1411> Supervisor {<0.4497.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.4364.1411>, <<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>) at pid <0.3939.1411>
51 2019-08-17 01:47:28.864 [debug] <0.4497.1411> Supervisor {<0.4497.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.4364.1411>, <<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>, <0.3939.1411>) at pid <0.3529.1411>
52 2019-08-17 01:47:28.864 [debug] <0.4497.1411> Supervisor {<0.4497.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943775>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.4364.1411>, <<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>) at pid <0.2943.1411>
53 2019-08-17 01:47:28.864 [debug] <0.4497.1411> Supervisor {<0.4497.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943775>, <0.4364.1411>, <0.3529.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>) at pid <0.3543.1411>
54 2019-08-17 01:47:29.005 [debug] <0.4497.1411> Supervisor {<0.4497.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943775>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>}) at pid <0.3201.1411>
55 2019-08-17 01:47:29.005 [debug] <0.4497.1411> Supervisor {<0.4497.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943775>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>}) at pid <0.5023.1411>
56 2019-08-17 01:47:29.075 [debug] <0.4570.1411> Supervisor {<0.4570.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>,1}) at pid <0.3605.1411>
57 2019-08-17 01:47:29.076 [debug] <0.4570.1411> Supervisor {<0.4570.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.4364.1411>, 1, <0.3605.1411>, {<<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>,1}) at pid <0.3413.1411>
58 2019-08-17 01:47:29.076 [debug] <0.4570.1411> Supervisor {<0.4570.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943775>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.3413.1411>, {<<"client 100.96.3.38:36709 -> 10.9.73.4:5672">>,1}) at pid <0.4291.1411>
59 2019-08-17 01:47:30.196 [debug] <0.3868.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
60 2019-08-17 01:47:30.291 [debug] <0.4279.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
61 2019-08-17 01:47:31.601 [debug] <0.3828.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
62 2019-08-17 01:47:31.808 [debug] <0.4621.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
63 2019-08-17 01:47:34.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
64 2019-08-17 01:47:34.673 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
65 2019-08-17 01:47:35.193 [debug] <0.4892.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
66 2019-08-17 01:47:35.288 [debug] <0.4780.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
67 2019-08-17 01:47:36.600 [debug] <0.3408.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
68 2019-08-17 01:47:40.198 [debug] <0.3086.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
69 2019-08-17 01:47:40.293 [debug] <0.2401.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
70 2019-08-17 01:47:41.600 [debug] <0.4346.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
71 2019-08-17 01:47:41.808 [debug] <0.4292.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
72 2019-08-17 01:47:43.831 [debug] <0.3577.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
73 2019-08-17 01:47:43.834 [debug] <0.4087.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
74 2019-08-17 01:47:43.843 [debug] <0.2333.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
75 2019-08-17 01:47:43.854 [debug] <0.4373.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
76 2019-08-17 01:47:44.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
77 2019-08-17 01:47:44.674 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
78 2019-08-17 01:47:45.196 [debug] <0.4858.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
79 2019-08-17 01:47:45.298 [debug] <0.3975.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
80 2019-08-17 01:47:46.599 [debug] <0.4066.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
81 2019-08-17 01:47:50.191 [debug] <0.4130.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
82 2019-08-17 01:47:50.288 [debug] <0.4900.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
83 2019-08-17 01:47:51.600 [debug] <0.4064.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
84 2019-08-17 01:47:51.811 [debug] <0.4782.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
85 2019-08-17 01:47:54.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
86 2019-08-17 01:47:54.672 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
87 2019-08-17 01:47:55.196 [debug] <0.4030.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
88 2019-08-17 01:47:55.291 [debug] <0.3714.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
89 2019-08-17 01:47:55.378 [info] <0.4106.1411> Federation exchange 'bob-request-test-1' in vhost '/' received 'basic.cancel'
90 2019-08-17 01:47:55.378 [debug] <0.2465.1411> Supervisor {<0.2465.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.4390.1411>
91 2019-08-17 01:47:55.378 [debug] <0.2465.1411> Supervisor {<0.2465.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3678.1411>, 2, <0.4390.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.4545.1411>
92 2019-08-17 01:47:55.378 [debug] <0.2465.1411> Supervisor {<0.2465.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943639>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4545.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3926.1411>
93 2019-08-17 01:47:55.847 [debug] <0.4086.1411> Supervisor {<0.4086.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.4410.1411>
94 2019-08-17 01:47:55.847 [debug] <0.4086.1411> Supervisor {<0.4086.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3678.1411>, 2, <0.4410.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.4906.1411>
95 2019-08-17 01:47:55.847 [debug] <0.4086.1411> Supervisor {<0.4086.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943639>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4906.1411>, {<<"client 100.96.3.38:43295 -> 10.10.17.110:5672">>,2}) at pid <0.3248.1411>
96 2019-08-17 01:47:56.478 [error] <0.418.0> Supervisor {<0.418.0>,rabbit_federation_link_sup} had child {upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example.net">>],
97 <<"bob-request-test-1">>,<<"bob-request-test-1">>,1000,1,5,10000,
98 10000,false,'on-publish',none,
99 <<"bridge_solicited_stream-eu-central-1">>,false} started with rabbit_federation_exchange_link:start_link({{upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example....">>],...},...}) at <0.4106.1411> exit with reason {shutdown,restart} in context child_terminated
100 2019-08-17 01:47:56.601 [debug] <0.3099.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
101 2019-08-17 01:47:59.238 [debug] <0.3564.1411> Supervisor {<0.3564.1411>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.4259.1411>
102 2019-08-17 01:47:59.238 [debug] <0.3564.1411> Supervisor {<0.3564.1411>,rabbit_connection_sup} started rabbit_reader:start_link(<0.4259.1411>, {acceptor,{0,0,0,0,0,0,0,0},5672}) at pid <0.3758.1411>
103 2019-08-17 01:47:59.238 [info] <0.3758.1411> accepting AMQP connection <0.3758.1411> (100.96.7.0:35134 -> 100.96.3.38:5672)
104 2019-08-17 01:47:59.287 [debug] <0.4261.1411> Supervisor {<0.4261.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.5150.1411>
105 2019-08-17 01:47:59.287 [debug] <0.4261.1411> Supervisor {<0.4261.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.5150.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-us-ea...",...}) at pid <0.2968.1411>
106 2019-08-17 01:47:59.308 [debug] <0.3758.1411> User 'FEDUSER' authenticated successfully by backend rabbit_auth_backend_internal
107 2019-08-17 01:47:59.373 [debug] <0.5150.1411> Supervisor {<0.5150.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.2968.1411>, <<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>) at pid <0.4020.1411>
108 2019-08-17 01:47:59.373 [debug] <0.5150.1411> Supervisor {<0.5150.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.2968.1411>, <<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>, <0.4020.1411>) at pid <0.4675.1411>
109 2019-08-17 01:47:59.373 [debug] <0.5150.1411> Supervisor {<0.5150.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943847>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.2968.1411>, <<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>) at pid <0.4037.1411>
110 2019-08-17 01:47:59.373 [debug] <0.5150.1411> Supervisor {<0.5150.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943847>, <0.2968.1411>, <0.4675.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>) at pid <0.3567.1411>
111 2019-08-17 01:47:59.378 [debug] <0.4259.1411> Supervisor {<0.4259.1411>,rabbit_connection_helper_sup} started rabbit_queue_collector:start_link(<<"100.96.7.0:35134 -> 100.96.3.38:5672">>) at pid <0.3853.1411>
112 2019-08-17 01:47:59.378 [debug] <0.4259.1411> Supervisor {<0.4259.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943842>, 10, #Fun<rabbit_reader.34.21003307>, {heartbeat_sender,<<"100.96.7.0:35134 -> 100.96.3.38:5672">>}) at pid <0.4297.1411>
113 2019-08-17 01:47:59.378 [debug] <0.4259.1411> Supervisor {<0.4259.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943842>, 10, #Fun<rabbit_reader.35.21003307>, {heartbeat_receiver,<<"100.96.7.0:35134 -> 100.96.3.38:5672">>}) at pid <0.3128.1411>
114 2019-08-17 01:47:59.378 [debug] <0.4259.1411> Supervisor {<0.4259.1411>,rabbit_connection_helper_sup} started rabbit_channel_sup_sup:start_link() at pid <0.3289.1411>
115 2019-08-17 01:47:59.378 [info] <0.3758.1411> connection <0.3758.1411> (100.96.7.0:35134 -> 100.96.3.38:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
116 2019-08-17 01:47:59.447 [debug] <0.4072.1411> Supervisor {<0.4072.1411>,rabbit_channel_sup} started rabbit_writer:start_link(#Port<0.12943842>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.3758.1411>, {<<"100.96.7.0:35134 -> 100.96.3.38:5672">>,1}, true) at pid <0.4229.1411>
117 2019-08-17 01:47:59.447 [debug] <0.4072.1411> Supervisor {<0.4072.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"100.96.7.0:35134 -> 100.96.3.38:5672">>,1}) at pid <0.4586.1411>
118 2019-08-17 01:47:59.447 [debug] <0.4072.1411> Supervisor {<0.4072.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.3758.1411>, <0.4229.1411>, <0.3758.1411>, <<"100.96.7.0:35134 -> 100.96.3.38:5672">>, rabbit_framing_amqp_0_9_1, {user,<<FEDUSER>>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.3853.1411>, <0.4586.1411>) at pid <0.4227.1411>
119 2019-08-17 01:47:59.514 [debug] <0.5150.1411> Supervisor {<0.5150.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943847>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>}) at pid <0.3643.1411>
120 2019-08-17 01:47:59.514 [debug] <0.5150.1411> Supervisor {<0.5150.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943847>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>}) at pid <0.4029.1411>
121 2019-08-17 01:47:59.584 [debug] <0.3197.1411> Supervisor {<0.3197.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>,1}) at pid <0.3279.1411>
122 2019-08-17 01:47:59.584 [debug] <0.3197.1411> Supervisor {<0.3197.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.2968.1411>, 1, <0.3279.1411>, {<<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>,1}) at pid <0.4846.1411>
123 2019-08-17 01:47:59.585 [debug] <0.3197.1411> Supervisor {<0.3197.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943847>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.4846.1411>, {<<"client 100.96.3.38:50577 -> 10.9.73.114:5672">>,1}) at pid <0.4482.1411>
124 2019-08-17 01:47:59.655 [info] <0.3758.1411> closing AMQP connection <0.3758.1411> (100.96.7.0:35134 -> 100.96.3.38:5672, vhost: '/', user: 'FEDUSER')
125 2019-08-17 01:48:00.196 [debug] <0.4785.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
126 2019-08-17 01:48:00.291 [debug] <0.4319.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
127 2019-08-17 01:48:01.486 [debug] <0.418.0> Supervisor {<0.418.0>,rabbit_federation_link_sup} started rabbit_federation_exchange_link:start_link({{upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example....">>],...},...}) at pid <0.3816.1411>
128 2019-08-17 01:48:01.486 [debug] <0.4451.1411> Supervisor {<0.4451.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.4351.1411>
129 2019-08-17 01:48:01.486 [debug] <0.4451.1411> Supervisor {<0.4451.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.4351.1411>, {amqp_params_direct,none,none,<<"/">>,'rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local',...}) at pid <0.4159.1411>
130 2019-08-17 01:48:01.487 [debug] <0.4351.1411> Supervisor {<0.4351.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(direct, <0.4159.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>) at pid <0.4421.1411>
131 2019-08-17 01:48:01.487 [debug] <0.4351.1411> Supervisor {<0.4351.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.4159.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>, <0.4421.1411>) at pid <0.4005.1411>
132 2019-08-17 01:48:01.487 [debug] <0.4351.1411> Supervisor {<0.4351.1411>,amqp_connection_type_sup} started rabbit_queue_collector:start_link(<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>) at pid <0.4116.1411>
133 2019-08-17 01:48:01.487 [debug] <0.4316.1411> Supervisor {<0.4316.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.4415.1411>
134 2019-08-17 01:48:01.487 [debug] <0.4316.1411> Supervisor {<0.4316.1411>,amqp_channel_sup} started amqp_channel:start_link(direct, <0.4159.1411>, 1, <0.4415.1411>, {<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.5214.1411>
135 2019-08-17 01:48:01.487 [debug] <0.4305.1411> Supervisor {<0.4305.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.3036.1411>
136 2019-08-17 01:48:01.487 [debug] <0.4305.1411> Supervisor {<0.4305.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.5214.1411>, <0.5214.1411>, <0.4159.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>, rabbit_framing_amqp_0_9_1, {user,<<"none">>,[],[{rabbit_auth_backend_dummy,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.4116.1411>, <0.3036.1411>) at pid <0.3629.1411>
137 2019-08-17 01:48:01.488 [debug] <0.4371.1411> Supervisor {<0.4371.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.3477.1411>
138 2019-08-17 01:48:01.488 [debug] <0.4371.1411> Supervisor {<0.4371.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.3477.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-eu-ce...",...}) at pid <0.3666.1411>
139 2019-08-17 01:48:01.600 [debug] <0.4021.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
140 2019-08-17 01:48:01.658 [debug] <0.3477.1411> Supervisor {<0.3477.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.3666.1411>, <<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>) at pid <0.4100.1411>
141 2019-08-17 01:48:01.658 [debug] <0.3477.1411> Supervisor {<0.3477.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.3666.1411>, <<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>, <0.4100.1411>) at pid <0.3797.1411>
142 2019-08-17 01:48:01.658 [debug] <0.3477.1411> Supervisor {<0.3477.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943772>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.3666.1411>, <<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>) at pid <0.2958.1411>
143 2019-08-17 01:48:01.658 [debug] <0.3477.1411> Supervisor {<0.3477.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943772>, <0.3666.1411>, <0.3797.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>) at pid <0.2761.1411>
144 2019-08-17 01:48:01.810 [debug] <0.3242.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
145 2019-08-17 01:48:01.972 [debug] <0.3477.1411> Supervisor {<0.3477.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943772>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>}) at pid <0.2760.1411>
146 2019-08-17 01:48:01.973 [debug] <0.3477.1411> Supervisor {<0.3477.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943772>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>}) at pid <0.3870.1411>
147 2019-08-17 01:48:02.130 [debug] <0.3018.1411> Supervisor {<0.3018.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,1}) at pid <0.4337.1411>
148 2019-08-17 01:48:02.130 [debug] <0.3018.1411> Supervisor {<0.3018.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3666.1411>, 1, <0.4337.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,1}) at pid <0.4666.1411>
149 2019-08-17 01:48:02.130 [debug] <0.3018.1411> Supervisor {<0.3018.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943772>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.4666.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,1}) at pid <0.4107.1411>
150 2019-08-17 01:48:02.758 [debug] <0.3732.1411> Supervisor {<0.3732.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4868.1411>
151 2019-08-17 01:48:02.758 [debug] <0.3732.1411> Supervisor {<0.3732.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3666.1411>, 2, <0.4868.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.3956.1411>
152 2019-08-17 01:48:02.758 [debug] <0.3732.1411> Supervisor {<0.3732.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943772>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.3956.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.5159.1411>
153 2019-08-17 01:48:03.227 [debug] <0.4394.1411> Supervisor {<0.4394.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4773.1411>
154 2019-08-17 01:48:03.227 [debug] <0.4394.1411> Supervisor {<0.4394.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3666.1411>, 2, <0.4773.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4650.1411>
155 2019-08-17 01:48:03.227 [debug] <0.4394.1411> Supervisor {<0.4394.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943772>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4650.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4910.1411>
156 2019-08-17 01:48:03.705 [debug] <0.4617.1411> Supervisor {<0.4617.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.4608.1411>
157 2019-08-17 01:48:03.705 [debug] <0.4617.1411> Supervisor {<0.4617.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.4608.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-eu-ce...",...}) at pid <0.3573.1411>
158 2019-08-17 01:48:03.875 [debug] <0.4608.1411> Supervisor {<0.4608.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.3573.1411>, <<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>) at pid <0.3463.1411>
159 2019-08-17 01:48:03.875 [debug] <0.4608.1411> Supervisor {<0.4608.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.3573.1411>, <<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>, <0.3463.1411>) at pid <0.4579.1411>
160 2019-08-17 01:48:03.875 [debug] <0.4608.1411> Supervisor {<0.4608.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943854>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.3573.1411>, <<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>) at pid <0.3892.1411>
161 2019-08-17 01:48:03.875 [debug] <0.4608.1411> Supervisor {<0.4608.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943854>, <0.3573.1411>, <0.4579.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>) at pid <0.4088.1411>
162 2019-08-17 01:48:04.189 [debug] <0.4608.1411> Supervisor {<0.4608.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943854>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>}) at pid <0.4542.1411>
163 2019-08-17 01:48:04.189 [debug] <0.4608.1411> Supervisor {<0.4608.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943854>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>}) at pid <0.3366.1411>
164 2019-08-17 01:48:04.346 [debug] <0.4187.1411> Supervisor {<0.4187.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>,1}) at pid <0.4714.1411>
165 2019-08-17 01:48:04.346 [debug] <0.4187.1411> Supervisor {<0.4187.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3573.1411>, 1, <0.4714.1411>, {<<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>,1}) at pid <0.5204.1411>
166 2019-08-17 01:48:04.346 [debug] <0.4187.1411> Supervisor {<0.4187.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943854>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.5204.1411>, {<<"client 100.96.3.38:59965 -> 10.10.17.110:5672">>,1}) at pid <0.4537.1411>
167 2019-08-17 01:48:04.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
168 2019-08-17 01:48:04.674 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
169 2019-08-17 01:48:04.992 [debug] <0.4914.1411> Supervisor {<0.4914.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4520.1411>
170 2019-08-17 01:48:04.993 [debug] <0.4914.1411> Supervisor {<0.4914.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3666.1411>, 2, <0.4520.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4353.1411>
171 2019-08-17 01:48:04.993 [debug] <0.4914.1411> Supervisor {<0.4914.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943772>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4353.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4084.1411>
172 2019-08-17 01:48:05.193 [debug] <0.3814.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
173 2019-08-17 01:48:05.286 [debug] <0.4248.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
174 2019-08-17 01:48:05.482 [info] <0.3816.1411> Federation exchange 'bob-request-test-1' in vhost '/' connected to exchange 'bob-request-test-1' in vhost '/' on amqp://rmq-fed-debug-eu-central-1.dev.example.net
175 2019-08-17 01:48:06.600 [debug] <0.3556.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
176 2019-08-17 01:48:10.196 [debug] <0.3845.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
177 2019-08-17 01:48:10.291 [debug] <0.4706.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
178 2019-08-17 01:48:11.604 [debug] <0.4673.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
179 2019-08-17 01:48:11.810 [debug] <0.4737.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
180 2019-08-17 01:48:13.831 [debug] <0.2693.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
181 2019-08-17 01:48:13.834 [debug] <0.5322.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
182 2019-08-17 01:48:13.842 [debug] <0.3917.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
183 2019-08-17 01:48:13.853 [debug] <0.3596.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
184 2019-08-17 01:48:14.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
185 2019-08-17 01:48:14.672 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
186 2019-08-17 01:48:15.191 [debug] <0.5012.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
187 2019-08-17 01:48:15.286 [debug] <0.3847.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
188 2019-08-17 01:48:16.599 [debug] <0.4395.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
189 2019-08-17 01:48:18.892 [debug] <0.4212.1411> Supervisor {<0.4212.1411>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.3832.1411>
190 2019-08-17 01:48:18.892 [debug] <0.4212.1411> Supervisor {<0.4212.1411>,rabbit_connection_sup} started rabbit_reader:start_link(<0.3832.1411>, {acceptor,{0,0,0,0,0,0,0,0},5672}) at pid <0.4777.1411>
191 2019-08-17 01:48:18.892 [info] <0.4777.1411> accepting AMQP connection <0.4777.1411> (10.11.61.24:62978 -> 100.96.3.38:5672)
192 2019-08-17 01:48:19.049 [debug] <0.4777.1411> User 'FEDUSER' authenticated successfully by backend rabbit_auth_backend_internal
193 2019-08-17 01:48:19.204 [debug] <0.3832.1411> Supervisor {<0.3832.1411>,rabbit_connection_helper_sup} started rabbit_queue_collector:start_link(<<"10.11.61.24:62978 -> 100.96.3.38:5672">>) at pid <0.4849.1411>
194 2019-08-17 01:48:19.205 [debug] <0.3832.1411> Supervisor {<0.3832.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943887>, 10, #Fun<rabbit_reader.34.21003307>, {heartbeat_sender,<<"10.11.61.24:62978 -> 100.96.3.38:5672">>}) at pid <0.3481.1411>
195 2019-08-17 01:48:19.205 [debug] <0.3832.1411> Supervisor {<0.3832.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943887>, 10, #Fun<rabbit_reader.35.21003307>, {heartbeat_receiver,<<"10.11.61.24:62978 -> 100.96.3.38:5672">>}) at pid <0.5452.1411>
196 2019-08-17 01:48:19.205 [info] <0.4777.1411> connection <0.4777.1411> (10.11.61.24:62978 -> 100.96.3.38:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
197 2019-08-17 01:48:19.205 [debug] <0.3832.1411> Supervisor {<0.3832.1411>,rabbit_connection_helper_sup} started rabbit_channel_sup_sup:start_link() at pid <0.4210.1411>
198 2019-08-17 01:48:19.361 [debug] <0.3450.1411> Supervisor {<0.3450.1411>,rabbit_channel_sup} started rabbit_writer:start_link(#Port<0.12943887>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.4777.1411>, {<<"10.11.61.24:62978 -> 100.96.3.38:5672">>,1}, true) at pid <0.4356.1411>
199 2019-08-17 01:48:19.361 [debug] <0.3450.1411> Supervisor {<0.3450.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"10.11.61.24:62978 -> 100.96.3.38:5672">>,1}) at pid <0.4616.1411>
200 2019-08-17 01:48:19.361 [debug] <0.3450.1411> Supervisor {<0.3450.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.4777.1411>, <0.4356.1411>, <0.4777.1411>, <<"10.11.61.24:62978 -> 100.96.3.38:5672">>, rabbit_framing_amqp_0_9_1, {user,<<FEDUSER>>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.4849.1411>, <0.4616.1411>) at pid <0.4877.1411>
201 2019-08-17 01:48:19.828 [info] <0.4777.1411> closing AMQP connection <0.4777.1411> (10.11.61.24:62978 -> 100.96.3.38:5672, vhost: '/', user: 'FEDUSER')
202 2019-08-17 01:48:20.194 [debug] <0.3234.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
203 2019-08-17 01:48:20.291 [debug] <0.2720.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
204 2019-08-17 01:48:21.600 [debug] <0.3863.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
205 2019-08-17 01:48:21.810 [debug] <0.5926.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
206 2019-08-17 01:48:24.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
207 2019-08-17 01:48:24.673 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
208 2019-08-17 01:48:25.194 [debug] <0.5017.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
209 2019-08-17 01:48:25.283 [debug] <0.3453.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
210 2019-08-17 01:48:26.599 [debug] <0.3412.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
211 2019-08-17 01:48:29.796 [debug] <0.3990.1411> Supervisor {<0.3990.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.4885.1411>
212 2019-08-17 01:48:29.796 [debug] <0.3990.1411> Supervisor {<0.3990.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.4885.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-us-ea...",...}) at pid <0.4442.1411>
213 2019-08-17 01:48:29.815 [debug] <0.4833.1411> Supervisor {<0.4833.1411>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.5215.1411>
214 2019-08-17 01:48:29.815 [debug] <0.4833.1411> Supervisor {<0.4833.1411>,rabbit_connection_sup} started rabbit_reader:start_link(<0.5215.1411>, {acceptor,{0,0,0,0,0,0,0,0},5672}) at pid <0.3598.1411>
215 2019-08-17 01:48:29.816 [info] <0.3598.1411> accepting AMQP connection <0.3598.1411> (100.96.11.0:32368 -> 100.96.3.38:5672)
216 2019-08-17 01:48:29.885 [debug] <0.4885.1411> Supervisor {<0.4885.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.4442.1411>, <<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>) at pid <0.4641.1411>
217 2019-08-17 01:48:29.886 [debug] <0.4885.1411> Supervisor {<0.4885.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.4442.1411>, <<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>, <0.4641.1411>) at pid <0.4184.1411>
218 2019-08-17 01:48:29.886 [debug] <0.4885.1411> Supervisor {<0.4885.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943912>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.4442.1411>, <<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>) at pid <0.5446.1411>
219 2019-08-17 01:48:29.886 [debug] <0.4885.1411> Supervisor {<0.4885.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943912>, <0.4442.1411>, <0.4184.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>) at pid <0.4789.1411>
220 2019-08-17 01:48:29.886 [debug] <0.3598.1411> User 'FEDUSER' authenticated successfully by backend rabbit_auth_backend_internal
221 2019-08-17 01:48:29.956 [debug] <0.5215.1411> Supervisor {<0.5215.1411>,rabbit_connection_helper_sup} started rabbit_queue_collector:start_link(<<"100.96.11.0:32368 -> 100.96.3.38:5672">>) at pid <0.3630.1411>
222 2019-08-17 01:48:29.956 [debug] <0.5215.1411> Supervisor {<0.5215.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943903>, 10, #Fun<rabbit_reader.34.21003307>, {heartbeat_sender,<<"100.96.11.0:32368 -> 100.96.3.38:5672">>}) at pid <0.3530.1411>
223 2019-08-17 01:48:29.957 [debug] <0.5215.1411> Supervisor {<0.5215.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943903>, 10, #Fun<rabbit_reader.35.21003307>, {heartbeat_receiver,<<"100.96.11.0:32368 -> 100.96.3.38:5672">>}) at pid <0.3830.1411>
224 2019-08-17 01:48:29.957 [debug] <0.5215.1411> Supervisor {<0.5215.1411>,rabbit_connection_helper_sup} started rabbit_channel_sup_sup:start_link() at pid <0.4412.1411>
225 2019-08-17 01:48:29.957 [info] <0.3598.1411> connection <0.3598.1411> (100.96.11.0:32368 -> 100.96.3.38:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
226 2019-08-17 01:48:30.027 [debug] <0.4885.1411> Supervisor {<0.4885.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943912>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>}) at pid <0.4046.1411>
227 2019-08-17 01:48:30.027 [debug] <0.4885.1411> Supervisor {<0.4885.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943912>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>}) at pid <0.3235.1411>
228 2019-08-17 01:48:30.027 [debug] <0.3707.1411> Supervisor {<0.3707.1411>,rabbit_channel_sup} started rabbit_writer:start_link(#Port<0.12943903>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.3598.1411>, {<<"100.96.11.0:32368 -> 100.96.3.38:5672">>,1}, true) at pid <0.3574.1411>
229 2019-08-17 01:48:30.027 [debug] <0.3707.1411> Supervisor {<0.3707.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"100.96.11.0:32368 -> 100.96.3.38:5672">>,1}) at pid <0.4138.1411>
230 2019-08-17 01:48:30.027 [debug] <0.3707.1411> Supervisor {<0.3707.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.3598.1411>, <0.3574.1411>, <0.3598.1411>, <<"100.96.11.0:32368 -> 100.96.3.38:5672">>, rabbit_framing_amqp_0_9_1, {user,<<FEDUSER>>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.3630.1411>, <0.4138.1411>) at pid <0.3866.1411>
231 2019-08-17 01:48:30.097 [debug] <0.3665.1411> Supervisor {<0.3665.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>,1}) at pid <0.4375.1411>
232 2019-08-17 01:48:30.097 [debug] <0.3665.1411> Supervisor {<0.3665.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.4442.1411>, 1, <0.4375.1411>, {<<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>,1}) at pid <0.3143.1411>
233 2019-08-17 01:48:30.097 [debug] <0.3665.1411> Supervisor {<0.3665.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943912>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.3143.1411>, {<<"client 100.96.3.38:56227 -> 10.9.73.114:5672">>,1}) at pid <0.2865.1411>
234 2019-08-17 01:48:30.194 [debug] <0.4404.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
235 2019-08-17 01:48:30.236 [info] <0.3598.1411> closing AMQP connection <0.3598.1411> (100.96.11.0:32368 -> 100.96.3.38:5672, vhost: '/', user: 'FEDUSER')
236 2019-08-17 01:48:30.324 [debug] <0.4285.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
237 2019-08-17 01:48:31.600 [debug] <0.4317.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
238 2019-08-17 01:48:31.811 [debug] <0.4135.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
239 2019-08-17 01:48:34.110 [info] <0.3816.1411> Federation exchange 'bob-request-test-1' in vhost '/' received 'basic.cancel'
240 2019-08-17 01:48:34.111 [debug] <0.4670.1411> Supervisor {<0.4670.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4294.1411>
241 2019-08-17 01:48:34.111 [debug] <0.4670.1411> Supervisor {<0.4670.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3666.1411>, 2, <0.4294.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4460.1411>
242 2019-08-17 01:48:34.111 [debug] <0.4670.1411> Supervisor {<0.4670.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943772>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4460.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,2}) at pid <0.4607.1411>
243 2019-08-17 01:48:34.581 [debug] <0.3762.1411> Supervisor {<0.3762.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,3}) at pid <0.5058.1411>
244 2019-08-17 01:48:34.581 [debug] <0.3762.1411> Supervisor {<0.3762.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3666.1411>, 3, <0.5058.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,3}) at pid <0.4120.1411>
245 2019-08-17 01:48:34.581 [debug] <0.3762.1411> Supervisor {<0.3762.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943772>, 3, 131072, rabbit_framing_amqp_0_9_1, <0.4120.1411>, {<<"client 100.96.3.38:60487 -> 10.10.17.48:5672">>,3}) at pid <0.4588.1411>
246 2019-08-17 01:48:34.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
247 2019-08-17 01:48:34.672 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
248 2019-08-17 01:48:35.194 [debug] <0.4063.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
249 2019-08-17 01:48:35.214 [error] <0.418.0> Supervisor {<0.418.0>,rabbit_federation_link_sup} had child {upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example.net">>],
250 <<"bob-request-test-1">>,<<"bob-request-test-1">>,1000,1,5,10000,
251 10000,false,'on-publish',none,
252 <<"bridge_solicited_stream-eu-central-1">>,false} started with rabbit_federation_exchange_link:start_link({{upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example....">>],...},...}) at <0.3816.1411> exit with reason {shutdown,restart} in context child_terminated
253 2019-08-17 01:48:35.289 [debug] <0.3399.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
254 2019-08-17 01:48:36.600 [debug] <0.3664.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
255 2019-08-17 01:48:40.194 [debug] <0.4808.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
256 2019-08-17 01:48:40.220 [debug] <0.418.0> Supervisor {<0.418.0>,rabbit_federation_link_sup} started rabbit_federation_exchange_link:start_link({{upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example....">>],...},...}) at pid <0.4176.1411>
257 2019-08-17 01:48:40.221 [debug] <0.3606.1411> Supervisor {<0.3606.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.5475.1411>
258 2019-08-17 01:48:40.221 [debug] <0.3606.1411> Supervisor {<0.3606.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.5475.1411>, {amqp_params_direct,none,none,<<"/">>,'rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local',...}) at pid <0.4236.1411>
259 2019-08-17 01:48:40.222 [debug] <0.5475.1411> Supervisor {<0.5475.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(direct, <0.4236.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>) at pid <0.5184.1411>
260 2019-08-17 01:48:40.222 [debug] <0.5475.1411> Supervisor {<0.5475.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.4236.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>, <0.5184.1411>) at pid <0.3565.1411>
261 2019-08-17 01:48:40.222 [debug] <0.5475.1411> Supervisor {<0.5475.1411>,amqp_connection_type_sup} started rabbit_queue_collector:start_link(<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>) at pid <0.4869.1411>
262 2019-08-17 01:48:40.222 [debug] <0.4776.1411> Supervisor {<0.4776.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.4749.1411>
263 2019-08-17 01:48:40.222 [debug] <0.4776.1411> Supervisor {<0.4776.1411>,amqp_channel_sup} started amqp_channel:start_link(direct, <0.4236.1411>, 1, <0.4749.1411>, {<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.3922.1411>
264 2019-08-17 01:48:40.222 [debug] <0.4073.1411> Supervisor {<0.4073.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2...">>,...}) at pid <0.4809.1411>
265 2019-08-17 01:48:40.222 [debug] <0.4073.1411> Supervisor {<0.4073.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.3922.1411>, <0.3922.1411>, <0.4236.1411>, <<"<rabbit@rmq-debug-rabbitmq-ha-0.rmq-debug-rabbitmq-ha-discovery.rmq-debug.svc.cluster.local.2.4...">>, rabbit_framing_amqp_0_9_1, {user,<<"none">>,[],[{rabbit_auth_backend_dummy,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.4869.1411>, <0.4809.1411>) at pid <0.4571.1411>
266 2019-08-17 01:48:40.222 [debug] <0.5296.1411> Supervisor {<0.5296.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.5149.1411>
267 2019-08-17 01:48:40.223 [debug] <0.5296.1411> Supervisor {<0.5296.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.5149.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-eu-ce...",...}) at pid <0.5237.1411>
268 2019-08-17 01:48:40.291 [debug] <0.4082.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
269 2019-08-17 01:48:40.414 [debug] <0.5149.1411> Supervisor {<0.5149.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.5237.1411>, <<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>) at pid <0.4521.1411>
270 2019-08-17 01:48:40.414 [debug] <0.5149.1411> Supervisor {<0.5149.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.5237.1411>, <<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>, <0.4521.1411>) at pid <0.4376.1411>
271 2019-08-17 01:48:40.415 [debug] <0.5149.1411> Supervisor {<0.5149.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943867>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.5237.1411>, <<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>) at pid <0.4553.1411>
272 2019-08-17 01:48:40.415 [debug] <0.5149.1411> Supervisor {<0.5149.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943867>, <0.5237.1411>, <0.4376.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>) at pid <0.3897.1411>
273 2019-08-17 01:48:40.729 [debug] <0.5149.1411> Supervisor {<0.5149.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943867>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>}) at pid <0.4572.1411>
274 2019-08-17 01:48:40.729 [debug] <0.5149.1411> Supervisor {<0.5149.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943867>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>}) at pid <0.4111.1411>
275 2019-08-17 01:48:40.886 [debug] <0.4464.1411> Supervisor {<0.4464.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,1}) at pid <0.3802.1411>
276 2019-08-17 01:48:40.886 [debug] <0.4464.1411> Supervisor {<0.4464.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.5237.1411>, 1, <0.3802.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,1}) at pid <0.4224.1411>
277 2019-08-17 01:48:40.886 [debug] <0.4464.1411> Supervisor {<0.4464.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943867>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.4224.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,1}) at pid <0.4522.1411>
278 2019-08-17 01:48:41.514 [debug] <0.3772.1411> Supervisor {<0.3772.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4504.1411>
279 2019-08-17 01:48:41.514 [debug] <0.3772.1411> Supervisor {<0.3772.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.5237.1411>, 2, <0.4504.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4561.1411>
280 2019-08-17 01:48:41.514 [debug] <0.3772.1411> Supervisor {<0.3772.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943867>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4561.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4817.1411>
281 2019-08-17 01:48:41.600 [debug] <0.4872.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
282 2019-08-17 01:48:41.810 [debug] <0.4539.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
283 2019-08-17 01:48:41.984 [debug] <0.4873.1411> Supervisor {<0.4873.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4190.1411>
284 2019-08-17 01:48:41.985 [debug] <0.4873.1411> Supervisor {<0.4873.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.5237.1411>, 2, <0.4190.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.3902.1411>
285 2019-08-17 01:48:41.985 [debug] <0.4873.1411> Supervisor {<0.4873.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943867>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.3902.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4448.1411>
286 2019-08-17 01:48:42.464 [debug] <0.5370.1411> Supervisor {<0.5370.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.4904.1411>
287 2019-08-17 01:48:42.464 [debug] <0.5370.1411> Supervisor {<0.5370.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.4904.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-eu-ce...",...}) at pid <0.4660.1411>
288 2019-08-17 01:48:42.636 [debug] <0.4904.1411> Supervisor {<0.4904.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.4660.1411>, <<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>) at pid <0.5148.1411>
289 2019-08-17 01:48:42.637 [debug] <0.4904.1411> Supervisor {<0.4904.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.4660.1411>, <<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>, <0.5148.1411>) at pid <0.4430.1411>
290 2019-08-17 01:48:42.637 [debug] <0.4904.1411> Supervisor {<0.4904.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943880>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.4660.1411>, <<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>) at pid <0.4903.1411>
291 2019-08-17 01:48:42.637 [debug] <0.4904.1411> Supervisor {<0.4904.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943880>, <0.4660.1411>, <0.4430.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>) at pid <0.5778.1411>
292 2019-08-17 01:48:42.951 [debug] <0.4904.1411> Supervisor {<0.4904.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943880>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>}) at pid <0.4792.1411>
293 2019-08-17 01:48:42.952 [debug] <0.4904.1411> Supervisor {<0.4904.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943880>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>}) at pid <0.4936.1411>
294 2019-08-17 01:48:43.108 [debug] <0.3401.1411> Supervisor {<0.3401.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>,1}) at pid <0.3391.1411>
295 2019-08-17 01:48:43.108 [debug] <0.3401.1411> Supervisor {<0.3401.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.4660.1411>, 1, <0.3391.1411>, {<<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>,1}) at pid <0.3272.1411>
296 2019-08-17 01:48:43.108 [debug] <0.3401.1411> Supervisor {<0.3401.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943880>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.3272.1411>, {<<"client 100.96.3.38:56453 -> 10.10.17.48:5672">>,1}) at pid <0.4055.1411>
297 2019-08-17 01:48:43.751 [debug] <0.5067.1411> Supervisor {<0.5067.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4847.1411>
298 2019-08-17 01:48:43.751 [debug] <0.5067.1411> Supervisor {<0.5067.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.5237.1411>, 2, <0.4847.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4140.1411>
299 2019-08-17 01:48:43.752 [debug] <0.5067.1411> Supervisor {<0.5067.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943867>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4140.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.3624.1411>
300 2019-08-17 01:48:43.832 [debug] <0.3548.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
301 2019-08-17 01:48:43.835 [debug] <0.5193.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
302 2019-08-17 01:48:43.843 [debug] <0.4239.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
303 2019-08-17 01:48:43.855 [debug] <0.4444.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
304 2019-08-17 01:48:44.236 [info] <0.4176.1411> Federation exchange 'bob-request-test-1' in vhost '/' connected to exchange 'bob-request-test-1' in vhost '/' on amqp://rmq-fed-debug-eu-central-1.dev.example.net
305 2019-08-17 01:48:44.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
306 2019-08-17 01:48:44.672 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
307 2019-08-17 01:48:45.194 [debug] <0.4296.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
308 2019-08-17 01:48:45.289 [debug] <0.3953.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
309 2019-08-17 01:48:46.600 [debug] <0.3246.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
310 2019-08-17 01:48:50.004 [debug] <0.3995.1411> Supervisor {<0.3995.1411>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.3675.1411>
311 2019-08-17 01:48:50.004 [debug] <0.3995.1411> Supervisor {<0.3995.1411>,rabbit_connection_sup} started rabbit_reader:start_link(<0.3675.1411>, {acceptor,{0,0,0,0,0,0,0,0},5672}) at pid <0.5007.1411>
312 2019-08-17 01:48:50.004 [info] <0.5007.1411> accepting AMQP connection <0.5007.1411> (100.96.9.0:39196 -> 100.96.3.38:5672)
313 2019-08-17 01:48:50.169 [debug] <0.5007.1411> User 'FEDUSER' authenticated successfully by backend rabbit_auth_backend_internal
314 2019-08-17 01:48:50.194 [debug] <0.3886.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
315 2019-08-17 01:48:50.289 [debug] <0.4177.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
316 2019-08-17 01:48:50.326 [debug] <0.3675.1411> Supervisor {<0.3675.1411>,rabbit_connection_helper_sup} started rabbit_queue_collector:start_link(<<"100.96.9.0:39196 -> 100.96.3.38:5672">>) at pid <0.4071.1411>
317 2019-08-17 01:48:50.326 [debug] <0.3675.1411> Supervisor {<0.3675.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943942>, 10, #Fun<rabbit_reader.34.21003307>, {heartbeat_sender,<<"100.96.9.0:39196 -> 100.96.3.38:5672">>}) at pid <0.3580.1411>
318 2019-08-17 01:48:50.327 [debug] <0.3675.1411> Supervisor {<0.3675.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943942>, 10, #Fun<rabbit_reader.35.21003307>, {heartbeat_receiver,<<"100.96.9.0:39196 -> 100.96.3.38:5672">>}) at pid <0.5110.1411>
319 2019-08-17 01:48:50.327 [debug] <0.3675.1411> Supervisor {<0.3675.1411>,rabbit_connection_helper_sup} started rabbit_channel_sup_sup:start_link() at pid <0.4972.1411>
320 2019-08-17 01:48:50.327 [info] <0.5007.1411> connection <0.5007.1411> (100.96.9.0:39196 -> 100.96.3.38:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
321 2019-08-17 01:48:50.484 [debug] <0.5174.1411> Supervisor {<0.5174.1411>,rabbit_channel_sup} started rabbit_writer:start_link(#Port<0.12943942>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.5007.1411>, {<<"100.96.9.0:39196 -> 100.96.3.38:5672">>,1}, true) at pid <0.4829.1411>
322 2019-08-17 01:48:50.484 [debug] <0.5174.1411> Supervisor {<0.5174.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"100.96.9.0:39196 -> 100.96.3.38:5672">>,1}) at pid <0.4619.1411>
323 2019-08-17 01:48:50.484 [debug] <0.5174.1411> Supervisor {<0.5174.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.5007.1411>, <0.4829.1411>, <0.5007.1411>, <<"100.96.9.0:39196 -> 100.96.3.38:5672">>, rabbit_framing_amqp_0_9_1, {user,<<FEDUSER>>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.4071.1411>, <0.4619.1411>) at pid <0.4445.1411>
324 2019-08-17 01:48:50.954 [info] <0.5007.1411> closing AMQP connection <0.5007.1411> (100.96.9.0:39196 -> 100.96.3.38:5672, vhost: '/', user: 'FEDUSER')
325 2019-08-17 01:48:51.601 [debug] <0.4358.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
326 2019-08-17 01:48:51.809 [debug] <0.4597.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
327 2019-08-17 01:48:54.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
328 2019-08-17 01:48:54.673 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
329 2019-08-17 01:48:55.194 [debug] <0.4338.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
330 2019-08-17 01:48:55.289 [debug] <0.4354.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
331 2019-08-17 01:48:56.601 [debug] <0.2663.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
332 2019-08-17 01:49:00.197 [debug] <0.4281.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
333 2019-08-17 01:49:00.291 [debug] <0.3763.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
334 2019-08-17 01:49:00.325 [debug] <0.3782.1411> Supervisor {<0.3782.1411>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.3801.1411>
335 2019-08-17 01:49:00.325 [debug] <0.3782.1411> Supervisor {<0.3782.1411>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.3801.1411>, {amqp_params_network,<<FEDUSER>>,<<"FEDPASS">>,<<"/">>,"rmq-fed-debug-us-ea...",...}) at pid <0.3423.1411>
336 2019-08-17 01:49:00.337 [debug] <0.4169.1411> Supervisor {<0.4169.1411>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.4853.1411>
337 2019-08-17 01:49:00.337 [debug] <0.4169.1411> Supervisor {<0.4169.1411>,rabbit_connection_sup} started rabbit_reader:start_link(<0.4853.1411>, {acceptor,{0,0,0,0,0,0,0,0},5672}) at pid <0.4995.1411>
338 2019-08-17 01:49:00.337 [info] <0.4995.1411> accepting AMQP connection <0.4995.1411> (100.96.7.0:35226 -> 100.96.3.38:5672)
339 2019-08-17 01:49:00.406 [debug] <0.3801.1411> Supervisor {<0.3801.1411>,amqp_connection_type_sup} started amqp_channel_sup_sup:start_link(network, <0.3423.1411>, <<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>) at pid <0.3976.1411>
340 2019-08-17 01:49:00.406 [debug] <0.3801.1411> Supervisor {<0.3801.1411>,amqp_connection_type_sup} started amqp_channels_manager:start_link(<0.3423.1411>, <<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>, <0.3976.1411>) at pid <0.3728.1411>
341 2019-08-17 01:49:00.407 [debug] <0.3801.1411> Supervisor {<0.3801.1411>,amqp_connection_type_sup} started rabbit_writer:start_link(#Port<0.12943904>, 0, 4096, rabbit_framing_amqp_0_9_1, <0.3423.1411>, <<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>) at pid <0.5243.1411>
342 2019-08-17 01:49:00.407 [debug] <0.3801.1411> Supervisor {<0.3801.1411>,amqp_connection_type_sup} started amqp_main_reader:start_link(#Port<0.12943904>, <0.3423.1411>, <0.3728.1411>, {method,rabbit_framing_amqp_0_9_1}, <<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>) at pid <0.4765.1411>
343 2019-08-17 01:49:00.408 [debug] <0.4995.1411> User 'FEDUSER' authenticated successfully by backend rabbit_auth_backend_internal
344 2019-08-17 01:49:00.478 [debug] <0.4853.1411> Supervisor {<0.4853.1411>,rabbit_connection_helper_sup} started rabbit_queue_collector:start_link(<<"100.96.7.0:35226 -> 100.96.3.38:5672">>) at pid <0.4488.1411>
345 2019-08-17 01:49:00.479 [debug] <0.4853.1411> Supervisor {<0.4853.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943894>, 10, #Fun<rabbit_reader.34.21003307>, {heartbeat_sender,<<"100.96.7.0:35226 -> 100.96.3.38:5672">>}) at pid <0.4113.1411>
346 2019-08-17 01:49:00.479 [debug] <0.4853.1411> Supervisor {<0.4853.1411>,rabbit_connection_helper_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943894>, 10, #Fun<rabbit_reader.35.21003307>, {heartbeat_receiver,<<"100.96.7.0:35226 -> 100.96.3.38:5672">>}) at pid <0.3889.1411>
347 2019-08-17 01:49:00.479 [debug] <0.4853.1411> Supervisor {<0.4853.1411>,rabbit_connection_helper_sup} started rabbit_channel_sup_sup:start_link() at pid <0.4587.1411>
348 2019-08-17 01:49:00.479 [info] <0.4995.1411> connection <0.4995.1411> (100.96.7.0:35226 -> 100.96.3.38:5672): user 'FEDUSER' authenticated and granted access to vhost '/'
349 2019-08-17 01:49:00.549 [debug] <0.5024.1411> Supervisor {<0.5024.1411>,rabbit_channel_sup} started rabbit_writer:start_link(#Port<0.12943894>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.4995.1411>, {<<"100.96.7.0:35226 -> 100.96.3.38:5672">>,1}, true) at pid <0.4745.1411>
350 2019-08-17 01:49:00.549 [debug] <0.3801.1411> Supervisor {<0.3801.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_sender(#Port<0.12943904>, 10, #Fun<amqp_network_connection.2.82346747>, {heartbeat_sender,<<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>}) at pid <0.3895.1411>
351 2019-08-17 01:49:00.550 [debug] <0.5024.1411> Supervisor {<0.5024.1411>,rabbit_channel_sup} started rabbit_limiter:start_link({<<"100.96.7.0:35226 -> 100.96.3.38:5672">>,1}) at pid <0.5076.1411>
352 2019-08-17 01:49:00.550 [debug] <0.3801.1411> Supervisor {<0.3801.1411>,amqp_connection_type_sup} started rabbit_heartbeat:start_heartbeat_receiver(#Port<0.12943904>, 10, #Fun<amqp_network_connection.3.82346747>, {heartbeat_receiver,<<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>}) at pid <0.3909.1411>
353 2019-08-17 01:49:00.550 [debug] <0.5024.1411> Supervisor {<0.5024.1411>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.4995.1411>, <0.4745.1411>, <0.4995.1411>, <<"100.96.7.0:35226 -> 100.96.3.38:5672">>, rabbit_framing_amqp_0_9_1, {user,<<FEDUSER>>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.4488.1411>, <0.5076.1411>) at pid <0.4417.1411>
354 2019-08-17 01:49:00.620 [debug] <0.4647.1411> Supervisor {<0.4647.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>,1}) at pid <0.3587.1411>
355 2019-08-17 01:49:00.620 [debug] <0.4647.1411> Supervisor {<0.4647.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.3423.1411>, 1, <0.3587.1411>, {<<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>,1}) at pid <0.4357.1411>
356 2019-08-17 01:49:00.620 [debug] <0.4647.1411> Supervisor {<0.4647.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943904>, 1, 131072, rabbit_framing_amqp_0_9_1, <0.4357.1411>, {<<"client 100.96.3.38:56273 -> 10.9.73.114:5672">>,1}) at pid <0.5144.1411>
357 2019-08-17 01:49:00.760 [info] <0.4995.1411> closing AMQP connection <0.4995.1411> (100.96.7.0:35226 -> 100.96.3.38:5672, vhost: '/', user: 'FEDUSER')
358 2019-08-17 01:49:01.598 [debug] <0.4455.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
359 2019-08-17 01:49:01.809 [debug] <0.4070.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
360 2019-08-17 01:49:04.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
361 2019-08-17 01:49:04.673 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
362 2019-08-17 01:49:05.194 [debug] <0.3999.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
363 2019-08-17 01:49:05.289 [debug] <0.4342.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
364 2019-08-17 01:49:06.601 [debug] <0.5319.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
365 2019-08-17 01:49:10.189 [debug] <0.5032.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
366 2019-08-17 01:49:10.287 [debug] <0.4688.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
367 2019-08-17 01:49:11.601 [debug] <0.2765.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
368 2019-08-17 01:49:11.808 [debug] <0.4705.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
369 2019-08-17 01:49:12.853 [info] <0.4176.1411> Federation exchange 'bob-request-test-1' in vhost '/' received 'basic.cancel'
370 2019-08-17 01:49:12.854 [debug] <0.4325.1411> Supervisor {<0.4325.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4383.1411>
371 2019-08-17 01:49:12.854 [debug] <0.4325.1411> Supervisor {<0.4325.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.5237.1411>, 2, <0.4383.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4876.1411>
372 2019-08-17 01:49:12.854 [debug] <0.4325.1411> Supervisor {<0.4325.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943867>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4876.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.3820.1411>
373 2019-08-17 01:49:13.324 [debug] <0.5378.1411> Supervisor {<0.5378.1411>,amqp_channel_sup} started amqp_gen_consumer:start_link(amqp_selective_consumer, [], {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4519.1411>
374 2019-08-17 01:49:13.324 [debug] <0.5378.1411> Supervisor {<0.5378.1411>,amqp_channel_sup} started amqp_channel:start_link(network, <0.5237.1411>, 2, <0.4519.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.4828.1411>
375 2019-08-17 01:49:13.324 [debug] <0.5378.1411> Supervisor {<0.5378.1411>,amqp_channel_sup} started rabbit_writer:start_link(#Port<0.12943867>, 2, 131072, rabbit_framing_amqp_0_9_1, <0.4828.1411>, {<<"client 100.96.3.38:40745 -> 10.10.17.110:5672">>,2}) at pid <0.5442.1411>
376 2019-08-17 01:49:13.833 [debug] <0.5911.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
377 2019-08-17 01:49:13.837 [debug] <0.5653.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
378 2019-08-17 01:49:13.845 [debug] <0.4912.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
379 2019-08-17 01:49:13.856 [debug] <0.5037.1411> User 'guest' authenticated successfully by backend rabbit_auth_backend_internal
380 2019-08-17 01:49:13.958 [error] <0.418.0> Supervisor {<0.418.0>,rabbit_federation_link_sup} had child {upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example.net">>],
381 <<"bob-request-test-1">>,<<"bob-request-test-1">>,1000,1,5,10000,
382 10000,false,'on-publish',none,
383 <<"bridge_solicited_stream-eu-central-1">>,false} started with rabbit_federation_exchange_link:start_link({{upstream,[<<"amqp://FEDUSER:FEDPASS@rmq-fed-debug-eu-central-1.dev.example....">>],...},...}) at <0.4176.1411> exit with reason {shutdown,restart} in context child_terminated
384 2019-08-17 01:49:14.671 [debug] <0.523.0> Peer discovery: checking for partitioned nodes to clean up.
385 2019-08-17 01:49:14.672 [debug] <0.523.0> Peer discovery: all known cluster nodes are up.
386 2019-08-17 01:49:15.192 [debug] <0.4854.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
387 2019-08-17 01:49:15.284 [debug] <0.4002.1411> User 'sysdig' authenticated successfully by backend rabbit_auth_backend_internal
388 2019-08-17 01:49:16.601 [debug] <0.3388.1411> User 'management' authenticated successfully by backend rabbit_auth_backend_internal
389 ^C
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment