config and qdstat looking for QDR load balancing issues.

a) Arch

The problem I am seeing is that the routers are not distributing the message load across other brokers on the network. Here is a little diagram:


RouterA-03 ----> ArtemisBrokerA
 |
 |
 |
 \/
RouterB-05 ----> ArtemisBrokerB


NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v 0.7.0


Connections from clients come in through an F5 VIP which forwards those connections to either host L-03-A or L-05-B. Each of those hosts have a Qpid Dispatch Router installed in front of an Artemis broker. dispatch router on L-05-B is listening on an additional port that dispatch router L-03-A connects to for inter-router communication.

If I go around my F5 VIP so i know I am connecting straight to L-05-B and I send any number of messages into the router there all of those messages end up in ArtemisBrokerB. I was hoping that some would go to BrokerB and some would go via RouterA over to BrokerA. Now when BrokerB is taken down CURRENTLY the messages sent to Router L-05-B ARE routed through Router L-03-A and then out to ArtemisBrokerB. (Currently receiving messages pull them in from both hosts so my questions only lies in message production at the moment.)

Do you agree that with this configuration you would expect messages to be load balanced between the two routes? I

b) L-03-A qdrouterd.conf

router {
    mode: interior
    id: Router.A
}

log {
    module: DEFAULT
    enable: debug+
    timestamp: yes
}

sslProfile {
    name: my-ssl
    certFile: /opt/org/my-ssl-info.pem
    keyFile: /opt/org/my-ssl-info.pem
    password: hellokitty42
}

listener {
    role: normal
    host: 0.0.0.0
    port: 5671
    authenticatePeer: no
    saslMechanisms: ANONYMOUS
    sslProfile: my-ssl
}

connector {
    name: local-artemis
    role: route-container
    host: L-03-A
    port: 61616
    saslMechanisms: ANONYMOUS
}

connector {
    name: routerb
    role: inter-router
    host: L-05-B
    port: 6671
    saslMechanisms: ANONYMOUS
}

address {
    prefix: foo.#
    waypoint: yes
    distribution: balanced
}

autoLink {
    addr: foo.bar
    dir: in
    connection: local-artemis
}

autoLink {
    addr: foo.bar
    dir: out
    connection: local-artemis
}

c) L-05-B qdrouterd.conf

router {
    mode: interior
    id: Router.B
}

log {
    module: DEFAULT
    enable: debug+
    timestamp: yes
}

sslProfile {
    name: my-ssl
    certFile: /opt/org/my-ssl-info.pem
    keyFile: /opt/org/my-ssl-info.pem
    password: hellokitty42
}

listener {
    role: normal
    host: 0.0.0.0
    port: 5671
    authenticatePeer: no
    saslMechanisms: ANONYMOUS
    sslProfile: my-ssl
}

listener {
    role: inter-router
    host: 0.0.0.0
    port: 6671
    authenticatePeer: no
    saslMechanisms: ANONYMOUS
    sslProfile: my-ssl
}

connector {
    name: local-artemis
    role: route-container
    host: L-05-B
    port: 61616
    saslMechanisms: ANONYMOUS
}

address {
    prefix: foo.#
    waypoint: yes
    distribution: balanced
}

autoLink {
    addr: foo.bar
    dir: in
    connection: local-artemis
}

autoLink {
    addr: foo.bar
    dir: out
    connection: local-artemis
}

d) L-03-A general.out

L-03-A ~]$ qdstat -b:5671 -g
Router Statistics
  attr       value
  =====================
  Mode       interior
  Area       0
  Router Id  Router.A

e) L-05-B general.out

L-05-B ~]$ qdstat -b:5671 -g
Router Statistics
  attr       value
  =====================
  Mode       interior
  Area       0
  Router Id  Router.B

f) L-03-A connections.out

L-03-A ~]$ qdstat -b:5671 -c
Connections
  Id      host             container                             role             dir  security     authentication
  ==================================================================================================================
  14      L-03-A:61616     L-03-A                                route-container  out  no-security  anonymous-user
  107132  L-05-B:6671      Router.B                              inter-router     out  no-security  anonymous-user
  107267  127.0.0.1:37916  180851ba-7ce4-44d0-a937-97de16e781c4  normal           in   no-security  anonymous-user

g) L-05-B connections.out

L-05-B ~]$ qdstat -b:5671 -c
Connections
  Id   host                container                             role             dir  security     authentication
  ==================================================================================================================
  1    L-05-B:61616        L-05-B                                route-container  out  no-security  anonymous-user
  2    10.<ip>.32:57182    Router.A                              inter-router     in   no-security  anonymous-user
  143  127.0.0.1:48610     93e6b3e3-443e-43d9-9c2c-0d648266ec19  normal           in   no-security  anonymous-user

h) L-03-A nodes.out

L-03-A ~]$ qdstat -b:5671 -n
Routers in the Network
  router-id  next-hop  link
  ===========================
  Router.A   (self)    -
  Router.B   -         0

i) L-05-B nodes.out

L-05-B ~]$ qdstat -b:5671 -n
Routers in the Network
  router-id  next-hop  link
  ===========================
  Router.A   -         0
  Router.B   (self)    -

j) L-03-A linkroutes.out

L-03-A ~]$ qdstat -b:5671 --linkroutes
Link Routes

k) L-05-B linkroutes.out

L-05-B ~]$ qdstat -b:5671 --linkroutes
Link Routes

l) L-03-A autolinks.out

L-03-A ~]$ qdstat -b:5671 --autolinks
AutoLinks
  addr     dir  phs  link  status  lastErr
  ==========================================
  foo.bar  in   1    8     active
  foo.bar  out  0    9     active

m) L-05-B autolinks.out

L-05-B ~]$ qdstat -b:5671 --autolinks
AutoLinks
  addr     dir  phs  link  status  lastErr
  ==========================================
  foo.bar  in   1    4     active
  foo.bar  out  0    5     active

n) L-03-A links.out

L-03-A ~]$ qdstat -b:5671 -l
Router Links
  type            dir  conn id  id  peer  class   addr                  phs  cap  undel  unsett  del  presett  acc  rej  rel  mod  admin    oper
  ================================================================================================================================================
  endpoint        in   14       8         mobile  foo.bar               1    250  0      0       80   0        80   0    0    0    enabled  up
  endpoint        out  14       9         mobile  foo.bar               0    250  0      0       40   0        40   0    0    0    enabled  up
  router-control  in   107132   15                                           250  0      0       524  524      0    0    0    0    enabled  up
  router-control  out  107132   16        local   qdhello                    250  0      0       527  527      0    0    0    0    enabled  up
  inter-router    in   107132   17                                           250  0      0       0    0        0    0    0    0    enabled  up
  inter-router    out  107132   18                                           250  0      0       0    0        0    0    0    0    enabled  up
  endpoint        in   107336   33        mobile  $management           0    250  0      0       1    0        1    0    0    0    enabled  up
  endpoint        out  107336   34        local   temp.liHSN39azHuUPkg       250  0      0       0    0        0    0    0    0    enabled  up

o) L-05-B links.out

L-05-B ~]$ qdstat -b:5671 -l
Router Links
  type            dir  conn id  id  peer  class   addr                  phs  cap  undel  unsett  del  presett  acc  rej  rel  mod  admin    oper
  ================================================================================================================================================
  endpoint        in   1        4         mobile  foo.bar               1    250  40     0       0    0        0    0    0    0    enabled  up
  endpoint        out  1        5         mobile  foo.bar               0    250  0      0       0    0        0    0    0    0    enabled  up
  router-control  out  2        6         local   qdhello                    250  0      0       524  524      0    0    0    0    enabled  up
  router-control  in   2        7                                            250  0      0       527  527      0    0    0    0    enabled  up
  inter-router    out  2        8                                            250  0      0       0    0        0    0    0    0    enabled  up
  inter-router    in   2        9                                            250  0      0       0    0        0    0    0    0    enabled  up
  endpoint        in   212      24        mobile  $management           0    250  0      0       1    0        1    0    0    0    enabled  up
  endpoint        out  212      25        local   temp.BJPESwMGFo+7qiy       250  0      0       0    0        0    0    0    0    enabled  up

p) L-03-A address.out

L-03-A ~]$ qdstat -b:5671 -a
Router Addresses
  class   addr                   phs  distrib    in-proc  local  remote  cntnr  in  out  thru   to-proc  from-proc
  ==================================================================================================================
  local   $_management_internal       closest    1        0      0       0      0   0    0      6        6
  local   $displayname                closest    1        0      0       0      0   0    0      0        0
  mobile  $management            0    closest    1        0      0       0      9   0    0      9        0
  local   $management                 closest    1        0      0       0      0   0    0      0        0
  router  Router.B                    closest    0        0      1       0      0   0    19     0        19
  mobile  foo.bar                1    balanced   0        0      0       0      80  520  0      0        0
  mobile  foo.bar                0    balanced   0        1      1       0      0   40   0      0        0
  local   qdhello                     flood      1        1      0       0      0   0    0      269,611  269,622
  local   qdrouter                    flood      1        0      0       0      0   0    0      12       0
  topo    qdrouter                    flood      1        0      1       0      0   0    8,992  8,991    8,996
  local   qdrouter.ma                 multicast  1        0      0       0      0   0    0      4        0
  topo    qdrouter.ma                 multicast  1        0      1       0      0   0    3      0        3
  local   temp.u6NMF1AFV0JpdBi        closest    0        1      0       0      0   0    0      0        0

q) L-05-B address.out

L-05-B ~]$ qdstat -b:5671 -a
Router Addresses
  class   addr                   phs  distrib    in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  =================================================================================================================
  local   $_management_internal       closest    1        0      0       0      0   0    0     6        6
  local   $displayname                closest    1        0      0       0      0   0    0     0        0
  mobile  $management            0    closest    1        0      0       0      9   0    0     9        0
  local   $management                 closest    1        0      0       0      0   0    0     0        0
  router  Router.A                    closest    0        0      1       0      0   0    6     0        6
  mobile  foo.bar                1    balanced   0        0      0       0      0   0    0     0        0
  mobile  foo.bar                0    balanced   0        1      1       0      0   0    0     0        0
  local   qdhello                     flood      1        1      0       0      0   0    0     570      571
  local   qdrouter                    flood      1        0      0       0      0   0    0     4        0
  topo    qdrouter                    flood      1        0      1       0      0   0    20    21       21
  local   qdrouter.ma                 multicast  1        0      0       0      0   0    0     4        0
  topo    qdrouter.ma                 multicast  1        0      1       0      0   0    0     0        1
  local   temp.xwHdS0zzku0s3lR        closest    0        1      0       0      0   0    0     0        0