jsquyres/mpirun.output

## mpirun.output
[7:35] savbu-usnic-a:~/mpi
$ mpirun --mca oob_tcp_keepalive_time 10 --mca errmgr_base_verbose 10 --mca oob_base_verbose 100 --host ivy01,ivy02 --mca btl usnic,sm,self mpi_sleep
[savbu-usnic-a:25277] mca: base: components_register: registering framework errmgr components
[savbu-usnic-a:25277] mca: base: components_register: found loaded component default_orted
[savbu-usnic-a:25277] mca: base: components_register: component default_orted register function successful
[savbu-usnic-a:25277] mca: base: components_register: found loaded component default_tool
[savbu-usnic-a:25277] mca: base: components_register: component default_tool register function successful
[savbu-usnic-a:25277] mca: base: components_register: found loaded component default_app
[savbu-usnic-a:25277] mca: base: components_register: component default_app register function successful
[savbu-usnic-a:25277] mca: base: components_register: found loaded component default_hnp
[savbu-usnic-a:25277] mca: base: components_register: component default_hnp register function successful
[savbu-usnic-a:25277] mca: base: components_open: opening errmgr components
[savbu-usnic-a:25277] mca: base: components_open: found loaded component default_orted
[savbu-usnic-a:25277] mca: base: components_open: component default_orted open function successful
[savbu-usnic-a:25277] mca: base: components_open: found loaded component default_tool
[savbu-usnic-a:25277] mca: base: components_open: component default_tool open function successful
[savbu-usnic-a:25277] mca: base: components_open: found loaded component default_app
[savbu-usnic-a:25277] mca: base: components_open: component default_app open function successful
[savbu-usnic-a:25277] mca: base: components_open: found loaded component default_hnp
[savbu-usnic-a:25277] mca: base: components_open: component default_hnp open function successful
[savbu-usnic-a:25277] mca: base: components_register: registering framework oob components
[savbu-usnic-a:25277] mca: base: components_register: found loaded component ud
[savbu-usnic-a:25277] mca: base: components_register: component ud register function successful
[savbu-usnic-a:25277] mca: base: components_register: found loaded component tcp
[savbu-usnic-a:25277] mca: base: components_register: component tcp register function successful
[savbu-usnic-a:25277] mca: base: components_register: found loaded component usock
[savbu-usnic-a:25277] mca: base: components_register: component usock register function successful
[savbu-usnic-a:25277] mca: base: components_open: opening oob components
[savbu-usnic-a:25277] mca: base: components_open: found loaded component ud
[savbu-usnic-a:25277] mca: base: components_open: component ud open function successful
[savbu-usnic-a:25277] mca: base: components_open: found loaded component tcp
[savbu-usnic-a:25277] mca: base: components_open: component tcp open function successful
[savbu-usnic-a:25277] mca: base: components_open: found loaded component usock
[savbu-usnic-a:25277] mca: base: components_open: component usock open function successful
[savbu-usnic-a:25277] mca:oob:select: checking available component ud
[savbu-usnic-a:25277] mca:oob:select: Querying component [ud]
[savbu-usnic-a:25277] oob:ud: component_available called
[savbu-usnic-a:25277] mca:oob:select: Skipping component [ud] - failed to startup
[savbu-usnic-a:25277] mca:oob:select: checking available component tcp
[savbu-usnic-a:25277] mca:oob:select: Querying component [tcp]
[savbu-usnic-a:25277] oob:tcp: component_available called
[savbu-usnic-a:25277] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init rejecting loopback interface lo
[savbu-usnic-a:25277] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.0.8.254 to our list of V4 connections
[savbu-usnic-a:25277] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.193.184.48 to our list of V4 connections
[savbu-usnic-a:25277] WORKING INTERFACE 4 KERNEL INDEX 4 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.193.184.49 to our list of V4 connections
[savbu-usnic-a:25277] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.10.0.254 to our list of V4 connections
[savbu-usnic-a:25277] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.2.0.254 to our list of V4 connections
[savbu-usnic-a:25277] WORKING INTERFACE 7 KERNEL INDEX 8 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.3.0.254 to our list of V4 connections
[savbu-usnic-a:25277] WORKING INTERFACE 8 KERNEL INDEX 8 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.3.0.252 to our list of V4 connections
[savbu-usnic-a:25277] WORKING INTERFACE 9 KERNEL INDEX 9 FAMILY: V4
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:init adding 10.50.0.254 to our list of V4 connections
[savbu-usnic-a:25277] [[13243,0],0] TCP STARTUP
[savbu-usnic-a:25277] [[13243,0],0] attempting to bind to IPv4 port 0
[savbu-usnic-a:25277] [[13243,0],0] assigned IPv4 port 45024
[savbu-usnic-a:25277] mca:oob:select: Adding component to end
[savbu-usnic-a:25277] mca:oob:select: checking available component usock
[savbu-usnic-a:25277] mca:oob:select: Querying component [usock]
[savbu-usnic-a:25277] oob:usock: component_available called
[savbu-usnic-a:25277] [[13243,0],0] USOCK STARTUP
[savbu-usnic-a:25277] SUNPATH: /tmp/openmpi-sessions-182726@savbu-usnic-a_0/13243/0/usock
[savbu-usnic-a:25277] [[13243,0],0] START USOCK LISTENING ON /tmp/openmpi-sessions-182726@savbu-usnic-a_0/13243/0/usock
[savbu-usnic-a:25277] mca:oob:select: Inserting component
[savbu-usnic-a:25277] mca:oob:select: Found 2 active transports
[savbu-usnic-a:25277] mca:base:select: Auto-selecting errmgr components
[savbu-usnic-a:25277] mca:base:select:(errmgr) Querying component [default_orted]
[savbu-usnic-a:25277] mca:base:select:(errmgr) Skipping component [default_orted]. Query failed to return a module
[savbu-usnic-a:25277] mca:base:select:(errmgr) Querying component [default_tool]
[savbu-usnic-a:25277] mca:base:select:(errmgr) Skipping component [default_tool]. Query failed to return a module
[savbu-usnic-a:25277] mca:base:select:(errmgr) Querying component [default_app]
[savbu-usnic-a:25277] mca:base:select:(errmgr) Skipping component [default_app]. Query failed to return a module
[savbu-usnic-a:25277] mca:base:select:(errmgr) Querying component [default_hnp]
[savbu-usnic-a:25277] mca:base:select:(errmgr) Query of component [default_hnp] set priority to 1000
[savbu-usnic-a:25277] mca:base:select:(errmgr) Selected component [default_hnp]
[savbu-usnic-a:25277] mca: base: close: component default_orted closed
[savbu-usnic-a:25277] mca: base: close: unloading component default_orted
[savbu-usnic-a:25277] mca: base: close: component default_tool closed
[savbu-usnic-a:25277] mca: base: close: unloading component default_tool
[savbu-usnic-a:25277] mca: base: close: component default_app closed
[savbu-usnic-a:25277] mca: base: close: unloading component default_app
[ivy02:05706] mca: base: components_register: registering framework errmgr components
[ivy02:05706] mca: base: components_register: found loaded component default_orted
[ivy02:05706] mca: base: components_register: component default_orted register function successful
[ivy02:05706] mca: base: components_register: found loaded component default_tool
[ivy02:05706] mca: base: components_register: component default_tool register function successful
[ivy02:05706] mca: base: components_register: found loaded component default_app
[ivy02:05706] mca: base: components_register: component default_app register function successful
[ivy02:05706] mca: base: components_register: found loaded component default_hnp
[ivy02:05706] mca: base: components_register: component default_hnp register function successful
[ivy02:05706] mca: base: components_open: opening errmgr components
[ivy02:05706] mca: base: components_open: found loaded component default_orted
[ivy02:05706] mca: base: components_open: component default_orted open function successful
[ivy02:05706] mca: base: components_open: found loaded component default_tool
[ivy02:05706] mca: base: components_open: component default_tool open function successful
[ivy02:05706] mca: base: components_open: found loaded component default_app
[ivy02:05706] mca: base: components_open: component default_app open function successful
[ivy02:05706] mca: base: components_open: found loaded component default_hnp
[ivy02:05706] mca: base: components_open: component default_hnp open function successful
[ivy02:05706] mca: base: components_register: registering framework oob components
[ivy02:05706] mca: base: components_register: found loaded component ud
[ivy02:05706] mca: base: components_register: component ud register function successful
[ivy02:05706] mca: base: components_register: found loaded component tcp
[ivy02:05706] mca: base: components_register: component tcp register function successful
[ivy02:05706] mca: base: components_register: found loaded component usock
[ivy02:05706] mca: base: components_register: component usock register function successful
[ivy02:05706] mca: base: components_open: opening oob components
[ivy02:05706] mca: base: components_open: found loaded component ud
[ivy02:05706] mca: base: components_open: component ud open function successful
[ivy02:05706] mca: base: components_open: found loaded component tcp
[ivy02:05706] mca: base: components_open: component tcp open function successful
[ivy02:05706] mca: base: components_open: found loaded component usock
[ivy02:05706] mca: base: components_open: component usock open function successful
[ivy02:05706] mca:oob:select: checking available component ud
[ivy02:05706] mca:oob:select: Querying component [ud]
[ivy02:05706] oob:ud: component_available called
[ivy02:05706] [[13243,0],2] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy02:05706] [[13243,0],2] oob:ud:device_setup error opening device. errno = 2
[ivy02:05706] [[13243,0],2] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy02:05706] [[13243,0],2] oob:ud:device_setup error opening device. errno = 2
[ivy02:05706] [[13243,0],2] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy02:05706] [[13243,0],2] oob:ud:device_setup error opening device. errno = 2
[ivy02:05706] [[13243,0],2] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy02:05706] [[13243,0],2] oob:ud:device_setup error opening device. errno = 2
[ivy02:05706] mca:oob:select: Skipping component [ud] - failed to startup
[ivy02:05706] mca:oob:select: checking available component tcp
[ivy02:05706] mca:oob:select: Querying component [tcp]
[ivy02:05706] oob:tcp: component_available called
[ivy02:05706] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[ivy02:05706] [[13243,0],2] oob:tcp:init rejecting loopback interface lo
[ivy02:05706] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[ivy02:05706] [[13243,0],2] oob:tcp:init adding 10.0.19.202 to our list of V4 connections
[ivy02:05706] WORKING INTERFACE 3 KERNEL INDEX 6 FAMILY: V4
[ivy02:05706] [[13243,0],2] oob:tcp:init adding 10.5.10.182 to our list of V4 connections
[ivy02:05706] WORKING INTERFACE 4 KERNEL INDEX 7 FAMILY: V4
[ivy02:05706] [[13243,0],2] oob:tcp:init adding 10.101.1.2 to our list of V4 connections
[ivy02:05706] WORKING INTERFACE 5 KERNEL INDEX 8 FAMILY: V4
[ivy02:05706] [[13243,0],2] oob:tcp:init adding 10.3.0.182 to our list of V4 connections
[ivy02:05706] WORKING INTERFACE 6 KERNEL INDEX 9 FAMILY: V4
[ivy02:05706] [[13243,0],2] oob:tcp:init adding 10.8.10.182 to our list of V4 connections
[ivy02:05706] [[13243,0],2] TCP STARTUP
[ivy02:05706] [[13243,0],2] attempting to bind to IPv4 port 0
[ivy02:05706] [[13243,0],2] assigned IPv4 port 36928
[ivy02:05706] mca:oob:select: Adding component to end
[ivy02:05706] mca:oob:select: checking available component usock
[ivy02:05706] mca:oob:select: Querying component [usock]
[ivy02:05706] oob:usock: component_available called
[ivy02:05706] [[13243,0],2] USOCK STARTUP
[ivy02:05706] SUNPATH: /tmp/openmpi-sessions-182726@ivy02_0/13243/0/usock
[ivy02:05706] [[13243,0],2] START USOCK LISTENING ON /tmp/openmpi-sessions-182726@ivy02_0/13243/0/usock
[ivy02:05706] mca:oob:select: Inserting component
[ivy02:05706] mca:oob:select: Found 2 active transports
[ivy02:05706] mca:base:select: Auto-selecting errmgr components
[ivy02:05706] mca:base:select:(errmgr) Querying component [default_orted]
[ivy02:05706] mca:base:select:(errmgr) Query of component [default_orted] set priority to 1000
[ivy02:05706] mca:base:select:(errmgr) Querying component [default_tool]
[ivy02:05706] mca:base:select:(errmgr) Skipping component [default_tool]. Query failed to return a module
[ivy02:05706] mca:base:select:(errmgr) Querying component [default_app]
[ivy02:05706] mca:base:select:(errmgr) Skipping component [default_app]. Query failed to return a module
[ivy02:05706] mca:base:select:(errmgr) Querying component [default_hnp]
[ivy02:05706] mca:base:select:(errmgr) Skipping component [default_hnp]. Query failed to return a module
[ivy02:05706] mca:base:select:(errmgr) Selected component [default_orted]
[ivy02:05706] mca: base: close: component default_tool closed
[ivy02:05706] mca: base: close: unloading component default_tool
[ivy02:05706] mca: base: close: component default_app closed
[ivy02:05706] mca: base: close: unloading component default_app
[ivy02:05706] mca: base: close: component default_hnp closed
[ivy02:05706] mca: base: close: unloading component default_hnp
[ivy02:05706] [[13243,0],2]: set_addr to uri 867893248.0;usock;tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy02:05706] [[13243,0],2]:set_addr checking if peer [[13243,0],0] is reachable via component usock
[ivy02:05706] [[13243,0],2]: peer [[13243,0],0] is NOT reachable via component usock
[ivy02:05706] [[13243,0],2]:set_addr checking if peer [[13243,0],0] is reachable via component tcp
[ivy02:05706] [[13243,0],2] oob:tcp: ignoring address usock
[ivy02:05706] [[13243,0],2] oob:tcp: working peer [[13243,0],0] address tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.0.8.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.193.184.48 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.193.184.49 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.10.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.2.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.3.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.3.0.252 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.50.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]: peer [[13243,0],0] is reachable via component tcp
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] SET_PEER ADDING PEER [[13243,0],0]
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.0.8.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.193.184.48 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.193.184.49 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.10.0.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.2.0.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.3.0.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.3.0.252 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.50.0.254 port 45024
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:10 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:10 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:438] queue pending to [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_nb: initiating connection to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:452] connect to [[13243,0],0]
[ivy02:05706] [[13243,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:peer creating socket to [[13243,0],0]
[ivy02:05706] [[13243,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],0] on socket 11
[ivy02:05706] [[13243,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],0] on 10.0.8.254:45024 - 0 retries
[ivy02:05706] [[13243,0],2] waiting for connect completion to [[13243,0],0] - activating send event
[savbu-usnic-a:25277] [[13243,0],0] mca_oob_tcp_listen_thread: new connection: (16, 0) 10.0.19.202:51879
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler CONNECTING
[ivy02:05706] [[13243,0],2]:tcp:complete_connect called for peer [[13243,0],0] on socket 11
[ivy02:05706] [[13243,0],2] tcp_peer_complete_connect: sending ack to [[13243,0],0]
[ivy02:05706] [[13243,0],2] SEND CONNECT ACK
[ivy02:05706] [[13243,0],2] SENDING CREDENTIAL OF SIZE 210
[ivy02:05706] [[13243,0],2] send blocking of 254 bytes to socket 11
[ivy02:05706] [[13243,0],2] blocking send complete to socket 11
[ivy02:05706] [[13243,0],2] tcp_peer_complete_connect: setting read event on connection to [[13243,0],0]
[savbu-usnic-a:25277] [[13243,0],0] connection_handler: working connection (16, 0) 10.0.19.202:51879
[savbu-usnic-a:25277] [[13243,0],0] accept_connection: 10.0.19.202:51879
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called
[savbu-usnic-a:25277] [[13243,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 16
[savbu-usnic-a:25277] [[13243,0],0] waiting for connect ack from UNKNOWN
[savbu-usnic-a:25277] [[13243,0],0] connect ack received from UNKNOWN
[savbu-usnic-a:25277] [[13243,0],0] connect-ack recvd from UNKNOWN
[savbu-usnic-a:25277] [[13243,0],0] mca_oob_tcp_recv_connect: connection from new peer
[savbu-usnic-a:25277] [[13243,0],0] connect-ack header from [[13243,0],2] is okay
[savbu-usnic-a:25277] [[13243,0],0] waiting for connect ack from [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] connect ack received from [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] connect-ack version from [[13243,0],2] matches ours
[savbu-usnic-a:25277] [[13243,0],0] connect-ack [[13243,0],2] authenticated
[savbu-usnic-a:25277] [[13243,0],0] tcp:peer_accept called for peer [[13243,0],2] in state UNKNOWN on socket 16
[savbu-usnic-a:25277] [[13243,0],0] SEND CONNECT ACK
[savbu-usnic-a:25277] [[13243,0],0] SENDING CREDENTIAL OF SIZE 165
[savbu-usnic-a:25277] [[13243,0],0] send blocking of 209 bytes to socket 16
[savbu-usnic-a:25277] [[13243,0],0] blocking send complete to socket 16
[savbu-usnic-a:25277] [[13243,0],0]-[[13243,0],2] tcp_peer_connected on socket 16
[savbu-usnic-a:25277] [[13243,0],0]-[[13243,0],2] accepted: 10.0.8.254 - 10.0.19.202 nodelay 1 sndbuf 249856 rcvbuf 262144 flags 00000802
[savbu-usnic-a:25277] [[13243,0],0] tcp:set_module called for peer [[13243,0],2]
[ivy02:05706] [[13243,0],2]:tcp:recv:handler called for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] RECV CONNECT ACK FROM [[13243,0],0] ON SOCKET 11
[ivy02:05706] [[13243,0],2] waiting for connect ack from [[13243,0],0]
[ivy02:05706] [[13243,0],2] connect ack received from [[13243,0],0]
[ivy02:05706] [[13243,0],2] connect-ack recvd from [[13243,0],0]
[ivy02:05706] [[13243,0],2] connect-ack header from [[13243,0],0] is okay
[ivy02:05706] [[13243,0],2] waiting for connect ack from [[13243,0],0]
[ivy02:05706] [[13243,0],2] connect ack received from [[13243,0],0]
[ivy02:05706] [[13243,0],2] connect-ack version from [[13243,0],0] matches ours
[ivy02:05706] [[13243,0],2] connect-ack [[13243,0],0] authenticated
[ivy02:05706] [[13243,0],2]-[[13243,0],0] tcp_peer_connected on socket 11
[ivy02:05706] [[13243,0],2]-[[13243,0],0] connected: 10.0.19.202 - 10.0.8.254 nodelay 1 sndbuf 262144 rcvbuf 262144 flags 00000802
[ivy02:05706] [[13243,0],2]:tcp:recv:handler starting send/recv events
[ivy02:05706] [[13243,0],2] tcp:set_module called for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 54999 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 54999
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 54999 BYTES FOR DEST [[13243,0],0] TAG 10
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 10 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]: set_addr to uri 867893248.2;usock;tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],2] is reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],2] is NOT reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],2] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: ignoring address usock
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: working peer [[13243,0],2] address tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.0.19.202 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.5.10.182 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.101.1.2 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.3.0.182 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.8.10.182 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],2] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0] OOB_SEND: rml_oob_send.c:210
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.0.19.202 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.5.10.182 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.101.1.2 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.3.0.182 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.8.10.182 port 36928
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send to target [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send known transport for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:send_nb to peer [[13243,0],2]:1 to channel=-1 seq = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb to peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:467] post send to [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:404] processing send to peer [[13243,0],2]:1 to channel =-1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb: already connected to [[13243,0],2] - queueing for send
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:431] queue send to [[13243,0],2] channel =-1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler called to send to peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler SENDING TO [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] MESSAGE SEND COMPLETE TO [[13243,0],2] OF 203 BYTES ON SOCKET 16
[ivy02:05706] [[13243,0],2]:tcp:recv:handler called for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:tcp:recv:handler CONNECTED
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate new recv msg
[ivy02:05706] [[13243,0],2]:tcp:recv:handler read hdr
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate data region of size 203
[ivy02:05706] [[13243,0],2] RECVD COMPLETE MESSAGE FROM [[13243,0],0] (ORIGIN [[13243,0],0]) OF 203 BYTES FOR DEST [[13243,0],2] TAG 1
[ivy02:05706] [[13243,0],2] DELIVERING TO RML tag = 1 channel = -1 seq_num = -1
[ivy01:05396] mca: base: components_register: registering framework errmgr components
[ivy01:05396] mca: base: components_register: found loaded component default_orted
[ivy01:05396] mca: base: components_register: component default_orted register function successful
[ivy01:05396] mca: base: components_register: found loaded component default_tool
[ivy01:05396] mca: base: components_register: component default_tool register function successful
[ivy01:05396] mca: base: components_register: found loaded component default_app
[ivy01:05396] mca: base: components_register: component default_app register function successful
[ivy01:05396] mca: base: components_register: found loaded component default_hnp
[ivy01:05396] mca: base: components_register: component default_hnp register function successful
[ivy01:05396] mca: base: components_open: opening errmgr components
[ivy01:05396] mca: base: components_open: found loaded component default_orted
[ivy01:05396] mca: base: components_open: component default_orted open function successful
[ivy01:05396] mca: base: components_open: found loaded component default_tool
[ivy01:05396] mca: base: components_open: component default_tool open function successful
[ivy01:05396] mca: base: components_open: found loaded component default_app
[ivy01:05396] mca: base: components_open: component default_app open function successful
[ivy01:05396] mca: base: components_open: found loaded component default_hnp
[ivy01:05396] mca: base: components_open: component default_hnp open function successful
[ivy01:05396] mca: base: components_register: registering framework oob components
[ivy01:05396] mca: base: components_register: found loaded component ud
[ivy01:05396] mca: base: components_register: component ud register function successful
[ivy01:05396] mca: base: components_register: found loaded component tcp
[ivy01:05396] mca: base: components_register: component tcp register function successful
[ivy01:05396] mca: base: components_register: found loaded component usock
[ivy01:05396] mca: base: components_register: component usock register function successful
[ivy01:05396] mca: base: components_open: opening oob components
[ivy01:05396] mca: base: components_open: found loaded component ud
[ivy01:05396] mca: base: components_open: component ud open function successful
[ivy01:05396] mca: base: components_open: found loaded component tcp
[ivy01:05396] mca: base: components_open: component tcp open function successful
[ivy01:05396] mca: base: components_open: found loaded component usock
[ivy01:05396] mca: base: components_open: component usock open function successful
[ivy01:05396] mca:oob:select: checking available component ud
[ivy01:05396] mca:oob:select: Querying component [ud]
[ivy01:05396] oob:ud: component_available called
[ivy01:05396] [[13243,0],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy01:05396] [[13243,0],1] oob:ud:device_setup error opening device. errno = 2
[ivy01:05396] [[13243,0],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy01:05396] [[13243,0],1] oob:ud:device_setup error opening device. errno = 2
[ivy01:05396] [[13243,0],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy01:05396] [[13243,0],1] oob:ud:device_setup error opening device. errno = 2
[ivy01:05396] [[13243,0],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae5c1440
[ivy01:05396] [[13243,0],1] oob:ud:device_setup error opening device. errno = 2
[ivy01:05396] mca:oob:select: Skipping component [ud] - failed to startup
[ivy01:05396] mca:oob:select: checking available component tcp
[ivy01:05396] mca:oob:select: Querying component [tcp]
[ivy01:05396] oob:tcp: component_available called
[ivy01:05396] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[ivy01:05396] [[13243,0],1] oob:tcp:init rejecting loopback interface lo
[ivy01:05396] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[ivy01:05396] [[13243,0],1] oob:tcp:init adding 10.0.20.187 to our list of V4 connections
[ivy01:05396] WORKING INTERFACE 3 KERNEL INDEX 6 FAMILY: V4
[ivy01:05396] [[13243,0],1] oob:tcp:init adding 10.5.10.181 to our list of V4 connections
[ivy01:05396] WORKING INTERFACE 4 KERNEL INDEX 7 FAMILY: V4
[ivy01:05396] [[13243,0],1] oob:tcp:init adding 10.101.1.1 to our list of V4 connections
[ivy01:05396] WORKING INTERFACE 5 KERNEL INDEX 8 FAMILY: V4
[ivy01:05396] [[13243,0],1] oob:tcp:init adding 10.3.0.181 to our list of V4 connections
[ivy01:05396] WORKING INTERFACE 6 KERNEL INDEX 9 FAMILY: V4
[ivy01:05396] [[13243,0],1] oob:tcp:init adding 10.8.10.181 to our list of V4 connections
[ivy01:05396] [[13243,0],1] TCP STARTUP
[ivy01:05396] [[13243,0],1] attempting to bind to IPv4 port 0
[ivy01:05396] [[13243,0],1] assigned IPv4 port 52741
[ivy01:05396] mca:oob:select: Adding component to end
[ivy01:05396] mca:oob:select: checking available component usock
[ivy01:05396] mca:oob:select: Querying component [usock]
[ivy01:05396] oob:usock: component_available called
[ivy01:05396] [[13243,0],1] USOCK STARTUP
[ivy01:05396] SUNPATH: /tmp/openmpi-sessions-182726@ivy01_0/13243/0/usock
[ivy01:05396] [[13243,0],1] START USOCK LISTENING ON /tmp/openmpi-sessions-182726@ivy01_0/13243/0/usock
[ivy01:05396] mca:oob:select: Inserting component
[ivy01:05396] mca:oob:select: Found 2 active transports
[ivy01:05396] mca:base:select: Auto-selecting errmgr components
[ivy01:05396] mca:base:select:(errmgr) Querying component [default_orted]
[ivy01:05396] mca:base:select:(errmgr) Query of component [default_orted] set priority to 1000
[ivy01:05396] mca:base:select:(errmgr) Querying component [default_tool]
[ivy01:05396] mca:base:select:(errmgr) Skipping component [default_tool]. Query failed to return a module
[ivy01:05396] mca:base:select:(errmgr) Querying component [default_app]
[ivy01:05396] mca:base:select:(errmgr) Skipping component [default_app]. Query failed to return a module
[ivy01:05396] mca:base:select:(errmgr) Querying component [default_hnp]
[ivy01:05396] mca:base:select:(errmgr) Skipping component [default_hnp]. Query failed to return a module
[ivy01:05396] mca:base:select:(errmgr) Selected component [default_orted]
[ivy01:05396] mca: base: close: component default_tool closed
[ivy01:05396] mca: base: close: unloading component default_tool
[ivy01:05396] mca: base: close: component default_app closed
[ivy01:05396] mca: base: close: unloading component default_app
[ivy01:05396] mca: base: close: component default_hnp closed
[ivy01:05396] mca: base: close: unloading component default_hnp
[ivy01:05396] [[13243,0],1]: set_addr to uri 867893248.0;usock;tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy01:05396] [[13243,0],1]:set_addr checking if peer [[13243,0],0] is reachable via component usock
[ivy01:05396] [[13243,0],1]: peer [[13243,0],0] is NOT reachable via component usock
[ivy01:05396] [[13243,0],1]:set_addr checking if peer [[13243,0],0] is reachable via component tcp
[ivy01:05396] [[13243,0],1] oob:tcp: ignoring address usock
[ivy01:05396] [[13243,0],1] oob:tcp: working peer [[13243,0],0] address tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.0.8.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.193.184.48 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.193.184.49 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.10.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.2.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.3.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.3.0.252 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.50.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]: peer [[13243,0],0] is reachable via component tcp
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] SET_PEER ADDING PEER [[13243,0],0]
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.0.8.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.193.184.48 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.193.184.49 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.10.0.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.2.0.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.3.0.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.3.0.252 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.50.0.254 port 45024
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:10 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:10 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:438] queue pending to [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_nb: initiating connection to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:452] connect to [[13243,0],0]
[ivy01:05396] [[13243,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:peer creating socket to [[13243,0],0]
[ivy01:05396] [[13243,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],0] on socket 11
[ivy01:05396] [[13243,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],0] on 10.0.8.254:45024 - 0 retries
[ivy01:05396] [[13243,0],1] waiting for connect completion to [[13243,0],0] - activating send event
[savbu-usnic-a:25277] [[13243,0],0] mca_oob_tcp_listen_thread: new connection: (17, 11) 10.0.20.187:34026
[savbu-usnic-a:25277] [[13243,0],0] connection_handler: working connection (17, 0) 10.0.20.187:34026
[savbu-usnic-a:25277] [[13243,0],0] accept_connection: 10.0.20.187:34026
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called
[savbu-usnic-a:25277] [[13243,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 17
[savbu-usnic-a:25277] [[13243,0],0] waiting for connect ack from UNKNOWN
[savbu-usnic-a:25277] [[13243,0],0] connect ack received from UNKNOWN
[savbu-usnic-a:25277] [[13243,0],0] connect-ack recvd from UNKNOWN
[savbu-usnic-a:25277] [[13243,0],0] mca_oob_tcp_recv_connect: connection from new peer
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler CONNECTING
[ivy01:05396] [[13243,0],1]:tcp:complete_connect called for peer [[13243,0],0] on socket 11
[ivy01:05396] [[13243,0],1] tcp_peer_complete_connect: sending ack to [[13243,0],0]
[ivy01:05396] [[13243,0],1] SEND CONNECT ACK
[ivy01:05396] [[13243,0],1] SENDING CREDENTIAL OF SIZE 210
[ivy01:05396] [[13243,0],1] send blocking of 254 bytes to socket 11
[ivy01:05396] [[13243,0],1] blocking send complete to socket 11
[ivy01:05396] [[13243,0],1] tcp_peer_complete_connect: setting read event on connection to [[13243,0],0]
[savbu-usnic-a:25277] [[13243,0],0] connect-ack header from [[13243,0],1] is okay
[savbu-usnic-a:25277] [[13243,0],0] waiting for connect ack from [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] connect ack received from [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] connect-ack version from [[13243,0],1] matches ours
[savbu-usnic-a:25277] [[13243,0],0] connect-ack [[13243,0],1] authenticated
[savbu-usnic-a:25277] [[13243,0],0] tcp:peer_accept called for peer [[13243,0],1] in state UNKNOWN on socket 17
[savbu-usnic-a:25277] [[13243,0],0] SEND CONNECT ACK
[savbu-usnic-a:25277] [[13243,0],0] SENDING CREDENTIAL OF SIZE 165
[savbu-usnic-a:25277] [[13243,0],0] send blocking of 209 bytes to socket 17
[savbu-usnic-a:25277] [[13243,0],0] blocking send complete to socket 17
[savbu-usnic-a:25277] [[13243,0],0]-[[13243,0],1] tcp_peer_connected on socket 17
[savbu-usnic-a:25277] [[13243,0],0]-[[13243,0],1] accepted: 10.0.8.254 - 10.0.20.187 nodelay 1 sndbuf 249856 rcvbuf 262144 flags 00000802
[savbu-usnic-a:25277] [[13243,0],0] tcp:set_module called for peer [[13243,0],1]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] RECV CONNECT ACK FROM [[13243,0],0] ON SOCKET 11
[ivy01:05396] [[13243,0],1] waiting for connect ack from [[13243,0],0]
[ivy01:05396] [[13243,0],1] connect ack received from [[13243,0],0]
[ivy01:05396] [[13243,0],1] connect-ack recvd from [[13243,0],0]
[ivy01:05396] [[13243,0],1] connect-ack header from [[13243,0],0] is okay
[ivy01:05396] [[13243,0],1] waiting for connect ack from [[13243,0],0]
[ivy01:05396] [[13243,0],1] connect ack received from [[13243,0],0]
[ivy01:05396] [[13243,0],1] connect-ack version from [[13243,0],0] matches ours
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 54999
[ivy01:05396] [[13243,0],1] connect-ack [[13243,0],0] authenticated
[ivy01:05396] [[13243,0],1]-[[13243,0],0] tcp_peer_connected on socket 11
[ivy01:05396] [[13243,0],1]-[[13243,0],0] connected: 10.0.20.187 - 10.0.8.254 nodelay 1 sndbuf 262144 rcvbuf 262144 flags 00000802
[ivy01:05396] [[13243,0],1]:tcp:recv:handler starting send/recv events
[ivy01:05396] [[13243,0],1] tcp:set_module called for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 54999 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 54999 BYTES FOR DEST [[13243,0],0] TAG 10
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 10 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]: set_addr to uri 867893248.1;usock;tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],1] is reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],1] is NOT reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],1] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: ignoring address usock
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: working peer [[13243,0],1] address tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.0.20.187 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.5.10.181 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.101.1.1 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.3.0.181 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.8.10.181 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],1] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0] OOB_SEND: rml_oob_send.c:210
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.0.20.187 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.5.10.181 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.101.1.1 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.3.0.181 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.8.10.181 port 52741
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send to target [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send known transport for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:send_nb to peer [[13243,0],1]:1 to channel=-1 seq = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb to peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:467] post send to [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:404] processing send to peer [[13243,0],1]:1 to channel =-1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb: already connected to [[13243,0],1] - queueing for send
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:431] queue send to [[13243,0],1] channel =-1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler called to send to peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler SENDING TO [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] MESSAGE SEND COMPLETE TO [[13243,0],1] OF 203 BYTES ON SOCKET 17
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate data region of size 203
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1] RECVD COMPLETE MESSAGE FROM [[13243,0],0] (ORIGIN [[13243,0],0]) OF 203 BYTES FOR DEST [[13243,0],1] TAG 1
[ivy01:05396] [[13243,0],1] DELIVERING TO RML tag = 1 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]: set_addr to uri 867893248.0;usock;tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[savbu-usnic-a:25277] [[13243,0],0]:set_addr peer [[13243,0],0] is me
[savbu-usnic-a:25277] [[13243,0],0]: set_addr to uri 867893248.1;usock;tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],1] is reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],1] is NOT reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],1] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: ignoring address usock
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: working peer [[13243,0],1] address tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.0.20.187 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.5.10.181 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.101.1.1 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.3.0.181 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.8.10.181 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],1] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0]: set_addr to uri 867893248.2;usock;tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],2] is reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],2] is NOT reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]:set_addr checking if peer [[13243,0],2] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: ignoring address usock
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp: working peer [[13243,0],2] address tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.0.19.202 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.5.10.182 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.101.1.2 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.3.0.182 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] PASSING ADDR 10.8.10.182 TO MODULE
[savbu-usnic-a:25277] [[13243,0],0]:tcp set addr for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]: peer [[13243,0],2] is reachable via component tcp
[savbu-usnic-a:25277] [[13243,0],0] OOB_SEND: rml_oob_send.c:210
[savbu-usnic-a:25277] [[13243,0],0] OOB_SEND: rml_oob_send.c:210
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.0.20.187 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.5.10.181 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.101.1.1 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.3.0.181 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],1] is listening on net 10.8.10.181 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.0.19.202 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.5.10.182 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.101.1.2 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.3.0.182 port 36928
[savbu-usnic-a:25277] [[13243,0],0]:tcp:processing set_peer cmd
[savbu-usnic-a:25277] [[13243,0],0] set_peer: peer [[13243,0],2] is listening on net 10.8.10.182 port 36928
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send to target [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send known transport for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:send_nb to peer [[13243,0],1]:15 to channel=-1 seq = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb to peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:467] post send to [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send to target [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send known transport for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:send_nb to peer [[13243,0],2]:15 to channel=-1 seq = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb to peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:467] post send to [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:404] processing send to peer [[13243,0],1]:15 to channel =-1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb: already connected to [[13243,0],1] - queueing for send
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:431] queue send to [[13243,0],1] channel =-1
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:404] processing send to peer [[13243,0],2]:15 to channel =-1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb: already connected to [[13243,0],2] - queueing for send
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:431] queue send to [[13243,0],2] channel =-1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler called to send to peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler SENDING TO [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] MESSAGE SEND COMPLETE TO [[13243,0],2] OF 1440 BYTES ON SOCKET 16
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler called to send to peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler SENDING TO [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] MESSAGE SEND COMPLETE TO [[13243,0],1] OF 1440 BYTES ON SOCKET 17
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate data region of size 1440
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1] RECVD COMPLETE MESSAGE FROM [[13243,0],0] (ORIGIN [[13243,0],0]) OF 1440 BYTES FOR DEST [[13243,0],1] TAG 15
[ivy01:05396] [[13243,0],1] DELIVERING TO RML tag = 15 channel = -1 seq_num = -1
[ivy02:05706] [[13243,0],2]:tcp:recv:handler called for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:tcp:recv:handler CONNECTED
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate new recv msg
[ivy02:05706] [[13243,0],2]:tcp:recv:handler read hdr
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate data region of size 1440
[ivy02:05706] [[13243,0],2] RECVD COMPLETE MESSAGE FROM [[13243,0],0] (ORIGIN [[13243,0],0]) OF 1440 BYTES FOR DEST [[13243,0],2] TAG 15
[ivy02:05706] [[13243,0],2] DELIVERING TO RML tag = 15 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1]: set_addr to uri 867893248.0;usock;tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy01:05396] [[13243,0],1]:set_addr checking if peer [[13243,0],0] is reachable via component usock
[ivy01:05396] [[13243,0],1]: peer [[13243,0],0] is NOT reachable via component usock
[ivy01:05396] [[13243,0],1]:set_addr checking if peer [[13243,0],0] is reachable via component tcp
[ivy01:05396] [[13243,0],1] oob:tcp: ignoring address usock
[ivy01:05396] [[13243,0],1] oob:tcp: working peer [[13243,0],0] address tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.0.8.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.193.184.48 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.193.184.49 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.10.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.2.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.3.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.3.0.252 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.50.0.254 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]: peer [[13243,0],0] is reachable via component tcp
[ivy01:05396] [[13243,0],1]: set_addr to uri 867893248.1;usock;tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[ivy01:05396] [[13243,0],1]:set_addr peer [[13243,0],1] is me
[ivy02:05706] [[13243,0],2]: set_addr to uri 867893248.0;usock;tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy02:05706] [[13243,0],2]:set_addr checking if peer [[13243,0],0] is reachable via component usock
[ivy02:05706] [[13243,0],2]: peer [[13243,0],0] is NOT reachable via component usock
[ivy02:05706] [[13243,0],2]:set_addr checking if peer [[13243,0],0] is reachable via component tcp
[ivy02:05706] [[13243,0],2] oob:tcp: ignoring address usock
[ivy02:05706] [[13243,0],2] oob:tcp: working peer [[13243,0],0] address tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:45024
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.0.8.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.193.184.48 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.193.184.49 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.10.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.2.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.3.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.3.0.252 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.50.0.254 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]: peer [[13243,0],0] is reachable via component tcp
[ivy02:05706] [[13243,0],2]: set_addr to uri 867893248.1;usock;tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[ivy02:05706] [[13243,0],2]:set_addr checking if peer [[13243,0],1] is reachable via component usock
[ivy02:05706] [[13243,0],2]: peer [[13243,0],1] is NOT reachable via component usock
[ivy02:05706] [[13243,0],2]:set_addr checking if peer [[13243,0],1] is reachable via component tcp
[ivy02:05706] [[13243,0],2] oob:tcp: ignoring address usock
[ivy02:05706] [[13243,0],2] oob:tcp: working peer [[13243,0],1] address tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.0.20.187 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.5.10.181 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.101.1.1 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.3.0.181 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] PASSING ADDR 10.8.10.181 TO MODULE
[ivy02:05706] [[13243,0],2]:tcp set addr for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2]: peer [[13243,0],1] is reachable via component tcp
[ivy02:05706] [[13243,0],2]: set_addr to uri 867893248.2;usock;tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[ivy02:05706] [[13243,0],2]:set_addr peer [[13243,0],2] is me
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.0.8.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1]: set_addr to uri 867893248.2;usock;tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[ivy01:05396] [[13243,0],1]:set_addr checking if peer [[13243,0],2] is reachable via component usock
[ivy01:05396] [[13243,0],1]: peer [[13243,0],2] is NOT reachable via component usock
[ivy01:05396] [[13243,0],1]:set_addr checking if peer [[13243,0],2] is reachable via component tcp
[ivy01:05396] [[13243,0],1] oob:tcp: ignoring address usock
[ivy01:05396] [[13243,0],1] oob:tcp: working peer [[13243,0],2] address tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.0.19.202 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.5.10.182 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.101.1.2 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.3.0.182 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] PASSING ADDR 10.8.10.182 TO MODULE
[ivy01:05396] [[13243,0],1]:tcp set addr for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1]: peer [[13243,0],2] is reachable via component tcp
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.0.8.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.193.184.48 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.193.184.49 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.10.0.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.2.0.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.3.0.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.3.0.252 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],0] is listening on net 10.50.0.254 port 45024
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] SET_PEER ADDING PEER [[13243,0],2]
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],2] is listening on net 10.0.19.202 port 36928
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],2] is listening on net 10.5.10.182 port 36928
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],2] is listening on net 10.101.1.2 port 36928
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],2] is listening on net 10.3.0.182 port 36928
[ivy01:05396] [[13243,0],1]:tcp:processing set_peer cmd
[ivy01:05396] [[13243,0],1] set_peer: peer [[13243,0],2] is listening on net 10.8.10.182 port 36928
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.193.184.48 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.193.184.49 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.10.0.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.2.0.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.3.0.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.3.0.252 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],0] is listening on net 10.50.0.254 port 45024
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] SET_PEER ADDING PEER [[13243,0],1]
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],1] is listening on net 10.0.20.187 port 52741
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],1] is listening on net 10.5.10.181 port 52741
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],1] is listening on net 10.101.1.1 port 52741
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],1] is listening on net 10.3.0.181 port 52741
[ivy02:05706] [[13243,0],2]:tcp:processing set_peer cmd
[ivy02:05706] [[13243,0],2] set_peer: peer [[13243,0],1] is listening on net 10.8.10.181 port 52741
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 73
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 73 BYTES FOR DEST [[13243,0],0] TAG 5
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 5 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:5 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:5 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 73 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:5 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:5 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 73 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 73
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 73 BYTES FOR DEST [[13243,0],0] TAG 5
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 5 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:5 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:5 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 29 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 29
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 29 BYTES FOR DEST [[13243,0],0] TAG 5
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 5 channel = -1 seq_num = -1
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:5 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:5 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 29 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 29
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 29 BYTES FOR DEST [[13243,0],0] TAG 5
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 5 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 200 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 200
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 200 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 488
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 488 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] mca: base: components_register: registering framework errmgr components
[ivy01:05551] mca: base: components_register: found loaded component default_orted
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 488 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 947 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 947
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 947 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] mca: base: components_register: component default_orted register function successful
[ivy01:05551] mca: base: components_register: found loaded component default_tool
[ivy01:05551] mca: base: components_register: component default_tool register function successful
[ivy01:05551] mca: base: components_register: found loaded component default_app
[ivy01:05551] mca: base: components_register: component default_app register function successful
[ivy01:05551] mca: base: components_register: found loaded component default_hnp
[ivy01:05551] mca: base: components_register: component default_hnp register function successful
[ivy01:05551] mca: base: components_open: opening errmgr components
[ivy01:05551] mca: base: components_open: found loaded component default_orted
[ivy01:05551] mca: base: components_open: component default_orted open function successful
[ivy01:05551] mca: base: components_open: found loaded component default_tool
[ivy01:05551] mca: base: components_open: component default_tool open function successful
[ivy01:05551] mca: base: components_open: found loaded component default_app
[ivy01:05551] mca: base: components_open: component default_app open function successful
[ivy01:05551] mca: base: components_open: found loaded component default_hnp
[ivy01:05551] mca: base: components_open: component default_hnp open function successful
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 186 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 186
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 186 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 119
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 119 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] mca: base: components_register: registering framework oob components
[ivy01:05551] mca: base: components_register: found loaded component ud
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 104
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 104 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] mca: base: components_register: component ud register function successful
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 119 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 104 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 120
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 120 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 120 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 197 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 197
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 197 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 713 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 124 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 872 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05551] mca: base: components_register: found loaded component tcp
[ivy01:05551] mca: base: components_register: component tcp register function successful
[ivy01:05551] mca: base: components_register: found loaded component usock
[ivy01:05551] mca: base: components_register: component usock register function successful
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 713
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 713 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 91 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 852 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 240 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 418 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 124
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 124 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] mca: base: components_open: opening oob components
[ivy01:05551] mca: base: components_open: found loaded component ud
[ivy01:05551] mca: base: components_open: component ud open function successful
[ivy01:05551] mca: base: components_open: found loaded component tcp
[ivy01:05551] mca: base: components_open: component tcp open function successful
[ivy01:05551] mca: base: components_open: found loaded component usock
[ivy01:05551] mca: base: components_open: component usock open function successful
[ivy01:05551] mca:oob:select: checking available component ud
[ivy01:05551] mca:oob:select: Querying component [ud]
[ivy01:05551] oob:ud: component_available called
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 872
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 872 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] [[13243,1],0] oob:ud:device_setup attempting to setup ib device 0x2aaaaea35440
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 91
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 91 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] [[13243,1],0] oob:ud:device_setup error opening device. errno = 2
[ivy01:05551] [[13243,1],0] oob:ud:device_setup attempting to setup ib device 0x2aaaaea35440
[ivy01:05551] [[13243,1],0] oob:ud:device_setup error opening device. errno = 2
[ivy01:05551] [[13243,1],0] oob:ud:device_setup attempting to setup ib device 0x2aaaaea35440
[ivy01:05551] [[13243,1],0] oob:ud:device_setup error opening device. errno = 2
[ivy01:05551] [[13243,1],0] oob:ud:device_setup attempting to setup ib device 0x2aaaaea35440
[ivy01:05551] [[13243,1],0] oob:ud:device_setup error opening device. errno = 2
[ivy01:05551] mca:oob:select: Skipping component [ud] - failed to startup
[ivy01:05551] mca:oob:select: checking available component tcp
[ivy01:05551] mca:oob:select: Querying component [tcp]
[ivy01:05551] oob:tcp: component_available called
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 852
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 852 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[ivy01:05551] [[13243,1],0] oob:tcp:init rejecting loopback interface lo
[ivy01:05551] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[ivy01:05551] [[13243,1],0] oob:tcp:init adding 10.0.20.187 to our list of V4 connections
[ivy01:05551] WORKING INTERFACE 3 KERNEL INDEX 6 FAMILY: V4
[ivy01:05551] [[13243,1],0] oob:tcp:init adding 10.5.10.181 to our list of V4 connections
[ivy01:05551] WORKING INTERFACE 4 KERNEL INDEX 7 FAMILY: V4
[ivy01:05551] [[13243,1],0] oob:tcp:init adding 10.101.1.1 to our list of V4 connections
[ivy01:05551] WORKING INTERFACE 5 KERNEL INDEX 8 FAMILY: V4
[ivy01:05551] [[13243,1],0] oob:tcp:init adding 10.3.0.181 to our list of V4 connections
[ivy01:05551] WORKING INTERFACE 6 KERNEL INDEX 9 FAMILY: V4
[ivy01:05551] [[13243,1],0] oob:tcp:init adding 10.8.10.181 to our list of V4 connections
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 240
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 240 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 418
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 418 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] [[13243,1],0] TCP STARTUP
[ivy01:05551] [[13243,1],0] attempting to bind to IPv4 port 0
[ivy01:05551] [[13243,1],0] assigned IPv4 port 35701
[ivy01:05551] mca:oob:select: Adding component to end
[ivy01:05551] mca:oob:select: checking available component usock
[ivy01:05551] mca:oob:select: Querying component [usock]
[ivy01:05551] oob:usock: component_available called
[ivy01:05551] [[13243,1],0] USOCK STARTUP
[ivy01:05551] SUNPATH: /tmp/openmpi-sessions-182726@ivy01_0/13243/0/usock
[ivy01:05551] [[13243,1],0] TCP SHUTDOWN
[ivy01:05551] mca:oob:select: Found 1 active transports
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 276
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 276 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 753
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 753 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] mca:base:select: Auto-selecting errmgr components
[ivy01:05551] mca:base:select:(errmgr) Querying component [default_orted]
[ivy01:05551] mca:base:select:(errmgr) Skipping component [default_orted]. Query failed to return a module
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 283
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 283 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05551] mca:base:select:(errmgr) Querying component [default_tool]
[ivy01:05551] mca:base:select:(errmgr) Skipping component [default_tool]. Query failed to return a module
[ivy01:05551] mca:base:select:(errmgr) Querying component [default_app]
[ivy01:05551] mca:base:select:(errmgr) Query of component [default_app] set priority to 1000
[ivy01:05551] mca:base:select:(errmgr) Querying component [default_hnp]
[ivy01:05551] mca:base:select:(errmgr) Skipping component [default_hnp]. Query failed to return a module
[ivy01:05551] mca:base:select:(errmgr) Selected component [default_app]
[ivy01:05551] mca: base: close: component default_orted closed
[ivy01:05551] mca: base: close: unloading component default_orted
[ivy01:05551] mca: base: close: component default_tool closed
[ivy01:05551] mca: base: close: unloading component default_tool
[ivy01:05551] mca: base: close: component default_hnp closed
[ivy01:05551] mca: base: close: unloading component default_hnp
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 276 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 753 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 283 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 200 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 310 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 1125 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] connection_event_handler: working connection (14, 0)
[ivy01:05396] [[13243,0],1] accept_connection
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 265 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 749 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1]:usock:recv:handler called
[ivy01:05396] [[13243,0],1] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[ivy01:05396] [[13243,0],1] waiting for connect ack from UNKNOWN
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 200
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 200 BYTES FOR DEST [[13243,0],0] TAG 2
[ivy01:05396] [[13243,0],1] connect ack received from UNKNOWN
[ivy01:05396] [[13243,0],1] connect-ack recvd from UNKNOWN
[ivy01:05396] [[13243,0],1] mca_oob_usock_recv_connect: connection from new peer
[ivy01:05396] [[13243,0],1] connect-ack header from [[13243,1],0] is okay
[ivy01:05396] [[13243,0],1] waiting for connect ack from [[13243,1],0]
[ivy01:05396] [[13243,0],1] connect ack received from [[13243,1],0]
[ivy01:05396] [[13243,0],1] connect-ack version from [[13243,1],0] matches ours
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 310
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 310 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 265
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 265 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] mca: base: components_register: registering framework errmgr components
[ivy02:05722] mca: base: components_register: found loaded component default_orted
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 1125
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 1125 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 749
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 749 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] mca: base: components_register: component default_orted register function successful
[ivy02:05722] mca: base: components_register: found loaded component default_tool
[ivy02:05722] mca: base: components_register: component default_tool register function successful
[ivy01:05551] [[13243,1],0]: set_addr to uri 867893248.1;usock;tcp://10.0.20.187,10.5.10.181,10.101.1.1,10.3.0.181,10.8.10.181:52741
[ivy01:05551] [[13243,1],0]:set_addr checking if peer [[13243,0],1] is reachable via component usock
[ivy01:05396] [[13243,0],1] connect-ack [[13243,1],0] authenticated
[ivy01:05396] [[13243,0],1] usock:peer_accept called for peer [[13243,1],0] in state UNKNOWN on socket 14
[ivy01:05396] [[13243,0],1] SEND CONNECT ACK
[ivy01:05396] [[13243,0],1] send blocking of 209 bytes to socket 14
[ivy01:05396] [[13243,0],1] blocking send complete to socket 14
[ivy01:05396] [[13243,0],1]-[[13243,1],0] usock_peer_connected on socket 14
[ivy01:05396] [[13243,0],1]-[[13243,1],0] accepted: nodelay 0 flags 00000802
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] usock:set_module called for peer [[13243,1],0]
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 163 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 682 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 99 BYTES ON SOCKET 11
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 333 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 186 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 119 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 104 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 120 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 830 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 80 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 124 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 630 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 273 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 91 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 954 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 428 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 128 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 186
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 276 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 1005 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 186 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 163
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 163 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 119
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 119 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 682
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 682 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] mca: base: components_register: found loaded component default_app
[ivy02:05722] mca: base: components_register: component default_app register function successful
[ivy02:05722] mca: base: components_register: found loaded component default_hnp
[ivy02:05722] mca: base: components_register: component default_hnp register function successful
[ivy02:05722] mca: base: components_open: opening errmgr components
[ivy02:05722] mca: base: components_open: found loaded component default_orted
[ivy02:05722] mca: base: components_open: component default_orted open function successful
[ivy02:05722] mca: base: components_open: found loaded component default_tool
[ivy02:05722] mca: base: components_open: component default_tool open function successful
[ivy02:05722] mca: base: components_open: found loaded component default_app
[ivy02:05722] mca: base: components_open: component default_app open function successful
[ivy02:05722] mca: base: components_open: found loaded component default_hnp
[ivy02:05722] mca: base: components_open: component default_hnp open function successful
[ivy01:05551] [[13243,1],0]:[oob_usock_component.c:319] connect to [[13243,0],1]
[ivy01:05551] [[13243,1],0]: peer [[13243,0],1] is reachable via component usock
[ivy01:05551] [[13243,1],0] orte_usock_peer_try_connect: attempting to connect to proc [[13243,0],1]
[ivy01:05551] [[13243,1],0] oob:usock:peer creating socket to [[13243,0],1]
[ivy01:05551] [[13243,1],0] orte_usock_peer_try_connect: attempting to connect to proc [[13243,0],1] on socket 20
[ivy01:05551] [[13243,1],0] orte_usock_peer_try_connect: attempting to connect to proc [[13243,0],1] - 0 retries
[ivy01:05551] [[13243,1],0] orte_usock_peer_try_connect: Connection across to proc [[13243,0],1] succeeded
[ivy01:05551] [[13243,1],0] SEND CONNECT ACK
[ivy02:05722] mca: base: components_register: registering framework oob components
[ivy02:05722] mca: base: components_register: found loaded component ud
[ivy01:05551] [[13243,1],0] send blocking of 254 bytes to socket 20
[ivy01:05551] [[13243,1],0] blocking send complete to socket 20
[ivy02:05722] mca: base: components_register: component ud register function successful
[ivy01:05551] [[13243,1],0]:usock:recv:handler called for peer [[13243,0],1]
[ivy01:05551] [[13243,1],0] RECV CONNECT ACK FROM [[13243,0],1] ON SOCKET 20
[ivy01:05551] [[13243,1],0] waiting for connect ack from [[13243,0],1]
[ivy01:05551] [[13243,1],0] connect ack received from [[13243,0],1]
[ivy01:05551] [[13243,1],0] connect-ack recvd from [[13243,0],1]
[ivy01:05551] [[13243,1],0] connect-ack header from [[13243,0],1] is okay
[ivy01:05551] [[13243,1],0] waiting for connect ack from [[13243,0],1]
[ivy01:05551] [[13243,1],0] connect ack received from [[13243,0],1]
[ivy01:05551] [[13243,1],0] connect-ack version from [[13243,0],1] matches ours
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 104
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 265 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] connection_event_handler: working connection (14, 0)
[ivy02:05706] [[13243,0],2] accept_connection
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 749 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2]:usock:recv:handler called
[ivy02:05706] [[13243,0],2] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[ivy02:05706] [[13243,0],2] waiting for connect ack from UNKNOWN
[ivy02:05706] [[13243,0],2] connect ack received from UNKNOWN
[ivy02:05706] [[13243,0],2] connect-ack recvd from UNKNOWN
[ivy02:05706] [[13243,0],2] mca_oob_usock_recv_connect: connection from new peer
[ivy02:05706] [[13243,0],2] connect-ack header from [[13243,1],1] is okay
[ivy02:05706] [[13243,0],2] waiting for connect ack from [[13243,1],1]
[ivy02:05706] [[13243,0],2] connect ack received from [[13243,1],1]
[ivy02:05706] [[13243,0],2] connect-ack version from [[13243,1],1] matches ours
[ivy02:05706] [[13243,0],2] connect-ack [[13243,1],1] authenticated
[ivy02:05706] [[13243,0],2] usock:peer_accept called for peer [[13243,1],1] in state UNKNOWN on socket 14
[ivy02:05706] [[13243,0],2] SEND CONNECT ACK
[ivy02:05706] [[13243,0],2] send blocking of 209 bytes to socket 14
[ivy02:05706] [[13243,0],2] blocking send complete to socket 14
[ivy02:05706] [[13243,0],2]-[[13243,1],1] usock_peer_connected on socket 14
[ivy02:05706] [[13243,0],2]-[[13243,1],1] accepted: nodelay 0 flags 00000802
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 99 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] usock:set_module called for peer [[13243,1],1]
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 172 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 104 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 99
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 99 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 605 BYTES ON SOCKET 11
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 401 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 120
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 120 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 333
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 333 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 830
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 830 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] mca: base: components_register: found loaded component tcp
[ivy01:05551] [[13243,1],0] connect-ack [[13243,0],1] authenticated
[ivy02:05722] mca: base: components_register: component tcp register function successful
[ivy01:05551] [[13243,1],0]-[[13243,0],1] usock_peer_connected on socket 20
[ivy01:05551] [[13243,1],0]-[[13243,0],1] connected: nodelay 0 flags 00000802
[ivy01:05551] [[13243,1],0]:usock:recv:handler starting send/recv events
[ivy01:05551] [[13243,1],0] usock:set_module called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 80
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 80 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] mca: base: components_register: found loaded component usock
[ivy02:05722] mca: base: components_register: component usock register function successful
[ivy02:05722] mca: base: components_open: opening oob components
[ivy02:05722] mca: base: components_open: found loaded component ud
[ivy02:05722] mca: base: components_open: component ud open function successful
[ivy02:05722] mca: base: components_open: found loaded component tcp
[ivy02:05722] mca: base: components_open: component tcp open function successful
[ivy02:05722] mca: base: components_open: found loaded component usock
[ivy02:05722] mca: base: components_open: component usock open function successful
[ivy02:05722] mca:oob:select: checking available component ud
[ivy02:05722] mca:oob:select: Querying component [ud]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 124
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 124 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] oob:ud: component_available called
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 630
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 630 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae92c440
[ivy02:05722] [[13243,1],1] oob:ud:device_setup error opening device. errno = 2
[ivy02:05722] [[13243,1],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae92c440
[ivy02:05722] [[13243,1],1] oob:ud:device_setup error opening device. errno = 2
[ivy02:05722] [[13243,1],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae92c440
[ivy02:05722] [[13243,1],1] oob:ud:device_setup error opening device. errno = 2
[ivy02:05722] [[13243,1],1] oob:ud:device_setup attempting to setup ib device 0x2aaaae92c440
[ivy02:05722] [[13243,1],1] oob:ud:device_setup error opening device. errno = 2
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 273
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 273 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 91
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 91 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] mca:oob:select: Skipping component [ud] - failed to startup
[ivy02:05722] mca:oob:select: checking available component tcp
[ivy02:05722] mca:oob:select: Querying component [tcp]
[ivy02:05722] oob:tcp: component_available called
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 954
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 954 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[ivy02:05722] [[13243,1],1] oob:tcp:init rejecting loopback interface lo
[ivy02:05722] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[ivy02:05722] [[13243,1],1] oob:tcp:init adding 10.0.19.202 to our list of V4 connections
[ivy02:05722] WORKING INTERFACE 3 KERNEL INDEX 6 FAMILY: V4
[ivy02:05722] [[13243,1],1] oob:tcp:init adding 10.5.10.182 to our list of V4 connections
[ivy02:05722] WORKING INTERFACE 4 KERNEL INDEX 7 FAMILY: V4
[ivy02:05722] [[13243,1],1] oob:tcp:init adding 10.101.1.2 to our list of V4 connections
[ivy02:05722] WORKING INTERFACE 5 KERNEL INDEX 8 FAMILY: V4
[ivy02:05722] [[13243,1],1] oob:tcp:init adding 10.3.0.182 to our list of V4 connections
[ivy02:05722] WORKING INTERFACE 6 KERNEL INDEX 9 FAMILY: V4
[ivy02:05722] [[13243,1],1] oob:tcp:init adding 10.8.10.182 to our list of V4 connections
[ivy02:05722] [[13243,1],1] TCP STARTUP
[ivy02:05722] [[13243,1],1] attempting to bind to IPv4 port 0
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 428
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 428 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 128
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 128 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1] assigned IPv4 port 47557
[ivy02:05722] mca:oob:select: Adding component to end
[ivy02:05722] mca:oob:select: checking available component usock
[ivy02:05722] mca:oob:select: Querying component [usock]
[ivy02:05722] oob:usock: component_available called
[ivy02:05722] [[13243,1],1] USOCK STARTUP
[ivy02:05722] SUNPATH: /tmp/openmpi-sessions-182726@ivy02_0/13243/0/usock
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 276
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 276 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1] TCP SHUTDOWN
[ivy02:05722] mca:oob:select: Found 1 active transports
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 1005
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 1005 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] mca:base:select: Auto-selecting errmgr components
[ivy02:05722] mca:base:select:(errmgr) Querying component [default_orted]
[ivy02:05722] mca:base:select:(errmgr) Skipping component [default_orted]. Query failed to return a module
[ivy02:05722] mca:base:select:(errmgr) Querying component [default_tool]
[ivy02:05722] mca:base:select:(errmgr) Skipping component [default_tool]. Query failed to return a module
[ivy02:05722] mca:base:select:(errmgr) Querying component [default_app]
[ivy02:05722] mca:base:select:(errmgr) Query of component [default_app] set priority to 1000
[ivy02:05722] mca:base:select:(errmgr) Querying component [default_hnp]
[ivy02:05722] mca:base:select:(errmgr) Skipping component [default_hnp]. Query failed to return a module
[ivy02:05722] mca:base:select:(errmgr) Selected component [default_app]
[ivy02:05722] mca: base: close: component default_orted closed
[ivy02:05722] mca: base: close: unloading component default_orted
[ivy02:05722] mca: base: close: component default_tool closed
[ivy02:05722] mca: base: close: unloading component default_tool
[ivy02:05722] mca: base: close: component default_hnp closed
[ivy02:05722] mca: base: close: unloading component default_hnp
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 265
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 265 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 749
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 749 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1]: set_addr to uri 867893248.2;usock;tcp://10.0.19.202,10.5.10.182,10.101.1.2,10.3.0.182,10.8.10.182:36928
[ivy02:05722] [[13243,1],1]:set_addr checking if peer [[13243,0],2] is reachable via component usock
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 99
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 99 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1]:[oob_usock_component.c:319] connect to [[13243,0],2]
[ivy02:05722] [[13243,1],1]: peer [[13243,0],2] is reachable via component usock
[ivy02:05722] [[13243,1],1] orte_usock_peer_try_connect: attempting to connect to proc [[13243,0],2]
[ivy02:05722] [[13243,1],1] oob:usock:peer creating socket to [[13243,0],2]
[ivy02:05722] [[13243,1],1] orte_usock_peer_try_connect: attempting to connect to proc [[13243,0],2] on socket 20
[ivy02:05722] [[13243,1],1] orte_usock_peer_try_connect: attempting to connect to proc [[13243,0],2] - 0 retries
[ivy02:05722] [[13243,1],1] orte_usock_peer_try_connect: Connection across to proc [[13243,0],2] succeeded
[ivy02:05722] [[13243,1],1] SEND CONNECT ACK
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 172
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 172 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1] send blocking of 254 bytes to socket 20
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 605
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 605 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1] blocking send complete to socket 20
[ivy02:05722] [[13243,1],1]:usock:recv:handler called for peer [[13243,0],2]
[ivy02:05722] [[13243,1],1] RECV CONNECT ACK FROM [[13243,0],2] ON SOCKET 20
[ivy02:05722] [[13243,1],1] waiting for connect ack from [[13243,0],2]
[ivy02:05722] [[13243,1],1] connect ack received from [[13243,0],2]
[ivy02:05722] [[13243,1],1] connect-ack recvd from [[13243,0],2]
[ivy02:05722] [[13243,1],1] connect-ack header from [[13243,0],2] is okay
[ivy02:05722] [[13243,1],1] waiting for connect ack from [[13243,0],2]
[ivy02:05722] [[13243,1],1] connect ack received from [[13243,0],2]
[ivy02:05722] [[13243,1],1] connect-ack version from [[13243,0],2] matches ours
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 401
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 401 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05722] [[13243,1],1] connect-ack [[13243,0],2] authenticated
[ivy02:05722] [[13243,1],1]-[[13243,0],2] usock_peer_connected on socket 20
[ivy02:05722] [[13243,1],1]-[[13243,0],2] connected: nodelay 0 flags 00000802
[ivy02:05722] [[13243,1],1]:usock:recv:handler starting send/recv events
[ivy02:05722] [[13243,1],1] usock:set_module called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],2]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],2]:35 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],2]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],2]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],2]:35 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:438] queue pending to [[13243,0],2]
[ivy01:05396] [[13243,0],1] tcp:send_nb: initiating connection to [[13243,0],2]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:452] connect to [[13243,0],2]
[ivy01:05396] [[13243,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],2]
[ivy01:05396] [[13243,0],1] oob:tcp:peer creating socket to [[13243,0],2]
[ivy01:05396] [[13243,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],2] on socket 17
[ivy01:05396] [[13243,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[13243,0],2] on 10.0.19.202:36928 - 0 retries
[ivy01:05396] [[13243,0],1] waiting for connect completion to [[13243,0],2] - activating send event
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] tcp:send_handler CONNECTING
[ivy01:05396] [[13243,0],1]:tcp:complete_connect called for peer [[13243,0],2] on socket 17
[ivy01:05396] [[13243,0],1] tcp_peer_complete_connect: sending ack to [[13243,0],2]
[ivy01:05396] [[13243,0],1] SEND CONNECT ACK
[ivy02:05706] [[13243,0],2] connection_event_handler: working connection (15, 0) 10.0.20.187:40549
[ivy02:05706] [[13243,0],2] accept_connection: 10.0.20.187:40549
[ivy02:05706] [[13243,0],2]:tcp:recv:handler called
[ivy02:05706] [[13243,0],2] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 15
[ivy02:05706] [[13243,0],2] waiting for connect ack from UNKNOWN
[ivy02:05706] [[13243,0],2] connect ack received from UNKNOWN
[ivy02:05706] [[13243,0],2] connect-ack recvd from UNKNOWN
[ivy02:05706] [[13243,0],2] connect-ack header from [[13243,0],1] is okay
[ivy02:05706] [[13243,0],2] waiting for connect ack from [[13243,0],1]
[ivy02:05706] [[13243,0],2] connect ack received from [[13243,0],1]
[ivy02:05706] [[13243,0],2] connect-ack version from [[13243,0],1] matches ours
[ivy01:05396] [[13243,0],1] SENDING CREDENTIAL OF SIZE 210
[ivy01:05396] [[13243,0],1] send blocking of 254 bytes to socket 17
[ivy01:05396] [[13243,0],1] blocking send complete to socket 17
[ivy01:05396] [[13243,0],1] tcp_peer_complete_connect: setting read event on connection to [[13243,0],2]
[ivy02:05706] [[13243,0],2] connect-ack [[13243,0],1] authenticated
[ivy02:05706] [[13243,0],2] tcp:peer_accept called for peer [[13243,0],1] in state UNCONNECTED on socket 15
[ivy02:05706] [[13243,0],2] SEND CONNECT ACK
[ivy02:05706] [[13243,0],2] SENDING CREDENTIAL OF SIZE 165
[ivy02:05706] [[13243,0],2] send blocking of 209 bytes to socket 15
[ivy02:05706] [[13243,0],2] blocking send complete to socket 15
[ivy02:05706] [[13243,0],2]-[[13243,0],1] tcp_peer_connected on socket 15
[ivy02:05706] [[13243,0],2]-[[13243,0],1] accepted: 10.0.19.202 - 10.0.20.187 nodelay 1 sndbuf 262144 rcvbuf 262144 flags 00000802
[ivy02:05706] [[13243,0],2] tcp:set_module called for peer [[13243,0],1]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] RECV CONNECT ACK FROM [[13243,0],2] ON SOCKET 17
[ivy01:05396] [[13243,0],1] waiting for connect ack from [[13243,0],2]
[ivy01:05396] [[13243,0],1] connect ack received from [[13243,0],2]
[ivy01:05396] [[13243,0],1] connect-ack recvd from [[13243,0],2]
[ivy01:05396] [[13243,0],1] connect-ack header from [[13243,0],2] is okay
[ivy01:05396] [[13243,0],1] waiting for connect ack from [[13243,0],2]
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],1]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],1]:35 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],1]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],1]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],1]:35 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],1] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],1] channel =-1
[ivy02:05706] [[13243,0],2]:tcp:recv:handler called for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2]:tcp:recv:handler CONNECTED
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate new recv msg
[ivy02:05706] [[13243,0],2]:tcp:recv:handler read hdr
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate data region of size 434
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],1]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],1] OF 401 BYTES ON SOCKET 15
[ivy01:05396] [[13243,0],1] connect ack received from [[13243,0],2]
[ivy01:05396] [[13243,0],1] connect-ack version from [[13243,0],2] matches ours
[ivy01:05396] [[13243,0],1] connect-ack [[13243,0],2] authenticated
[ivy01:05396] [[13243,0],1]-[[13243,0],2] tcp_peer_connected on socket 17
[ivy01:05396] [[13243,0],1]-[[13243,0],2] connected: 10.0.20.187 - 10.0.19.202 nodelay 1 sndbuf 262144 rcvbuf 262144 flags 00000802
[ivy01:05396] [[13243,0],1]:tcp:recv:handler starting send/recv events
[ivy01:05396] [[13243,0],1] tcp:set_module called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],2]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],2] OF 434 BYTES ON SOCKET 17
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy02:05706] [[13243,0],2]:tcp:recv:handler called for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2]:tcp:recv:handler CONNECTED
[ivy02:05706] [[13243,0],2] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 434 BYTES FOR DEST [[13243,0],2] TAG 35
[ivy02:05706] [[13243,0],2] DELIVERING TO RML tag = 35 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate data region of size 401
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 401 BYTES FOR DEST [[13243,0],1] TAG 35
[ivy01:05396] [[13243,0],1] DELIVERING TO RML tag = 35 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],2]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],2]:35 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],2]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],2]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],2]:35 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],2] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],2] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],2]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],2] OF 57 BYTES ON SOCKET 17
[ivy02:05706] [[13243,0],2]:tcp:recv:handler called for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2]:tcp:recv:handler CONNECTED
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate new recv msg
[ivy02:05706] [[13243,0],2]:tcp:recv:handler read hdr
[ivy02:05706] [[13243,0],2]:tcp:recv:handler allocate data region of size 57
[ivy02:05706] [[13243,0],2] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 57 BYTES FOR DEST [[13243,0],2] TAG 35
[ivy02:05706] [[13243,0],2] DELIVERING TO RML tag = 35 channel = -1 seq_num = -1
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],1]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],1]:35 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],1]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],1]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],1]:35 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],1] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],1] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],1]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],1]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],1] OF 57 BYTES ON SOCKET 15
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate data region of size 57
[ivy01:05396] [[13243,0],1] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 57 BYTES FOR DEST [[13243,0],1] TAG 35
[ivy01:05396] [[13243,0],1] DELIVERING TO RML tag = 35 channel = -1 seq_num = -1
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:26 2015: PID 5722 on host ivy02 sleeping...
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:26 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:28 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:28 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:30 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:30 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:32 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:32 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:34 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:34 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:36 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
Fri May 22 07:35:36 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:38 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:38 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:40 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:40 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
Fri May 22 07:35:42 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:42 2015: PID 5722 on host ivy02 sleeping...
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:44 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:44 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
Fri May 22 07:35:46 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy02:05706] [[13243,0],2] OOB_SEND: rml_oob_send.c:210
[ivy02:05706] [[13243,0],2] oob:base:send to target [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:base:send known transport for peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy02:05706] [[13243,0],2] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy02:05706] [[13243,0],2]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy02:05706] [[13243,0],2] tcp:send_handler called to send to peer [[13243,0],0]
[ivy02:05706] [[13243,0],2] tcp:send_handler SENDING TO [[13243,0],0]
[ivy02:05706] [[13243,0],2] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],2] (ORIGIN [[13243,0],2]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:46 2015: PID 5722 on host ivy02 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
Fri May 22 07:35:48 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
Fri May 22 07:35:50 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:52 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:35:54 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
Fri May 22 07:35:56 2015: PID 5551 on host ivy01 sleeping...
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
Fri May 22 07:35:58 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:36:00 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
Fri May 22 07:36:02 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:36:04 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:36:06 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:36:08 2015: PID 5551 on host ivy01 sleeping...
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate data region of size 92
[ivy01:05396] [[13243,0],1] OOB_SEND: rml_oob_send.c:210
[ivy01:05396] [[13243,0],1] oob:base:send to target [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:base:send known transport for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] oob:tcp:send_nb to peer [[13243,0],0]:2 to channel=-1 seq = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:467] post send to [[13243,0],0]
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:404] processing send to peer [[13243,0],0]:2 to channel =-1 seq_num = -1
[ivy01:05396] [[13243,0],1] tcp:send_nb: already connected to [[13243,0],0] - queueing for send
[ivy01:05396] [[13243,0],1]:[oob_tcp.c:431] queue send to [[13243,0],0] channel =-1
[ivy01:05396] [[13243,0],1] tcp:send_handler called to send to peer [[13243,0],0]
[ivy01:05396] [[13243,0],1] tcp:send_handler SENDING TO [[13243,0],0]
[ivy01:05396] [[13243,0],1] MESSAGE SEND COMPLETE TO [[13243,0],0] OF 92 BYTES ON SOCKET 11
[savbu-usnic-a:25277] [[13243,0],0] RECVD COMPLETE MESSAGE FROM [[13243,0],1] (ORIGIN [[13243,0],1]) OF 92 BYTES FOR DEST [[13243,0],0] TAG 2
[savbu-usnic-a:25277] [[13243,0],0] DELIVERING TO RML tag = 2 channel = -1 seq_num = -1
Fri May 22 07:36:10 2015: PID 5551 on host ivy01 sleeping...
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1]-[[13243,0],2] mca_oob_tcp_msg_recv: readv failed: Connection timed out (110)
[ivy01:05396] [[13243,0],1]:tcp:recv:handler error reading bytes - closing connection
[ivy01:05396] [[13243,0],1] tcp_peer_close for [[13243,0],2] sd 17 state CONNECTED
[ivy01:05396] [[13243,0],1] tcp:lost connection called for peer [[13243,0],2]
[ivy01:05396] [[13243,0],1] errmgr:default_orted:proc_errors process [[13243,0],2] error state COMMUNICATION FAILURE
[ivy01:05396] [[13243,0],1] errmgr:default:orted daemon [[13243,0],2] exited
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler CONNECTED
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler allocate new recv msg
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler read hdr
[savbu-usnic-a:25277] [[13243,0],0]-[[13243,0],2] mca_oob_tcp_msg_recv: readv failed: Connection timed out (110)
[savbu-usnic-a:25277] [[13243,0],0]:tcp:recv:handler error reading bytes - closing connection
[savbu-usnic-a:25277] [[13243,0],0] tcp_peer_close for [[13243,0],2] sd 16 state CONNECTED
[savbu-usnic-a:25277] [[13243,0],0] tcp:lost connection called for peer [[13243,0],2]
[savbu-usnic-a:25277] [[13243,0],0] errmgr:default_hnp: for proc [[13243,0],2] state COMMUNICATION FAILURE
[savbu-usnic-a:25277] [[13243,0],0] Comm failure: daemon [[13243,0],2] - aborting
[savbu-usnic-a:25277] [[13243,0],0] errmgr:default_hnp: abort called on job [13243,0]
[savbu-usnic-a:25277] [[13243,0],0] errmgr:default_hnp: ordering orted termination
[savbu-usnic-a:25277] [[13243,0],0] OOB_SEND: rml_oob_send.c:210
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send to target [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] oob:base:send known transport for peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] oob:tcp:send_nb to peer [[13243,0],1]:15 to channel=-1 seq = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb to peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:467] post send to [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:404] processing send to peer [[13243,0],1]:15 to channel =-1 seq_num = -1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_nb: already connected to [[13243,0],1] - queueing for send
[savbu-usnic-a:25277] [[13243,0],0]:[oob_tcp.c:431] queue send to [[13243,0],1] channel =-1
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler called to send to peer [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] tcp:send_handler SENDING TO [[13243,0],1]
[savbu-usnic-a:25277] [[13243,0],0] MESSAGE SEND COMPLETE TO [[13243,0],1] OF 50 BYTES ON SOCKET 17
[ivy01:05396] [[13243,0],1]:tcp:recv:handler called for peer [[13243,0],0]
[ivy01:05396] [[13243,0],1]:tcp:recv:handler CONNECTED
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate new recv msg
[ivy01:05396] [[13243,0],1]:tcp:recv:handler read hdr
[ivy01:05396] [[13243,0],1]:tcp:recv:handler allocate data region of size 50
[ivy01:05396] [[13243,0],1] RECVD COMPLETE MESSAGE FROM [[13243,0],0] (ORIGIN [[13243,0],0]) OF 50 BYTES FOR DEST [[13243,0],1] TAG 15
[ivy01:05396] [[13243,0],1] DELIVERING TO RML tag = 15 channel = -1 seq_num = -1
[ivy01:05396] mca: base: close: component default_orted closed
[ivy01:05396] mca: base: close: unloading component default_orted
[ivy01:05396] [[13243,0],1] USOCK SHUTDOWN
[ivy01:05396] [[13243,0],1] TCP SHUTDOWN
[ivy01:05396] [[13243,0],1] RELEASING PEER OBJ [[13243,0],2]
[ivy01:05396] [[13243,0],1] RELEASING PEER OBJ [[13243,0],0]
[ivy01:05396] [[13243,0],1] CLOSING SOCKET 11
[ivy01:05396] [[13243,0],1] oob:ud:component_close entering
[ivy01:05396] mca: base: close: component ud closed
[ivy01:05396] mca: base: close: unloading component ud
[ivy01:05396] mca: base: close: component tcp closed
[ivy01:05396] mca: base: close: unloading component tcp
[ivy01:05396] mca: base: close: component usock closed
[ivy01:05396] mca: base: close: unloading component usock

[...job hangs here...]