Skip to content

Instantly share code, notes, and snippets.

@jsquyres

jsquyres/np=4.ppn=2.out.txt

Last active September 25, 2017 21:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jsquyres/5351c7cfc29c6fd78392cc467f2ef158 to your computer and use it in GitHub Desktop.
Save jsquyres/5351c7cfc29c6fd78392cc467f2ef158 to your computer and use it in GitHub Desktop.
Download ZIP
TCP BTL run across 2 16-core nodes, each with 8 non-loopback IP interfaces, with mca_btl_base_verbose=100
Raw
np=4.ppn=2.out.txt
[mpi020:28517] mca: base: components_register: registering framework btl components
[mpi020:28517] mca: base: components_register: found loaded component self
[mpi020:28517] mca: base: components_register: component self register function successful
[mpi020:28517] mca: base: components_register: found loaded component tcp
[mpi020:28518] mca: base: components_register: registering framework btl components
[mpi020:28518] mca: base: components_register: found loaded component self
[mpi019:22888] mca: base: components_register: registering framework btl components
[mpi019:22888] mca: base: components_register: found loaded component self
[mpi019:22889] mca: base: components_register: registering framework btl components
[mpi019:22889] mca: base: components_register: found loaded component self
[mpi020:28517] mca: base: components_register: component tcp register function successful
[mpi020:28517] mca: base: components_register: found loaded component vader
[mpi019:22888] mca: base: components_register: component self register function successful
[mpi019:22888] mca: base: components_register: found loaded component tcp
[mpi020:28518] mca: base: components_register: component self register function successful
[mpi020:28518] mca: base: components_register: found loaded component tcp
[mpi019:22889] mca: base: components_register: component self register function successful
[mpi019:22889] mca: base: components_register: found loaded component tcp
[mpi019:22888] mca: base: components_register: component tcp register function successful
[mpi019:22888] mca: base: components_register: found loaded component vader
[mpi020:28517] mca: base: components_register: component vader register function successful
[mpi020:28517] mca: base: components_open: opening btl components
[mpi020:28517] mca: base: components_open: found loaded component self
[mpi020:28517] mca: base: components_open: component self open function successful
[mpi020:28517] mca: base: components_open: found loaded component tcp
[mpi020:28517] mca: base: components_open: component tcp open function successful
[mpi020:28517] mca: base: components_open: found loaded component vader
[mpi020:28517] mca: base: components_open: component vader open function successful
[mpi020:28517] select: initializing btl component self
[mpi020:28517] select: init of component self returned success
[mpi020:28518] mca: base: components_register: component tcp register function successful
[mpi019:22889] mca: base: components_register: component tcp register function successful
[mpi019:22889] mca: base: components_register: found loaded component vader
[mpi020:28517] select: initializing btl component tcp
[mpi020:28518] mca: base: components_register: found loaded component vader
[mpi020:28517] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[mpi019:22888] mca: base: components_register: component vader register function successful
[mpi019:22888] mca: base: components_open: opening btl components
[mpi019:22888] mca: base: components_open: found loaded component self
[mpi019:22888] mca: base: components_open: component self open function successful
[mpi019:22888] mca: base: components_open: found loaded component tcp
[mpi019:22888] mca: base: components_open: component tcp open function successful
[mpi019:22888] mca: base: components_open: found loaded component vader
[mpi019:22888] mca: base: components_open: component vader open function successful
[mpi019:22888] select: initializing btl component self
[mpi019:22888] select: init of component self returned success
[mpi019:22888] select: initializing btl component tcp
[mpi019:22888] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[mpi019:22888] btl: tcp: Found match: 127.0.0.1 (lo)
[mpi019:22889] mca: base: components_register: component vader register function successful
[mpi020:28517] btl: tcp: Found match: 127.0.0.1 (lo)
[mpi020:28518] mca: base: components_register: component vader register function successful
[mpi020:28518] mca: base: components_open: opening btl components
[mpi020:28518] mca: base: components_open: found loaded component self
[mpi020:28518] mca: base: components_open: component self open function successful
[mpi020:28518] mca: base: components_open: found loaded component tcp
[mpi020:28518] mca: base: components_open: component tcp open function successful
[mpi020:28518] mca: base: components_open: found loaded component vader
[mpi020:28517] btl:tcp: Attempting to bind to AF_INET port 1024
[mpi020:28517] btl:tcp: Attempting to bind to AF_INET port 1025
[mpi020:28517] btl:tcp: Attempting to bind to AF_INET port 1026
[mpi020:28517] btl:tcp: Attempting to bind to AF_INET port 1027
[mpi020:28517] btl:tcp: Successfully bound to AF_INET port 1027
[mpi020:28517] btl:tcp: my listening v4 socket is 0.0.0.0:1027
[mpi020:28517] btl:tcp: examining interface vic20
[mpi020:28517] btl:tcp: using ipv6 interface vic20
[mpi020:28517] btl:tcp: examining interface vic21
[mpi020:28517] btl:tcp: using ipv6 interface vic21
[mpi020:28517] btl:tcp: examining interface eth6
[mpi020:28517] btl:tcp: using ipv6 interface eth6
[mpi020:28518] mca: base: components_open: component vader open function successful
[mpi020:28518] select: initializing btl component self
[mpi020:28518] select: init of component self returned success
[mpi020:28518] select: initializing btl component tcp
[mpi020:28517] select: init of component tcp returned success
[mpi020:28517] select: initializing btl component vader
[mpi020:28518] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[mpi019:22889] mca: base: components_open: opening btl components
[mpi019:22889] mca: base: components_open: found loaded component self
[mpi019:22889] mca: base: components_open: component self open function successful
[mpi019:22889] mca: base: components_open: found loaded component tcp
[mpi019:22889] mca: base: components_open: component tcp open function successful
[mpi019:22889] mca: base: components_open: found loaded component vader
[mpi019:22889] mca: base: components_open: component vader open function successful
[mpi020:28518] btl: tcp: Found match: 127.0.0.1 (lo)
[mpi019:22888] btl:tcp: Attempting to bind to AF_INET port 1024
[mpi019:22888] btl:tcp: Attempting to bind to AF_INET port 1025
[mpi019:22888] btl:tcp: Successfully bound to AF_INET port 1025
[mpi019:22889] select: initializing btl component self
[mpi019:22889] select: init of component self returned success
[mpi019:22889] select: initializing btl component tcp
[mpi020:28518] btl:tcp: Attempting to bind to AF_INET port 1024
[mpi020:28518] btl:tcp: Attempting to bind to AF_INET port 1025
[mpi020:28518] btl:tcp: Attempting to bind to AF_INET port 1026
[mpi020:28518] btl:tcp: Attempting to bind to AF_INET port 1027
[mpi020:28518] btl:tcp: Attempting to bind to AF_INET port 1028
[mpi020:28518] btl:tcp: Successfully bound to AF_INET port 1028
[mpi020:28518] btl:tcp: my listening v4 socket is 0.0.0.0:1028
[mpi020:28518] btl:tcp: examining interface vic20
[mpi020:28518] btl:tcp: using ipv6 interface vic20
[mpi020:28518] btl:tcp: examining interface vic21
[mpi019:22888] btl:tcp: my listening v4 socket is 0.0.0.0:1025
[mpi019:22888] btl:tcp: examining interface vic20
[mpi020:28517] select: init of component vader returned success
[mpi019:22888] btl:tcp: using ipv6 interface vic20
[mpi019:22888] btl:tcp: examining interface vic21
[mpi019:22888] btl:tcp: using ipv6 interface vic21
[mpi019:22888] btl:tcp: examining interface eth6
[mpi019:22888] btl:tcp: using ipv6 interface eth6
[mpi020:28518] btl:tcp: using ipv6 interface vic21
[mpi020:28518] btl:tcp: examining interface eth6
[mpi020:28518] btl:tcp: using ipv6 interface eth6
[mpi020:28518] select: init of component tcp returned success
[mpi020:28518] select: initializing btl component vader
[mpi019:22889] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[mpi019:22889] btl: tcp: Found match: 127.0.0.1 (lo)
[mpi019:22888] select: init of component tcp returned success
[mpi019:22888] select: initializing btl component vader
[mpi019:22889] btl:tcp: Attempting to bind to AF_INET port 1024
[mpi019:22889] btl:tcp: Attempting to bind to AF_INET port 1025
[mpi019:22889] btl:tcp: Attempting to bind to AF_INET port 1026
[mpi019:22889] btl:tcp: Attempting to bind to AF_INET port 1027
[mpi019:22889] btl:tcp: Successfully bound to AF_INET port 1027
[mpi019:22889] btl:tcp: my listening v4 socket is 0.0.0.0:1027
[mpi019:22889] btl:tcp: examining interface vic20
[mpi019:22889] btl:tcp: using ipv6 interface vic20
[mpi019:22889] btl:tcp: examining interface vic21
[mpi019:22889] btl:tcp: using ipv6 interface vic21
[mpi019:22889] btl:tcp: examining interface eth6
[mpi019:22889] btl:tcp: using ipv6 interface eth6
[mpi020:28518] select: init of component vader returned success
[mpi019:22889] select: init of component tcp returned success
[mpi019:22889] select: initializing btl component vader
[mpi019:22888] select: init of component vader returned success
[mpi019:22889] select: init of component vader returned success
[mpi020:28518] mca: bml: Using self btl for send to [[28727,1],3] on node mpi020
[mpi020:28517] mca: bml: Using self btl for send to [[28727,1],2] on node mpi020
[mpi019:22888] mca: bml: Using self btl for send to [[28727,1],0] on node mpi019
[mpi020:28518] mca: bml: Using vader btl for send to [[28727,1],2] on node mpi020
[mpi019:22889] mca: bml: Using self btl for send to [[28727,1],1] on node mpi019
[mpi020:28517] mca: bml: Using vader btl for send to [[28727,1],3] on node mpi020
[mpi019:22888] mca: bml: Using vader btl for send to [[28727,1],1] on node mpi019
[mpi019:22889] mca: bml: Using vader btl for send to [[28727,1],0] on node mpi019
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
MPITEST info (0): Starting MPI_Isend_rtoa: Root Isends TO All test
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] mca: bml: Using tcp btl for send to [[28727,1],2] on node (null)
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] mca: bml: Using tcp btl for send to [[28727,1],2] on node (null)
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] mca: bml: Using tcp btl for send to [[28727,1],2] on node (null)
[mpi019:22888] btl: tcp: attempting to connect() to [[28727,1],2] address 10.2.0.20 on port 1027
[mpi019:22888] btl:tcp: would block, so allowing background progress
[mpi019:22888] btl:tcp: connect() to 10.2.0.20:1027 completed (complete_connect), sending connect ACK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl: tcp: Match incoming connection from [[28727,1],0] 10.2.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28517] btl:tcp: now connected to 10.2.0.19, process [[28727,1],0]
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] mca: bml: Using tcp btl for send to [[28727,1],3] on node (null)
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] mca: bml: Using tcp btl for send to [[28727,1],3] on node (null)
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22888] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22888] mca: bml: Using tcp btl for send to [[28727,1],3] on node (null)
[mpi019:22888] btl: tcp: attempting to connect() to [[28727,1],3] address 10.2.0.20 on port 1028
[mpi019:22888] btl:tcp: would block, so allowing background progress
[mpi019:22888] btl:tcp: connect() to 10.2.0.20:1028 completed (complete_connect), sending connect ACK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl: tcp: Match incoming connection from [[28727,1],0] 10.2.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28518] btl:tcp: now connected to 10.2.0.19, process [[28727,1],0]
[mpi019:22888] btl: tcp: attempting to connect() to [[28727,1],2] address 10.10.0.20 on port 1027
[mpi019:22888] btl:tcp: would block, so allowing background progress
[mpi019:22888] btl:tcp: connect() to 10.10.0.20:1027 completed (complete_connect), sending connect ACK
[mpi020:28517] mca: bml: Using tcp btl for send to [[28727,1],0] on node (null)
[mpi020:28517] mca: bml: Using tcp btl for send to [[28727,1],0] on node (null)
[mpi020:28517] mca: bml: Using tcp btl for send to [[28727,1],0] on node (null)
[mpi020:28517] btl:tcp: now connected to 10.10.0.19, process [[28727,1],0]
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] mca: bml: Using tcp btl for send to [[28727,1],3] on node (null)
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] mca: bml: Using tcp btl for send to [[28727,1],1] on node (null)
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] mca: bml: Using tcp btl for send to [[28727,1],3] on node (null)
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] mca: bml: Using tcp btl for send to [[28727,1],1] on node (null)
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28518] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28518] mca: bml: Using tcp btl for send to [[28727,1],1] on node (null)
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] mca: bml: Using tcp btl for send to [[28727,1],3] on node (null)
[mpi020:28518] btl: tcp: attempting to connect() to [[28727,1],1] address 10.2.0.19 on port 1027
[mpi019:22889] btl: tcp: attempting to connect() to [[28727,1],3] address 10.2.0.20 on port 1028
[mpi019:22889] btl:tcp: would block, so allowing background progress
[mpi020:28518] btl:tcp: would block, so allowing background progress
[mpi020:28518] btl:tcp: connect() to 10.2.0.19:1027 completed (complete_connect), sending connect ACK
[mpi019:22889] btl:tcp: connect() to 10.2.0.20:1028 completed (complete_connect), sending connect ACK
[mpi019:22889] btl: tcp: Match incoming connection from [[28727,1],3] 10.2.0.20 with locally known IP 10.10.0.20 failed (iface 0/3)!
[mpi019:22889] btl:tcp: now connected to 10.2.0.20, process [[28727,1],3]
[mpi020][[28727,1],3][../../../../../opal/mca/btl/tcp/btl_tcp.c:549:mca_btl_tcp_recv_blocking] remote peer unexpectedly closed connection while I was waiting for blocking message
[mpi020:28518] btl: tcp: Match incoming connection from [[28727,1],1] 10.2.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28518] btl:tcp: now connected to 10.2.0.19, process [[28727,1],1]
--------------------------------------------------------------------------
WARNING: Open MPI failed to handshake with a connecting peer MPI
process over TCP. This should not happen.
Your Open MPI job may now fail.
Local host: mpi020
PID: 28518
Message: did not receive entire connect ACK from peer
--------------------------------------------------------------------------
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] mca: bml: Using tcp btl for send to [[28727,1],2] on node (null)
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] mca: bml: Using tcp btl for send to [[28727,1],2] on node (null)
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.10.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.10.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.2.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] btl:tcp: path from 10.2.0.19 to 10.3.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.10.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.2.0.20: IPV4 PRIVATE DIFFERENT NETWORK
[mpi019:22889] btl:tcp: path from 10.3.0.19 to 10.3.0.20: IPV4 PRIVATE SAME NETWORK
[mpi019:22889] mca: bml: Using tcp btl for send to [[28727,1],2] on node (null)
[mpi019:22889] btl: tcp: attempting to connect() to [[28727,1],2] address 10.2.0.20 on port 1027
[mpi019:22889] btl:tcp: would block, so allowing background progress
[mpi019:22889] btl:tcp: connect() to 10.2.0.20:1027 completed (complete_connect), sending connect ACK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.10.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.10.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.2.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl:tcp: path from 10.2.0.20 to 10.3.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.10.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.2.0.19: IPV4 PRIVATE DIFFERENT NETWORK
[mpi020:28517] btl:tcp: path from 10.3.0.20 to 10.3.0.19: IPV4 PRIVATE SAME NETWORK
[mpi020:28517] btl: tcp: Match incoming connection from [[28727,1],1] 10.2.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28517] btl:tcp: now connected to 10.2.0.19, process [[28727,1],1]
[mpi019:22889] btl: tcp: attempting to connect() to [[28727,1],3] address 10.10.0.20 on port 1028
[mpi019:22889] btl:tcp: would block, so allowing background progress
[mpi019:22889] btl:tcp: connect() to 10.10.0.20:1028 completed (complete_connect), sending connect ACK
[mpi020:28518] btl:tcp: now connected to 10.10.0.19, process [[28727,1],1]
[mpi019:22888] btl: tcp: attempting to connect() to [[28727,1],2] address 10.3.0.20 on port 1027
[mpi019:22888] btl:tcp: would block, so allowing background progress
[mpi019:22889] btl: tcp: attempting to connect() to [[28727,1],3] address 10.3.0.20 on port 1028
[mpi019:22889] btl:tcp: would block, so allowing background progress
[mpi019:22889] btl:tcp: connect() to 10.3.0.20:1028 completed (complete_connect), sending connect ACK
[mpi019:22888] btl:tcp: connect() to 10.3.0.20:1027 completed (complete_connect), sending connect ACK
[mpi020:28517] btl: tcp: Match incoming connection from [[28727,1],0] 10.3.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28517] btl: tcp: Match incoming connection from [[28727,1],0] 10.3.0.19 with locally known IP 10.2.0.19 failed (iface 1/3)!
[mpi020:28517] btl:tcp: now connected to 10.3.0.19, process [[28727,1],0]
[mpi020:28518] btl: tcp: Match incoming connection from [[28727,1],1] 10.3.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28518] btl: tcp: Match incoming connection from [[28727,1],1] 10.3.0.19 with locally known IP 10.2.0.19 failed (iface 1/3)!
[mpi020:28518] btl:tcp: now connected to 10.3.0.19, process [[28727,1],1]
[mpi020:28517] mca: bml: Using tcp btl for send to [[28727,1],1] on node (null)
[mpi020:28517] mca: bml: Using tcp btl for send to [[28727,1],1] on node (null)
[mpi020:28517] mca: bml: Using tcp btl for send to [[28727,1],1] on node (null)
[mpi020:28518] mca: bml: Using tcp btl for send to [[28727,1],0] on node (null)
[mpi020:28518] mca: bml: Using tcp btl for send to [[28727,1],0] on node (null)
[mpi020:28518] mca: bml: Using tcp btl for send to [[28727,1],0] on node (null)
[mpi019:22888] btl: tcp: attempting to connect() to [[28727,1],3] address 10.10.0.20 on port 1028
[mpi019:22888] btl:tcp: would block, so allowing background progress
[mpi019:22888] btl:tcp: connect() to 10.10.0.20:1028 completed (complete_connect), sending connect ACK
[mpi020:28518] btl:tcp: now connected to 10.10.0.19, process [[28727,1],0]
[mpi019:22889] btl: tcp: attempting to connect() to [[28727,1],2] address 10.10.0.20 on port 1027
[mpi019:22889] btl:tcp: would block, so allowing background progress
[mpi019:22889] btl:tcp: connect() to 10.10.0.20:1027 completed (complete_connect), sending connect ACK
[mpi020:28517] btl:tcp: now connected to 10.10.0.19, process [[28727,1],1]
[mpi019:22888] btl: tcp: attempting to connect() to [[28727,1],3] address 10.3.0.20 on port 1028
[mpi019:22888] btl:tcp: would block, so allowing background progress
[mpi019:22888] btl:tcp: connect() to 10.3.0.20:1028 completed (complete_connect), sending connect ACK
[mpi020:28518] btl: tcp: Match incoming connection from [[28727,1],0] 10.3.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28518] btl: tcp: Match incoming connection from [[28727,1],0] 10.3.0.19 with locally known IP 10.2.0.19 failed (iface 1/3)!
[mpi020:28518] btl:tcp: now connected to 10.3.0.19, process [[28727,1],0]
[mpi019:22889] btl: tcp: attempting to connect() to [[28727,1],2] address 10.3.0.20 on port 1027
[mpi019:22889] btl:tcp: would block, so allowing background progress
[mpi019:22889] btl:tcp: connect() to 10.3.0.20:1027 completed (complete_connect), sending connect ACK
[mpi020:28517] btl: tcp: Match incoming connection from [[28727,1],1] 10.3.0.19 with locally known IP 10.10.0.19 failed (iface 0/3)!
[mpi020:28517] btl: tcp: Match incoming connection from [[28727,1],1] 10.3.0.19 with locally known IP 10.2.0.19 failed (iface 1/3)!
[mpi020:28517] btl:tcp: now connected to 10.3.0.19, process [[28727,1],1]
MPITEST_results: MPI_Isend_rtoa: Root Isends TO All all tests PASSED (19656)
[mpi020:28517] mca: base: close: component self closed
[mpi020:28517] mca: base: close: unloading component self
[mpi020:28517] mca: base: close: component tcp closed
[mpi020:28517] mca: base: close: unloading component tcp
[mpi019:22888] mca: base: close: component self closed
[mpi019:22888] mca: base: close: unloading component self
[mpi020:28517] mca: base: close: component vader closed
[mpi020:28517] mca: base: close: unloading component vader
[mpi019:22889] mca: base: close: component self closed
[mpi019:22889] mca: base: close: unloading component self
[mpi020:28518] mca: base: close: component self closed
[mpi020:28518] mca: base: close: unloading component self
[mpi019:22888] mca: base: close: component tcp closed
[mpi019:22888] mca: base: close: unloading component tcp
[mpi019:22888] mca: base: close: component vader closed
[mpi019:22888] mca: base: close: unloading component vader
[mpi020:28518] mca: base: close: component tcp closed
[mpi020:28518] mca: base: close: unloading component tcp
[mpi020:28518] mca: base: close: component vader closed
[mpi020:28518] mca: base: close: unloading component vader
[mpi019:22889] mca: base: close: component tcp closed
[mpi019:22889] mca: base: close: unloading component tcp
[mpi019:22889] mca: base: close: component vader closed
[mpi019:22889] mca: base: close: unloading component vader
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment