Skip to content

Instantly share code, notes, and snippets.

@hiroyuki-sato

hiroyuki-sato/MLNX_OFED.md

Last active December 17, 2020 16:15
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save hiroyuki-sato/b8f9023c9667a956ed5f to your computer and use it in GitHub Desktop.
Save hiroyuki-sato/b8f9023c9667a956ed5f to your computer and use it in GitHub Desktop.
Download ZIP
MLNX OFED setting on CentOS7
Raw
MLNX_OFED.md

I want to fix this problem iSER connection disconnected

When I test iSER connection over RoCE. the following value are increased And iSER connection disconnected.

Target side enp3s0 interface error/drop count.

metrics val
rx_errors 0
rx_fifo_errors 293
rx_frame_errors 0
rx_length_errors 0
rx_missed_errors 293
rx_over_errors 293
tx_aborted_errors 0
tx_carrier_errors 0
tx_errors 0
tx_fifo_errors 0
tx_heartbeat_errors 0
tx_window_errors 0
rx_dropped 293
tx_dropped 0

image

  • CentOS Linux release 7.1.1503
  • MLNX_OFED: MLNX_OFED_LINUX-3.0-2.0.1 (OFED-3.0-2.0.0):
  • Switch: SX_PPC_M460EX / Product release: SX_3.4.0008
  • VLAN 211 (for storage vlan)
  • Servers are connect to ports 15 and 16.

Switch PFC configuration

##
## DCBX PFC configuration
##
   dcb priority-flow-control enable force
   dcb priority-flow-control priority 3 enable
   interface ethernet 1/15 dcb priority-flow-control mode on force
   interface ethernet 1/16 dcb priority-flow-control mode on force
   interface ethernet 1/17 dcb priority-flow-control mode on force
   interface ethernet 1/18 dcb priority-flow-control mode on force
##
## Interface Ethernet configuration
##
   interface ethernet 1/15 mtu 4200 force
   interface ethernet 1/16 mtu 4200 force
   interface ethernet 1/15 switchport mode hybrid
   interface ethernet 1/16 switchport mode hybrid
   interface ethernet 1/17 switchport mode hybrid
   interface ethernet 1/18 switchport mode hybrid
   
##
## VLAN configuration
##
   vlan 100-101
   vlan 211-213
   interface ethernet 1/15 switchport hybrid allowed-vlan all
   interface ethernet 1/16 switchport hybrid allowed-vlan all
   interface ethernet 1/18 switchport hybrid allowed-vlan all

verify dcb priority check

#  show dcb priority-flow-control

PFC enabled
Priority Enabled List    :3 
Priority Disabled List   :0 1 2 4 5 6 7 

TC     Lossless
---    ----------
0           N
1           Y
2           Y
3           N


Interface      PFC admin        PFC oper
------------   --------------   -------------
Eth1/1           Disabled         Disabled
Eth1/2           Disabled         Disabled
Eth1/15          On               Enabled
Eth1/16          On               Enabled
Eth1/17          On               Enabled
Eth1/18          On               Enabled

OS Settings

Enable pfcrx/pfctx

echo "options mlx4_en pfctx=0x08 pfcrx=0x08" > /etc/modprobe.d/mlx4_en.conf

systemctl restart openibd

/sys/module/mlx4_en/parameters/pfc*
8
8

vconfig (no need?)

for i in {0..7}; do vconfig set_egress_map enp3s0.211 $i 3 ; done

tc_wrap.py (skprio change)

Before

# tc_wrap.py -i enp3s0
UP  0
	skprio: 0
	skprio: 1
	skprio: 2 (tos: 8)
	skprio: 3
	skprio: 4 (tos: 24)
	skprio: 5
	skprio: 6 (tos: 16)
	skprio: 7
	skprio: 8
	skprio: 9
	skprio: 10
	skprio: 11
	skprio: 12
	skprio: 13
	skprio: 14
	skprio: 15
UP  1
UP  2
UP  3
	skprio: 0 (vlan 211)
	skprio: 1 (vlan 211)
	skprio: 2 (vlan 211 tos: 8)
	skprio: 3 (vlan 211)
	skprio: 4 (vlan 211 tos: 24)
	skprio: 5 (vlan 211)
	skprio: 6 (vlan 211 tos: 16)
	skprio: 7 (vlan 211)
UP  4
UP  5
UP  6
UP  7

After

# tc_wrap.py -i enp3s0 -u 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3
UP  0
UP  1
UP  2
UP  3
	skprio: 0
	skprio: 1
	skprio: 2 (tos: 8)
	skprio: 3
	skprio: 4 (tos: 24)
	skprio: 5
	skprio: 6 (tos: 16)
	skprio: 7
	skprio: 8
	skprio: 9
	skprio: 10
	skprio: 11
	skprio: 12
	skprio: 13
	skprio: 14
	skprio: 15
	skprio: 0 (vlan 211)
	skprio: 1 (vlan 211)
	skprio: 2 (vlan 211 tos: 8)
	skprio: 3 (vlan 211)
	skprio: 4 (vlan 211 tos: 24)
	skprio: 5 (vlan 211)
	skprio: 6 (vlan 211 tos: 16)
	skprio: 7 (vlan 211)
UP  4
UP  5
UP  6
UP  7

check

udaddy

udaddy
udaddy: starting server
receiving data transfers
sending replies
data transfers complete
test complete
return status 0

udaddy -s 192.168.5.211
udaddy: starting client
udaddy: connecting
initiating data transfers
receiving data transfers
data transfers complete
test complete
return status 0

ib_write_bw

ib_write_bw -R --report_gbits --port=12500 -D 10 

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx4_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 2048[B]
 Link type       : Ethernet
 Gid index       : 0
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 Waiting for client rdma_cm QP to connect
 Please run the same command with the IB/RoCE interface IP
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0254 PSN 0x6c43a
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:30:210:10
 remote address: LID 0000 QPN 0x0254 PSN 0x5aa001
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:30:210:20
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      415300           0.00               36.29  		   0.069209
---------------------------------------------------------------------------------------

ib_write_bw -F -R --report_gbits 172.30.211.10  --port=12500 -D 10
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx4_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 2048[B]
 Link type       : Ethernet
 Gid index       : 0
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0254 PSN 0x5aa001
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:30:210:20
 remote address: LID 0000 QPN 0x0254 PSN 0x6c43a
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:30:210:10
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 2601.457000 != 2432.308000
Test integrity may be harmed !
Warning: measured timestamp frequency 2099.76 differs from nominal 2601.46 MHz
 65536      415300           0.00               36.29  		   0.069209
---------------------------------------------------------------------------------------

ibv_devinfo

ibv_devinfo
hca_id:	mlx4_0
	transport:			InfiniBand (0)
	fw_ver:				2.32.5100
	node_guid:			0025:90ff:ffdf:82b8
	sys_image_guid:			0025:90ff:ffdf:82bb
	vendor_id:			0x02c9
	vendor_part_id:			4099
	hw_ver:				0x1
	board_id:			SM_2241000001000
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet


show interfaces ethernet 1/15 counters priority 3

Rx
  165411805            packets
  165411803            unicast packets
  0                    multicast packets
  2                    broadcast packets
  185055775152         bytes
  0                    pause packets
  0                    pause duration milliseconds

Tx
  7360982              packets
  7360982              unicast packets
  0                    multicast packets
  0                    broadcast packets
  864278904            bytes
  2193                 pause packets
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment