Created
August 24, 2013 16:43
-
-
Save Jerry-Fix/6329138 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| I spend some time to have some test, all are redhat 6.2, here is the test. | |
| 1. Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet(bnx2), host8, E5606x2 | |
| # lspci | grep net | |
| 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) | |
| 01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) | |
| 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) | |
| 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) | |
| # lspci -vvvv | |
| 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) | |
| Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet | |
| Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ | |
| Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- | |
| Latency: 0, Cache Line Size: 64 bytes | |
| Interrupt: pin A routed to IRQ 36 | |
| Region 0: Memory at d6000000 (64-bit, non-prefetchable) [size=32M] | |
| Capabilities: [48] Power Management version 3 | |
| Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) | |
| Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- | |
| Capabilities: [50] Vital Product Data | |
| Product Name: Broadcom NetXtreme II Ethernet Controller | |
| Read-only fields: | |
| [PN] Part number: BCM95709C0 | |
| [EC] Engineering changes: 220197-2 | |
| [SN] Serial number: 0123456789 | |
| [MN] Manufacture ID: 31 30 32 38 | |
| [V0] Vendor specific: 6.2.12 | |
| [RV] Reserved: checksum good, 22 byte(s) reserved | |
| End | |
| Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+ | |
| Address: 0000000000000000 Data: 0000 | |
| Capabilities: [a0] MSI-X: Enable+ Count=9 Masked- | |
| Vector table: BAR=0 offset=0000c000 | |
| PBA: BAR=0 offset=0000e000 | |
| Capabilities: [ac] Express (v2) Endpoint, MSI 00 | |
| DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us | |
| ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- | |
| DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+ | |
| RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ | |
| MaxPayload 256 bytes, MaxReadReq 512 bytes | |
| DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- | |
| LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <2us | |
| ClockPM- Surprise- LLActRep- BwNot- | |
| LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ | |
| ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- | |
| LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- | |
| DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ | |
| DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- | |
| LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB | |
| Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- | |
| Compliance De-emphasis: -6dB | |
| LnkSta2: Current De-emphasis Level: -6dB | |
| Capabilities: [100] Device Serial Number 14-fe-b5-ff-fe-da-84-0f | |
| Capabilities: [110] Advanced Error Reporting | |
| UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- | |
| CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ | |
| CEMsk: RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ | |
| AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- | |
| Capabilities: [150] Power Budgeting <?> | |
| Capabilities: [160] Virtual Channel <?> | |
| Kernel driver in use: bnx2 | |
| Kernel modules: bnx2 | |
| # lscpu | |
| Architecture: x86_64 | |
| CPU op-mode(s): 32-bit, 64-bit | |
| Byte Order: Little Endian | |
| CPU(s): 8 | |
| On-line CPU(s) list: 0-7 | |
| Thread(s) per core: 1 | |
| Core(s) per socket: 4 | |
| CPU socket(s): 2 | |
| NUMA node(s): 2 | |
| Vendor ID: GenuineIntel | |
| CPU family: 6 | |
| Model: 44 | |
| Stepping: 2 | |
| CPU MHz: 2128.094 | |
| BogoMIPS: 4255.89 | |
| Virtualization: VT-x | |
| L1d cache: 32K | |
| L1i cache: 32K | |
| L2 cache: 256K | |
| L3 cache: 8192K | |
| NUMA node0 CPU(s): 0,2,4,6 | |
| NUMA node1 CPU(s): 1,3,5,7 | |
| # cat /proc/interrupts | grep em | |
| 61: 9578 19 0 0 0 0 0 0 PCI-MSI-edge em1-0 | |
| 62: 4771 254 0 0 0 0 0 0 PCI-MSI-edge em1-1 | |
| 63: 29 1103 0 0 0 0 0 0 PCI-MSI-edge em1-2 | |
| 64: 32 1118 0 0 0 0 0 0 PCI-MSI-edge em1-3 | |
| 65: 108444 74 0 0 0 0 0 0 PCI-MSI-edge em1-4 | |
| 66: 27 2074 0 0 0 0 0 0 PCI-MSI-edge em1-5 | |
| 67: 1539 0 0 0 0 0 0 0 PCI-MSI-edge em1-6 | |
| 68: 24 2330 0 0 0 0 0 0 PCI-MSI-edge em1-7 | |
| # echo 0-7 > /proc/irq/61/smp_affinity_list | |
| # echo 0-7 > /proc/irq/62/smp_affinity_list | |
| # echo 0-7 > /proc/irq/63/smp_affinity_list | |
| # echo 0-7 > /proc/irq/64/smp_affinity_list | |
| # echo 0-7 > /proc/irq/65/smp_affinity_list | |
| # echo 0-7 > /proc/irq/66/smp_affinity_list | |
| # echo 0-7 > /proc/irq/67/smp_affinity_list | |
| # echo 0-7 > /proc/irq/68/smp_affinity_list | |
| # cat /proc/irq/6?/smp_affinity_list | |
| 0-7 | |
| 0-7 | |
| 0-7 | |
| 0-7 | |
| 0-7 | |
| 0-7 | |
| 0-7 | |
| 0-7 | |
| # cat /proc/irq/6?/smp_affinity | |
| 000000ff | |
| 000000ff | |
| 000000ff | |
| 000000ff | |
| 000000ff | |
| 000000ff | |
| 000000ff | |
| 000000ff | |
| Create traffic using iperf with -P parameter(multi thread), and the interrupts did not distrubuted as I expected, they are just like the below test as irq 61 for core 0, irq 62 for core 1... What I expect to see is irq 61 is distributed from core 0 to core 7, irq 62 is also distrubuted from core 0 to core 7, not to a dedicated core. | |
| Then, I bind each queue(interrupt) to each core: | |
| # echo 0 > /proc/irq/61/smp_affinity_list | |
| # echo 1 > /proc/irq/62/smp_affinity_list | |
| # echo 2 > /proc/irq/63/smp_affinity_list | |
| # echo 3 > /proc/irq/64/smp_affinity_list | |
| # echo 4 > /proc/irq/65/smp_affinity_list | |
| # echo 5 > /proc/irq/66/smp_affinity_list | |
| # echo 6 > /proc/irq/67/smp_affinity_list | |
| # echo 7 > /proc/irq/68/smp_affinity_list | |
| # cat /proc/irq/6?/smp_affinity_list | |
| 0 | |
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| # cat /proc/irq/6?/smp_affinity | |
| 00000001 | |
| 00000002 | |
| 00000004 | |
| 00000008 | |
| 00000010 | |
| 00000020 | |
| 00000040 | |
| 00000080 | |
| The result is as I expected, irq 61 dedicated to core 0, irq 62 dedicated to core 1... | |
| So test 1 is really confusing, and test 2 is clear. | |
| 2. Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet(bnx2), host5, E5606x2 | |
| # lspci | grep net | |
| 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20) | |
| 01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20) | |
| # lspci -vvv | |
| 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20) | |
| Subsystem: Dell PowerEdge R410 BCM5716 Gigabit Ethernet | |
| Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ | |
| Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- | |
| Latency: 0, Cache Line Size: 64 bytes | |
| Interrupt: pin A routed to IRQ 36 | |
| Region 0: Memory at da000000 (64-bit, non-prefetchable) [size=32M] | |
| Capabilities: [48] Power Management version 3 | |
| Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) | |
| Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- | |
| Capabilities: [50] Vital Product Data | |
| Product Name: Broadcom NetXtreme II Ethernet Controller | |
| Read-only fields: | |
| [PN] Part number: BCM95716C1 | |
| [EC] Engineering changes: 220197-3 | |
| [SN] Serial number: 0123456789 | |
| [MN] Manufacture ID: 31 30 32 38 | |
| [V0] Vendor specific: 6.2.12 | |
| [RV] Reserved: checksum good, 22 byte(s) reserved | |
| End | |
| Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+ | |
| Address: 0000000000000000 Data: 0000 | |
| Capabilities: [a0] MSI-X: Enable+ Count=9 Masked- | |
| Vector table: BAR=0 offset=0000c000 | |
| PBA: BAR=0 offset=0000e000 | |
| Capabilities: [ac] Express (v2) Endpoint, MSI 00 | |
| DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us | |
| ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- | |
| DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+ | |
| RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ | |
| MaxPayload 256 bytes, MaxReadReq 512 bytes | |
| DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- | |
| LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <2us | |
| ClockPM- Surprise- LLActRep- BwNot- | |
| LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ | |
| ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- | |
| LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- | |
| DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ | |
| DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- | |
| LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB | |
| Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- | |
| Compliance De-emphasis: -6dB | |
| LnkSta2: Current De-emphasis Level: -6dB | |
| Capabilities: [100] Device Serial Number 78-2b-cb-ff-fe-43-b5-08 | |
| Capabilities: [110] Advanced Error Reporting | |
| UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- | |
| CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ | |
| CEMsk: RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ | |
| AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- | |
| Capabilities: [150] Power Budgeting <?> | |
| Capabilities: [160] Virtual Channel <?> | |
| Kernel driver in use: bnx2 | |
| Kernel modules: bnx2 | |
| # lscpu | |
| Architecture: x86_64 | |
| CPU op-mode(s): 32-bit, 64-bit | |
| CPU(s): 8 | |
| Thread(s) per core: 1 | |
| Core(s) per socket: 4 | |
| CPU socket(s): 2 | |
| NUMA node(s): 2 | |
| Vendor ID: GenuineIntel | |
| CPU family: 6 | |
| Model: 44 | |
| Stepping: 2 | |
| CPU MHz: 2127.930 | |
| Virtualization: VT-x | |
| L1d cache: 32K | |
| L1i cache: 32K | |
| L2 cache: 256K | |
| L3 cache: 8192K | |
| NUMA node0 CPU(s): 0,2,4,6 | |
| NUMA node1 CPU(s): 1,3,5,7 | |
| # cat /proc/interrupts | grep eth | |
| 59: 433533 0 0 1 0 12691162 0 0 PCI-MSI-edge eth0-0 | |
| 60: 7931736 1525662 12127904 0 53 0 70 58412 PCI-MSI-edge eth0-1 | |
| 61: 114719 3 626859 21945507 0 0 0 0 PCI-MSI-edge eth0-2 | |
| 62: 59 0 324 802009 179 148494 21672640 0 PCI-MSI-edge eth0-3 | |
| 63: 30 71 80 153663 21539313 33 0 0 PCI-MSI-edge eth0-4 | |
| 64: 21 4 0 0 0 778691 0 25031761 PCI-MSI-edge eth0-5 | |
| 65: 21 110866 52144714 5 0 0 813106 6 PCI-MSI-edge eth0-6 | |
| 66: 23 27450909 0 0 0 0 0 1679245 PCI-MSI-edge eth0-7 | |
| The result is similar to the above, when I set like this, does not work as expected: | |
| # echo 0-7 > /proc/irq/59/smp_affinity_list | |
| # echo 0-7 > /proc/irq/60/smp_affinity_list | |
| # echo 0-7 > /proc/irq/61/smp_affinity_list | |
| # echo 0-7 > /proc/irq/62/smp_affinity_list | |
| # echo 0-7 > /proc/irq/63/smp_affinity_list | |
| # echo 0-7 > /proc/irq/64/smp_affinity_list | |
| # echo 0-7 > /proc/irq/65/smp_affinity_list | |
| # echo 0-7 > /proc/irq/66/smp_affinity_list | |
| When I set like this, it is ok: | |
| #echo 0 > /proc/irq/59/smp_affinity_list | |
| #echo 1 > /proc/irq/60/smp_affinity_list | |
| #echo 2 > /proc/irq/61/smp_affinity_list | |
| #echo 3 > /proc/irq/62/smp_affinity_list | |
| #echo 4 > /proc/irq/63/smp_affinity_list | |
| #echo 5 > /proc/irq/64/smp_affinity_list | |
| #echo 6 > /proc/irq/65/smp_affinity_list | |
| #echo 7 > /proc/irq/66/smp_affinity_list | |
| 3. Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe (tg3), hostlvs1, E5-2430x2 | |
| # lspci | grep net | |
| 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe | |
| 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe | |
| # lspci -vvv | |
| 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe | |
| Subsystem: Dell Device 04f8 | |
| Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ | |
| Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- | |
| Latency: 0, Cache Line Size: 64 bytes | |
| Interrupt: pin A routed to IRQ 16 | |
| Region 0: Memory at d90a0000 (64-bit, prefetchable) [size=64K] | |
| Region 2: Memory at d90b0000 (64-bit, prefetchable) [size=64K] | |
| Region 4: Memory at d90c0000 (64-bit, prefetchable) [size=64K] | |
| Expansion ROM at dd000000 [disabled] [size=256K] | |
| Capabilities: [48] Power Management version 3 | |
| Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) | |
| Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- | |
| Capabilities: [50] Vital Product Data | |
| Product Name: Broadcom NetXtreme Gigabit Ethernet | |
| Read-only fields: | |
| [PN] Part number: BCM95720 | |
| [MN] Manufacture ID: 31 30 32 38 | |
| [V0] Vendor specific: FFV7.4.8 | |
| [V1] Vendor specific: DSV1028VPDR.VER1.0 | |
| [V2] Vendor specific: NPY2 | |
| [V3] Vendor specific: PMT1 | |
| [V4] Vendor specific: NMVBroadcom Corp | |
| [V5] Vendor specific: DTINIC | |
| [V6] Vendor specific: DCM1001008d452101008d45 | |
| [RV] Reserved: checksum good, 235 byte(s) reserved | |
| End | |
| Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+ | |
| Address: 0000000000000000 Data: 0000 | |
| Capabilities: [a0] MSI-X: Enable+ Count=17 Masked- | |
| Vector table: BAR=4 offset=00000000 | |
| PBA: BAR=4 offset=00001000 | |
| Capabilities: [ac] Express (v2) Endpoint, MSI 00 | |
| DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us | |
| ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ | |
| DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+ | |
| RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop- FLReset- | |
| MaxPayload 128 bytes, MaxReadReq 2048 bytes | |
| DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- | |
| LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <1us, L1 <2us | |
| ClockPM+ Surprise- LLActRep- BwNot- | |
| LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ | |
| ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- | |
| LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- | |
| DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ | |
| DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- | |
| LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB | |
| Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- | |
| Compliance De-emphasis: -6dB | |
| LnkSta2: Current De-emphasis Level: -6dB | |
| Capabilities: [100] Advanced Error Reporting | |
| UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- | |
| CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ | |
| CEMsk: RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ | |
| AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- | |
| Capabilities: [13c] Device Serial Number 00-00-90-b1-1c-37-9f-07 | |
| Capabilities: [150] Power Budgeting <?> | |
| Capabilities: [160] Virtual Channel <?> | |
| Kernel driver in use: tg3 | |
| Kernel modules: tg3 | |
| # lscpu | |
| Architecture: x86_64 | |
| CPU op-mode(s): 32-bit, 64-bit | |
| Byte Order: Little Endian | |
| CPU(s): 24 | |
| On-line CPU(s) list: 0-23 | |
| Thread(s) per core: 2 | |
| Core(s) per socket: 6 | |
| CPU socket(s): 2 | |
| NUMA node(s): 2 | |
| Vendor ID: GenuineIntel | |
| CPU family: 6 | |
| Model: 45 | |
| Stepping: 7 | |
| CPU MHz: 2199.539 | |
| BogoMIPS: 4399.41 | |
| Virtualization: VT-x | |
| L1d cache: 32K | |
| L1i cache: 32K | |
| L2 cache: 256K | |
| L3 cache: 15360K | |
| NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22 | |
| NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23 | |
| # cat /proc/interrupts | grep em1 | |
| 104: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-0 | |
| 105: 20208495 1168670 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-1 | |
| 106: 4851065 0 364270 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-2 | |
| 107: 20 6879492 0 332342 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-3 | |
| 108: 4253731 428386 0 0 518654 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-4 | |
| When I set like this: | |
| # cat /proc/irq/10[4-8]/smp_affinity_list | |
| 0 | |
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| There aren't any interrupts to core 0 at all, but core 1 to core 4 just work as expected. | |
| When I set as this: | |
| # cat /proc/irq/10[4-8]/smp_affinity_list | |
| 0-23 | |
| 0-23 | |
| 0-23 | |
| 0-23 | |
| 0-23 | |
| The same as the test 1, really strange. | |
| 4. Intel Corporation I350 Gigabit Network Connection(igb), host23, E5-2630x2 | |
| # lspci | grep net | |
| 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) | |
| 01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) | |
| 01:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) | |
| 01:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) | |
| # lspci -vvv | |
| 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) | |
| Subsystem: Dell Intel GbE 4P I350crNDC | |
| Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ | |
| Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- | |
| Latency: 0, Cache Line Size: 64 bytes | |
| Interrupt: pin D routed to IRQ 38 | |
| Region 0: Memory at dcb00000 (32-bit, non-prefetchable) [size=1M] | |
| Region 2: I/O ports at fc80 [size=32] | |
| Region 3: Memory at dcff0000 (32-bit, non-prefetchable) [size=16K] | |
| Expansion ROM at dc000000 [disabled] [size=512K] | |
| Capabilities: [40] Power Management version 3 | |
| Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) | |
| Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- | |
| Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ | |
| Address: 0000000000000000 Data: 0000 | |
| Masking: 00000000 Pending: 00000000 | |
| Capabilities: [70] MSI-X: Enable+ Count=10 Masked- | |
| Vector table: BAR=3 offset=00000000 | |
| PBA: BAR=3 offset=00002000 | |
| Capabilities: [a0] Express (v2) Endpoint, MSI 00 | |
| DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us | |
| ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ | |
| DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+ | |
| RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- | |
| MaxPayload 256 bytes, MaxReadReq 512 bytes | |
| DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- | |
| LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <32us | |
| ClockPM- Surprise- LLActRep- BwNot- | |
| LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ | |
| ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- | |
| LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- | |
| DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ | |
| DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- | |
| LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB | |
| Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- | |
| Compliance De-emphasis: -6dB | |
| LnkSta2: Current De-emphasis Level: -6dB | |
| Capabilities: [e0] Vital Product Data | |
| Product Name: I350 GbE Controller | |
| Read-only fields: | |
| [V0] Vendor specific: FFV13.5.6 | |
| [PN] Part number: G10565 | |
| [MN] Manufacture ID: 31 30 32 38 | |
| [V1] Vendor specific: DSV1028VPDR.VER1.0 | |
| [V3] Vendor specific: DTINIC | |
| [V4] Vendor specific: DCM1001008FC52101008FC53201008FC54301008FC5 | |
| [V5] Vendor specific: NPY4 | |
| [V6] Vendor specific: PMT13 | |
| [V7] Vendor specific: NMVIntel Corp | |
| [RV] Reserved: checksum good, 2 byte(s) reserved | |
| End | |
| Capabilities: [100] Advanced Error Reporting | |
| UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- | |
| UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- | |
| CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ | |
| CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ | |
| AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- | |
| Capabilities: [140] Device Serial Number bc-30-5b-ff-ff-f3-37-df | |
| Capabilities: [150] Alternative Routing-ID Interpretation (ARI) | |
| ARICap: MFVC- ACS-, Next Function: 1 | |
| ARICtl: MFVC- ACS-, Function Group: 0 | |
| Capabilities: [1a0] Transaction Processing Hints | |
| Device specific mode supported | |
| Steering table in TPH capability structure | |
| Capabilities: [1c0] Latency Tolerance Reporting | |
| Max snoop latency: 0ns | |
| Max no snoop latency: 0ns | |
| Capabilities: [1d0] Access Control Services | |
| ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- | |
| ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- | |
| Kernel driver in use: igb | |
| Kernel modules: igb | |
| # lscpu | |
| Architecture: x86_64 | |
| CPU op-mode(s): 32-bit, 64-bit | |
| Byte Order: Little Endian | |
| CPU(s): 24 | |
| On-line CPU(s) list: 0-23 | |
| Thread(s) per core: 2 | |
| Core(s) per socket: 6 | |
| CPU socket(s): 2 | |
| NUMA node(s): 2 | |
| Vendor ID: GenuineIntel | |
| CPU family: 6 | |
| Model: 45 | |
| Stepping: 7 | |
| CPU MHz: 2299.574 | |
| BogoMIPS: 4600.70 | |
| Virtualization: VT-x | |
| L1d cache: 32K | |
| L1i cache: 32K | |
| L2 cache: 256K | |
| L3 cache: 15360K | |
| NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22 | |
| NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23 | |
| # cat /proc/interrupts | grep em1 | |
| 105: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1 | |
| 106: 85308062 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-0 | |
| 107: 87922486 170 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-1 | |
| 108: 88712000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-2 | |
| 109: 88028576 1337499 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-3 | |
| 110: 205 86846516 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-4 | |
| 111: 192 86342803 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-5 | |
| 112: 174 87468610 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-6 | |
| 113: 329 144261786 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge em1-TxRx-7 | |
| When I set like this: | |
| # echo 0-23 > /proc/irq/105/smp_affinity_list | |
| # echo 0-23 > /proc/irq/106/smp_affinity_list | |
| # echo 0-23 > /proc/irq/107/smp_affinity_list | |
| # echo 0-23 > /proc/irq/108/smp_affinity_list | |
| # echo 0-23 > /proc/irq/109/smp_affinity_list | |
| # echo 0-23 > /proc/irq/110/smp_affinity_list | |
| # echo 0-23 > /proc/irq/111/smp_affinity_list | |
| # echo 0-23 > /proc/irq/112/smp_affinity_list | |
| # echo 0-23 > /proc/irq/113/smp_affinity_list | |
| all interrupts goes to core 0 and core 1 , no other core. Also irq 105 may be different for other irqs, because it has no "-TxRx-NUM" suffix. | |
| When I set: | |
| # echo 0 > /proc/irq/105/smp_affinity_list | |
| # echo 1 > /proc/irq/106/smp_affinity_list | |
| # echo 2 > /proc/irq/107/smp_affinity_list | |
| # echo 3 > /proc/irq/108/smp_affinity_list | |
| # echo 4 > /proc/irq/109/smp_affinity_list | |
| # echo 5 > /proc/irq/110/smp_affinity_list | |
| # echo 6 > /proc/irq/111/smp_affinity_list | |
| # echo 7 > /proc/irq/112/smp_affinity_list | |
| # echo 8 > /proc/irq/113/smp_affinity_list | |
| all goes well except irq 105, and it has not any interrput at all. | |
| So, after all the test, some setting's results are as expected, but others not. I'm confused at this. Please have a look, thank you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment