Skip to content

Instantly share code, notes, and snippets.

@pklaus

pklaus/About_SR-IOV_with_Intel-Gigabit-ET-Dual-Port-Server.md

Last active August 29, 2015 14:07
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pklaus/99f399a946a20a1758af to your computer and use it in GitHub Desktop.
Save pklaus/99f399a946a20a1758af to your computer and use it in GitHub Desktop.
Download ZIP
SR-IOV with a Intel Gigabit ET Dual Port Server Adapter on a Dell PowerEdge T20. See http://go.klaus.pw/ET_Dual
Raw
About_SR-IOV_with_Intel-Gigabit-ET-Dual-Port-Server.md
Raw
check-for-support.txt
[philipp@lion ~]$ dmesg |grep DMAR
[ 0.000000] ACPI: DMAR 0x00000000D8FFE4C8 0000B8 (v01 INTEL HSW 00000001 INTL 00000001)
[philipp@lion ~]$ dmesg |grep IOMMU
[ 0.019212] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[ 0.019217] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[ 0.019284] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1
[philipp@lion ~]$ sudo lspci -vv | grep -A 10 "Single Root I/O Virtualization"
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy-
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 7, Function Dependency Link: 00
VF offset: 384, stride: 2, Device ID: 10ca
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000f7c48000 (64-bit, non-prefetchable)
Region 3: Memory at 00000000f7c68000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: igb
--
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy-
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 7, Function Dependency Link: 01
VF offset: 384, stride: 2, Device ID: 10ca
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000f7c88000 (64-bit, non-prefetchable)
Region 3: Memory at 00000000f7ca8000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: igb
[philipp@lion ~]$ dmesg | egrep -i "DMAR|IOMMU"
[ 0.000000] ACPI: DMAR 0x00000000D8FFE4C8 0000B8 (v01 INTEL HSW 00000001 INTL 00000001)
[ 0.019205] dmar: Host address width 39
[ 0.019206] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.019212] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[ 0.019213] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.019217] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[ 0.019218] dmar: RMRR base: 0x000000dbf7c000 end: 0x000000dbf89fff
[ 0.019218] dmar: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[ 0.019284] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1
Raw
dmesg.txt
After
modprobe -r igb
modprobe igb max_vfs=1
[ 36.482454] igb 0000:04:00.1: removed PHC on enp4s0f1
[ 36.505742] igb 0000:04:00.0: removed PHC on enp4s0f0
[ 45.712698] dca service started, version 1.12.1
[ 45.720985] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.5-k
[ 45.720991] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 45.721483] igb 0000:04:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
[ 45.721531] igb 0000:04:00.0: irq 49 for MSI/MSI-X
[ 45.721544] igb 0000:04:00.0: irq 50 for MSI/MSI-X
[ 45.721554] igb 0000:04:00.0: irq 51 for MSI/MSI-X
[ 45.822728] pci 0000:05:10.0: [8086:10ca] type 00 class 0x020000
[ 45.822912] pci 0000:05:10.2: [8086:10ca] type 00 class 0x020000
[ 45.823071] pci 0000:05:10.4: [8086:10ca] type 00 class 0x020000
[ 45.823226] pci 0000:05:10.6: [8086:10ca] type 00 class 0x020000
[ 45.823376] pci 0000:05:11.0: [8086:10ca] type 00 class 0x020000
[ 45.823528] pci 0000:05:11.2: [8086:10ca] type 00 class 0x020000
[ 45.823678] pci 0000:05:11.4: [8086:10ca] type 00 class 0x020000
[ 45.823819] igb 0000:04:00.0: 7 VFs allocated
[ 45.826870] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.0.2-k
[ 45.826871] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 45.980871] igb 0000:04:00.0: irq 49 for MSI/MSI-X
[ 45.980876] igb 0000:04:00.0: irq 50 for MSI/MSI-X
[ 45.980879] igb 0000:04:00.0: irq 51 for MSI/MSI-X
[ 46.191452] igb 0000:04:00.0: added PHC on eth0
[ 46.191455] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 46.191457] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:1b:21:43:31:90
[ 46.191460] igb 0000:04:00.0: eth0: PBA No: Unknown
[ 46.191461] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 1 tx queue(s)
[ 46.191900] igb 0000:04:00.1: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
[ 46.191920] igb 0000:04:00.1: irq 52 for MSI/MSI-X
[ 46.191924] igb 0000:04:00.1: irq 53 for MSI/MSI-X
[ 46.191927] igb 0000:04:00.1: irq 54 for MSI/MSI-X
[ 46.293113] pci 0000:05:10.1: [8086:10ca] type 00 class 0x020000
[ 46.293306] igbvf 0000:05:10.1: enabling device (0000 -> 0002)
[ 46.293354] igbvf 0000:05:10.1: irq 55 for MSI/MSI-X
[ 46.293358] igbvf 0000:05:10.1: irq 56 for MSI/MSI-X
[ 46.293361] igbvf 0000:05:10.1: irq 57 for MSI/MSI-X
[ 46.294730] igbvf 0000:05:10.1: PF still in reset state. Is the PF interface up?
[ 46.294731] igbvf 0000:05:10.1: Assigning random MAC address.
[ 46.296078] igbvf 0000:05:10.1: PF still resetting
[ 46.296183] igbvf 0000:05:10.1: Intel(R) 82576 Virtual Function
[ 46.296184] igbvf 0000:05:10.1: Address: aa:42:84:59:fa:9c
[ 46.296234] pci 0000:05:10.3: [8086:10ca] type 00 class 0x020000
[ 46.296387] igbvf 0000:05:10.3: enabling device (0000 -> 0002)
[ 46.296415] igbvf 0000:05:10.3: irq 59 for MSI/MSI-X
[ 46.296418] igbvf 0000:05:10.3: irq 60 for MSI/MSI-X
[ 46.296421] igbvf 0000:05:10.3: irq 61 for MSI/MSI-X
[ 46.297781] igbvf 0000:05:10.3: PF still in reset state. Is the PF interface up?
[ 46.297782] igbvf 0000:05:10.3: Assigning random MAC address.
[ 46.299129] igbvf 0000:05:10.3: PF still resetting
[ 46.299214] igbvf 0000:05:10.3: Intel(R) 82576 Virtual Function
[ 46.299215] igbvf 0000:05:10.3: Address: 2e:08:4f:6a:3e:96
[ 46.299263] pci 0000:05:10.5: [8086:10ca] type 00 class 0x020000
[ 46.299411] igbvf 0000:05:10.5: enabling device (0000 -> 0002)
[ 46.299437] igbvf 0000:05:10.5: irq 62 for MSI/MSI-X
[ 46.299440] igbvf 0000:05:10.5: irq 63 for MSI/MSI-X
[ 46.299444] igbvf 0000:05:10.5: irq 64 for MSI/MSI-X
[ 46.300811] igbvf 0000:05:10.5: PF still in reset state. Is the PF interface up?
[ 46.300812] igbvf 0000:05:10.5: Assigning random MAC address.
[ 46.302161] igbvf 0000:05:10.5: PF still resetting
[ 46.302239] igbvf 0000:05:10.5: Intel(R) 82576 Virtual Function
[ 46.302240] igbvf 0000:05:10.5: Address: d2:af:54:57:c3:70
[ 46.302287] pci 0000:05:10.7: [8086:10ca] type 00 class 0x020000
[ 46.302430] igbvf 0000:05:10.7: enabling device (0000 -> 0002)
[ 46.302457] igbvf 0000:05:10.7: irq 65 for MSI/MSI-X
[ 46.302460] igbvf 0000:05:10.7: irq 66 for MSI/MSI-X
[ 46.302463] igbvf 0000:05:10.7: irq 67 for MSI/MSI-X
[ 46.303848] igbvf 0000:05:10.7: PF still in reset state. Is the PF interface up?
[ 46.303849] igbvf 0000:05:10.7: Assigning random MAC address.
[ 46.305196] igbvf 0000:05:10.7: PF still resetting
[ 46.305291] igbvf 0000:05:10.7: Intel(R) 82576 Virtual Function
[ 46.305292] igbvf 0000:05:10.7: Address: ce:62:33:ba:7b:d1
[ 46.305383] pci 0000:05:11.1: [8086:10ca] type 00 class 0x020000
[ 46.305553] igbvf 0000:05:11.1: enabling device (0000 -> 0002)
[ 46.305578] igbvf 0000:05:11.1: irq 69 for MSI/MSI-X
[ 46.305581] igbvf 0000:05:11.1: irq 70 for MSI/MSI-X
[ 46.305584] igbvf 0000:05:11.1: irq 71 for MSI/MSI-X
[ 46.306979] igbvf 0000:05:11.1: PF still in reset state. Is the PF interface up?
[ 46.306980] igbvf 0000:05:11.1: Assigning random MAC address.
[ 46.308329] igbvf 0000:05:11.1: PF still resetting
[ 46.308422] igbvf 0000:05:11.1: Intel(R) 82576 Virtual Function
[ 46.308423] igbvf 0000:05:11.1: Address: fa:41:93:de:f5:56
[ 46.308512] pci 0000:05:11.3: [8086:10ca] type 00 class 0x020000
[ 46.308679] igbvf 0000:05:11.3: enabling device (0000 -> 0002)
[ 46.308705] igbvf 0000:05:11.3: irq 72 for MSI/MSI-X
[ 46.308708] igbvf 0000:05:11.3: irq 73 for MSI/MSI-X
[ 46.308711] igbvf 0000:05:11.3: irq 74 for MSI/MSI-X
[ 46.310116] igbvf 0000:05:11.3: PF still in reset state. Is the PF interface up?
[ 46.310117] igbvf 0000:05:11.3: Assigning random MAC address.
[ 46.311467] igbvf 0000:05:11.3: PF still resetting
[ 46.311561] igbvf 0000:05:11.3: Intel(R) 82576 Virtual Function
[ 46.311562] igbvf 0000:05:11.3: Address: ba:61:a5:ab:48:ed
[ 46.311648] pci 0000:05:11.5: [8086:10ca] type 00 class 0x020000
[ 46.311815] igbvf 0000:05:11.5: enabling device (0000 -> 0002)
[ 46.311841] igbvf 0000:05:11.5: irq 75 for MSI/MSI-X
[ 46.311844] igbvf 0000:05:11.5: irq 76 for MSI/MSI-X
[ 46.311846] igbvf 0000:05:11.5: irq 77 for MSI/MSI-X
[ 46.313231] igbvf 0000:05:11.5: PF still in reset state. Is the PF interface up?
[ 46.313232] igbvf 0000:05:11.5: Assigning random MAC address.
[ 46.314585] igbvf 0000:05:11.5: PF still resetting
[ 46.314676] igbvf 0000:05:11.5: Intel(R) 82576 Virtual Function
[ 46.314677] igbvf 0000:05:11.5: Address: f6:74:fe:c6:b1:7d
[ 46.314729] igb 0000:04:00.1: 7 VFs allocated
[ 46.471331] igb 0000:04:00.1: irq 52 for MSI/MSI-X
[ 46.471339] igb 0000:04:00.1: irq 53 for MSI/MSI-X
[ 46.471345] igb 0000:04:00.1: irq 54 for MSI/MSI-X
[ 46.648435] igb 0000:04:00.1: added PHC on eth8
[ 46.648438] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 46.648439] igb 0000:04:00.1: eth8: (PCIe:2.5Gb/s:Width x4) 00:1b:21:43:31:91
[ 46.648442] igb 0000:04:00.1: eth8: PBA No: Unknown
[ 46.648443] igb 0000:04:00.1: Using MSI-X interrupts. 2 rx queue(s), 1 tx queue(s)
[ 46.648613] igbvf 0000:05:10.0: enabling device (0000 -> 0002)
[ 46.648664] igbvf 0000:05:10.0: irq 78 for MSI/MSI-X
[ 46.648668] igbvf 0000:05:10.0: irq 79 for MSI/MSI-X
[ 46.648671] igbvf 0000:05:10.0: irq 80 for MSI/MSI-X
[ 46.650058] igbvf 0000:05:10.0: PF still in reset state. Is the PF interface up?
[ 46.650060] igbvf 0000:05:10.0: Assigning random MAC address.
[ 46.651436] igbvf 0000:05:10.0: PF still resetting
[ 46.673411] systemd-udevd[738]: renamed network interface eth8 to enp4s0f1
[ 46.673650] igbvf 0000:05:10.0: Intel(R) 82576 Virtual Function
[ 46.673652] igbvf 0000:05:10.0: Address: 92:fb:9e:2e:47:f7
[ 46.673685] igbvf 0000:05:10.2: enabling device (0000 -> 0002)
[ 46.673720] igbvf 0000:05:10.2: irq 81 for MSI/MSI-X
[ 46.673724] igbvf 0000:05:10.2: irq 82 for MSI/MSI-X
[ 46.673727] igbvf 0000:05:10.2: irq 83 for MSI/MSI-X
[ 46.675091] igbvf 0000:05:10.2: PF still in reset state. Is the PF interface up?
[ 46.675092] igbvf 0000:05:10.2: Assigning random MAC address.
[ 46.676436] igbvf 0000:05:10.2: PF still resetting
[ 46.700085] systemd-udevd[730]: renamed network interface eth0 to enp4s0f0
[ 46.700188] igbvf 0000:05:10.2: Intel(R) 82576 Virtual Function
[ 46.700191] igbvf 0000:05:10.2: Address: 96:2f:0f:32:86:4c
[ 46.700224] igbvf 0000:05:10.4: enabling device (0000 -> 0002)
[ 46.700258] igbvf 0000:05:10.4: irq 84 for MSI/MSI-X
[ 46.700261] igbvf 0000:05:10.4: irq 85 for MSI/MSI-X
[ 46.700265] igbvf 0000:05:10.4: irq 86 for MSI/MSI-X
[ 46.701660] igbvf 0000:05:10.4: PF still in reset state. Is the PF interface up?
[ 46.701661] igbvf 0000:05:10.4: Assigning random MAC address.
[ 46.703021] igbvf 0000:05:10.4: PF still resetting
[ 46.703093] igbvf 0000:05:10.4: Intel(R) 82576 Virtual Function
[ 46.703094] igbvf 0000:05:10.4: Address: 36:67:02:21:e2:c1
[ 46.703160] igbvf 0000:05:10.6: enabling device (0000 -> 0002)
[ 46.703202] igbvf 0000:05:10.6: irq 87 for MSI/MSI-X
[ 46.703205] igbvf 0000:05:10.6: irq 88 for MSI/MSI-X
[ 46.703209] igbvf 0000:05:10.6: irq 89 for MSI/MSI-X
[ 46.704591] igbvf 0000:05:10.6: PF still in reset state. Is the PF interface up?
[ 46.704592] igbvf 0000:05:10.6: Assigning random MAC address.
[ 46.705937] igbvf 0000:05:10.6: PF still resetting
[ 46.706015] igbvf 0000:05:10.6: Intel(R) 82576 Virtual Function
[ 46.706016] igbvf 0000:05:10.6: Address: 2a:60:ef:bb:00:55
[ 46.706056] igbvf 0000:05:11.0: enabling device (0000 -> 0002)
[ 46.706109] igbvf 0000:05:11.0: irq 90 for MSI/MSI-X
[ 46.706112] igbvf 0000:05:11.0: irq 91 for MSI/MSI-X
[ 46.706115] igbvf 0000:05:11.0: irq 92 for MSI/MSI-X
[ 46.707503] igbvf 0000:05:11.0: PF still in reset state. Is the PF interface up?
[ 46.707504] igbvf 0000:05:11.0: Assigning random MAC address.
[ 46.708864] igbvf 0000:05:11.0: PF still resetting
[ 46.708939] igbvf 0000:05:11.0: Intel(R) 82576 Virtual Function
[ 46.708940] igbvf 0000:05:11.0: Address: 92:53:a5:cd:ac:f1
[ 46.708998] igbvf 0000:05:11.2: enabling device (0000 -> 0002)
[ 46.709033] igbvf 0000:05:11.2: irq 93 for MSI/MSI-X
[ 46.709035] igbvf 0000:05:11.2: irq 94 for MSI/MSI-X
[ 46.709047] igbvf 0000:05:11.2: irq 95 for MSI/MSI-X
[ 46.710413] igbvf 0000:05:11.2: PF still in reset state. Is the PF interface up?
[ 46.710414] igbvf 0000:05:11.2: Assigning random MAC address.
[ 46.711766] igbvf 0000:05:11.2: PF still resetting
[ 46.711839] igbvf 0000:05:11.2: Intel(R) 82576 Virtual Function
[ 46.711840] igbvf 0000:05:11.2: Address: de:7e:ae:b2:3a:78
[ 46.711897] igbvf 0000:05:11.4: enabling device (0000 -> 0002)
[ 46.711941] igbvf 0000:05:11.4: irq 96 for MSI/MSI-X
[ 46.711944] igbvf 0000:05:11.4: irq 97 for MSI/MSI-X
[ 46.711947] igbvf 0000:05:11.4: irq 98 for MSI/MSI-X
[ 46.713304] igbvf 0000:05:11.4: PF still in reset state. Is the PF interface up?
[ 46.713305] igbvf 0000:05:11.4: Assigning random MAC address.
[ 46.714659] igbvf 0000:05:11.4: PF still resetting
[ 46.714725] igbvf 0000:05:11.4: Intel(R) 82576 Virtual Function
[ 46.714726] igbvf 0000:05:11.4: Address: ba:72:91:f6:69:a0
[ 46.733475] systemd-udevd[736]: renamed network interface eth6 to enp5s17f3
[ 46.753561] systemd-udevd[731]: renamed network interface eth1 to enp5s16f1
[ 46.793599] systemd-udevd[737]: renamed network interface eth7 to enp5s17f5
[ 46.820207] systemd-udevd[726]: renamed network interface eth9 to enp5s16f4
[ 46.860320] systemd-udevd[728]: renamed network interface eth11 to enp5s17
[ 46.900363] systemd-udevd[729]: renamed network interface eth12 to enp5s17f2
[ 46.927038] systemd-udevd[719]: renamed network interface eth8 to enp5s16
[ 46.953726] systemd-udevd[735]: renamed network interface eth5 to enp5s17f1
[ 46.980417] systemd-udevd[732]: renamed network interface eth2 to enp5s16f3
[ 47.007093] systemd-udevd[733]: renamed network interface eth3 to enp5s16f5
[ 47.033794] systemd-udevd[725]: renamed network interface eth0 to enp5s16f2
[ 47.073829] systemd-udevd[727]: renamed network interface eth10 to enp5s16f6
[ 47.100502] systemd-udevd[718]: renamed network interface eth4 to enp5s16f7
[ 47.117138] systemd-udevd[734]: renamed network interface eth13 to enp5s17f4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment