test dpdk testpmd k8s

Dockerfile

FROM ubuntu:bionic as ubuntu-build
RUN apt-get update && \
    apt-get -y install \
        build-essential \
        git \
        libnuma-dev

ARG DPDK_VER='master'
ENV DPDK_DIR='/dpdk'
ENV RTE_TARGET='x86_64-native-linuxapp-gcc'
RUN git clone -b $DPDK_VER -q --depth 1 http://dpdk.org/git/dpdk-stable $DPDK_DIR 2>&1
RUN cd ${DPDK_DIR} && \
    sed -ri 's,(IGB_UIO=).*,\1n,' config/common_linux* && \
    sed -ri 's,(KNI_KMOD=).*,\1n,' config/common_linux* && \
    make config T=x86_64-native-linuxapp-gcc && \
    make -j $CPUS
ENV PATH="$PATH:$DPDK_DIR/build/app/"

sriov-dpdk.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: dpdk-1711
  annotations:
      k8s.v1.cni.cncf.io/networks: sriov-net-dpdk
spec:
  containers:
  - name: dpdk
    image: krsna1729/dpdk:v17.11
    command: [ "sleep", "100000" ]
    securityContext:
      capabilities:
        add:
        - IPC_LOCK
    resources:
      limits:
        intel.com/sriov_vfio: '1'
---
apiVersion: v1
kind: Pod
metadata:
  name: dpdk-1811
  annotations:
      k8s.v1.cni.cncf.io/networks: sriov-net-dpdk
spec:
  containers:
  - name: dpdk
    image: krsna1729/dpdk:v18.11
    command: [ "sleep", "100000" ]
    securityContext:
      capabilities:
        add:
        - IPC_LOCK
    resources:
      limits:
        intel.com/sriov_vfio: '1'
---
apiVersion: v1
kind: Pod
metadata:
  name: dpdk-1911
  annotations:
      k8s.v1.cni.cncf.io/networks: sriov-net-dpdk
spec:
  containers:
  - name: dpdk
    image: krsna1729/dpdk:v19.11
    command: [ "sleep", "100000" ]
    securityContext:
      capabilities:
        add:
        - IPC_LOCK
    resources:
      limits:
        intel.com/sriov_vfio: '1'

docker build --build-arg DPDK_VER=v17.11 -t krsna1729/dpdk:v17.11 .
docker build --build-arg DPDK_VER=v18.11 -t krsna1729/dpdk:v18.11 .
docker build --build-arg DPDK_VER=v19.11 -t krsna1729/dpdk:v19.11 .

kubectl exec -it dpdk-1711 -- bash -c "testpmd --no-huge -m 2048 -- --stats-period=1 --nb-cores=1 --nb-ports=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=rxonly"
kubectl exec -it dpdk-1811 -- bash -c "testpmd --no-huge -m 2048 -- --stats-period=1 --nb-cores=1 --nb-ports=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=rxonly"
kubectl exec -it dpdk-1911 -- bash -c "testpmd --no-huge -m 2048 -- --stats-period=1 --nb-cores=1 --nb-ports=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=rxonly"

With --legacy-mem for 18.05+

kubectl exec -it dpdk-1811 -- bash -c "testpmd --no-huge -m 2048 --legacy-mem -- --stats-period=1 --nb-cores=1 --nb-ports=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=rxonly"
kubectl exec -it dpdk-1911 -- bash -c "testpmd --no-huge -m 2048 --legacy-mem -- --stats-period=1 --nb-cores=1 --nb-ports=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=rxonly"

v17.11

EAL: PCI device 0000:07:08.2 on NUMA socket 0
EAL:   probe driver: 8086:154c net_i40e_vf
EAL:   using IOMMU type 1 (Type 1)
…
Auto-start selected
Set rxonly packet forwarding mode
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=2048, size=2176, socket=0
USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=2048, size=2176, socket=1
Configuring Port 0 (socket 0)
Port 0: 56:86:3F:D6:6F:B6
Checking link statuses...
Done
No commandline core given, start packet forwarding
rxonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP over anonymous pages disabled
Logical Core 1 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  rxonly packet forwarding - CRC stripping enabled - packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=1
  RX queues=1 - RX desc=128 - RX free threshold=32
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=1 - TX desc=512 - TX free threshold=32
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=32 - TXQ flags=0xf01

Port statistics ====================================
  ######################## NIC statistics for port 0  ########################
  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-errors: 0
  RX-nombuf:  0
  TX-packets: 0          TX-errors: 0          TX-bytes:  0

  Throughput (since last show)
  Rx-pps:            0
  Tx-pps:            0
  ############################################################################

v18.11

EAL: PCI device 0000:07:06.4 on NUMA socket 0
EAL:   probe driver: 8086:154c net_i40e_vf
EAL: Getting a vfio_dev_fd for 0000:07:06.4 failed
EAL: Requested device 0000:07:06.4 cannot be used
…
testpmd: No probed ethernet devices
EAL: Error - exiting with code: 1
  Cause: Invalid port 1
command terminated with exit code 1

v19.11

EAL: PCI device 0000:07:09.5 on NUMA socket 0
EAL:   probe driver: 8086:154c net_i40e_vf
EAL: Getting a vfio_dev_fd for 0000:07:09.5 failed
EAL: Requested device 0000:07:09.5 cannot be used
...
testpmd: No probed ethernet devices
EAL: Error - exiting with code: 1
  Cause: Invalid port 1
command terminated with exit code 1

It works fine if the first VF gets allocated, same effect when all VF are assigned to same container.

EAL: PCI device 0000:07:06.0 on NUMA socket 0
EAL:   probe driver: 8086:154c net_i40e_vf
EAL:   using IOMMU type 1 (Type 1)

Component versions

multus    3.4.2
sriovdp   3.2
kernel    4.15.0-72-generic

# Not relevant
sriov-cni 2.3

relevant kernel commandline

intel_iommu=on

Update:

DPDK stable branches (not tags! note the absent v when passing --build-arg DPDK_VER)

NAME              READY   STATUS    RESTARTS   AGE
dpdk-1711         1/1     Running   0          3m5s
dpdk-1811         0/1     Error     0          3m5s
dpdk-1911         1/1     Running   0          3m5s
dpdk-2002         0/1     Error     0          3m5s

clearlinux/cloud-native-setup@0990006

Relevant fix
http://git.dpdk.org/dpdk-stable/commit/?h=19.11&id=5732151922c287abf7c21b92291d041ef1cc0cdb

Tested with 20.02. Does not cleanly apply to 18.11

dpdk-1711         1/1     Running   0          44s
dpdk-1811         0/1     Error     0          44s
dpdk-1911         1/1     Running   0          44s
dpdk-2002         1/1     Running   0          44s