amshinde

Passing a graphics card to a Clear Container

Host system setup:

The directions below are specific to an Ubuntu 16.04 host system equipped with a GeForce GTX 550 Ti graphics card. While the process should be the same, YMMV.

1. Make sure your system supports IOMMU groups

todo -- get directions from SRIOV page

amshinde

Connecting a veth device to tap

• veth device from CNI/CNM plugin: eth0
• tap device that connects to the VM: tap0

Redirecting traffic between the two devices

tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol all u32 match u8 0 0 action mirred egress redirect dev tap0

amshinde

/etc/modprobe.d/overlay.conf with the following contents:

options overlay index=off

check in /sys/modules/overlay/parameters/index

amshinde

How to use VFIO to assign a device to nested VM

• Here the vfio-pci device is passed in into the L1 VM
• The L1 VM is setup with kernel_irqchip=split
• The L0 exposes a virtual IOMMU to the L1 VM

qemu-system-x86_64 \
    -machine q35,accel=kvm,kernel_irqchip=split \
    -enable-kvm \

amshinde

Failed to add inotify watch : no space left on disk

This is due to the inotify watch limit being hit. cat /proc/sys/fs/inotify/max_user_watches

sudo sysctl fs.inotify.max_user_watches=524288

or edit

amshinde

The netfilter hooks in the kernel and where they hook in the packet flow

The figure below calls out

• The netfilter hooks
• The order of table traversal

amshinde

Setting up a overlay network using point to multi-point GRE tunnels

Multi-Point GRE Tunnels

Typically VxLAN is used to setup a point to multi-point overlay networks. However GRE also offers a similar capability which is often not heavily used. This document calls out how to setup an effient overlay mesh network to link up multiples sits together using multi point GRE tunnels.

Setup

The setup used in this case assumes there is a need to link three sites hosting three different subnets together using a simple overlay mesh network.

amshinde

#side by side terminal diff
git difftool -y -x "sdiff -w $COLUMNS" | less

amshinde

sudo -E qemu-lite-system-x86_64 -machine pc,accel=kvm,kernel_irqchip -cpu host      -m 2G,slots=3,maxmem=16G -object memory-backend-file,id=mem,size=2G,mem-path=/dev/shm,share=on -numa node,memdev=mem        -device virtio-serial-pci,id=serial0 -device virtio-blk-pci,drive=image -drive if=none,id=image,file=kata-containers.img,format=raw   -kernel vmlinuz.container -append 'console=ttyS0 console=hvc0 iommu=false root=/dev/vda1 rw rootfstype=ext4 data=ordered rcupdate.rcu_expedited=1 tsc=reliable no_timer_check loglevel=7 apic=verbose noapic noapictimer virtio_mmio.device=0x200@0x540000000:14' -smp 2,maxcpus=4 -device virtconsole,chardev=charconsole0,id=console0 -chardev stdio,id=charconsole0 -monitor telnet:127.0.0.1:55555,server,nowait -device isa-debugcon,iobase=0x402,chardev=debugcon -chardev file,path=/tmp/debug-log,id=debugcon -device isa-debugcon,iobase=0x3f8,chardev=serialcon -chardev file,path=/tmp/serial-log,id=serialcon
[sudo] password for amshinde: 
[    0.223920] brd: module loaded
[    0.226454] 

amshinde

Create a process with its own network and pid namespace

$ unshare --user --map-root-user --net --mount
[root@incensed-gawain ~]# echo $$
2646

Run the slirp process on the host