Skip to content

Instantly share code, notes, and snippets.

@mcastelino
Last active May 25, 2018 14:54
Show Gist options
  • Save mcastelino/d4190c5e5fbe7bd9206c452cd7b5fb25 to your computer and use it in GitHub Desktop.
Save mcastelino/d4190c5e5fbe7bd9206c452cd7b5fb25 to your computer and use it in GitHub Desktop.
Debugging Clear Containers Kernel and Rootfs using qemu-lite

Debug the kernel and rootfs for Clear Containers with pc lite

sudo qemu-lite-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm -cpu host -m 256,maxmem=1G,slots=2 -smp 2 -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -kernel ./vmlinux-4.9.4-53.container -append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1  init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false " -device virtio-serial-pci,id=virtio-serial0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -nographic -object memory-backend-file,id=mem0,share,mem-path=./clear-13280-containers.img,size=235929600 -device nvdimm,memdev=mem0,id=nv0 -no-reboot -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:07:83 -netdev tap,id=net0,ifname=net0,script=no,downscript=no,queues=2,vhost=off

You will see an output such as

char device redirected to /dev/pts/4 (label charconsole0)

Now connect to it

 minicom -D /dev/pts/4

Installation of OVS and DPDK

There is tight coupling between OVS and DPDK versioning, and what is available via distributions is often very out of date. To install a working OVS-DPDK setup for your (ubuntu) host system:

sudo apt-get install autoconf automake kernel-common \
    libpcap-dev libtool python python-six 
git clone http://dpdk.org/git/dpdk -b v16.11 $HOME/dpdk
cd $HOME/dpdk
export RTE_SDK=$(pwd)
export RTE_TARGET=x86_64-native-linuxapp-gcc
make -j install T=$RTE_TARGET DESTDIR=install EXTRA_CFLAGS='-g'
sudo cp -f install/lib/lib* /lib64/
export DPDK_BUILD_DIR=x86_64-native-linuxapp-gcc
git clone https://github.com/openvswitch/ovs -b branch-2.6 $HOME/ovs
cd $HOME/ovs
./boot.sh
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var \
    --with-dpdk=$RTE_SDK/$DPDK_BUILD_DIR  --disable-ssl --with-debug  CFLAGS='-g'
sudo -E make install -j

Setting up ovs-dpdk

sudo mkdir -p /var/run/openvswitch
sudo killall ovsdb-server ovs-vswitchd
rm -f /var/run/openvswitch/vhost-user*
sudo rm -f /etc/openvswitch/conf.db

export DB_SOCK=/var/run/openvswitch/db.sock
sudo -E ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema
sudo -E ovsdb-server --remote=punix:$DB_SOCK --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach

sudo -E ovs-vsctl --no-wait init
sudo -E ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x3
sudo -E ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024
sudo -E ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

sudo sysctl -w vm.nr_hugepages=1024
sudo mount -t hugetlbfs -o pagesize=2048k none /dev/hugepages

sudo -E ovs-vswitchd unix:$DB_SOCK --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log
ps -ae | grep ovs

Setting up the bridge and interfaces

sudo ovs-vsctl add-br ovsbr -- set bridge ovsbr datapath_type=netdev
sudo ovs-vsctl add-port ovsbr vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
sudo ovs-vsctl add-port ovsbr vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser
sudo ovs-vsctl show

Testing DPDK

Launch First VM

sudo qemu-lite-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm  -cpu host -m 256,maxmem=1G,slots=2 -smp 2 -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -kernel ./vmlinux-4.9.4-53.container.1 -append 'reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1  init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false' -device virtio-serial-pci,id=virtio-serial0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -nographic -object memory-backend-file,id=mem0,share,mem-path=./1-clear-13280-containers.img,size=235929600 -device nvdimm,memdev=mem0,id=nv0 -no-reboot  -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:07:83 -netdev tap,id=net0,ifname=net0,script=no,downscript=no,queues=2,vhost=off -chardev socket,id=char1,path=/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 -object memory-backend-file,id=mem,size=256M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem

Launch Second VM

sudo qemu-lite-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm  -cpu host -m 256,maxmem=1G,slots=2 -smp 2 -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -kernel ./vmlinux-4.9.4-53.container.2 -append 'reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1  init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false' -device virtio-serial-pci,id=virtio-serial0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -nographic -object memory-backend-file,id=mem0,share,mem-path=./2-clear-13280-containers.img,size=235929600 -device nvdimm,memdev=mem0,id=nv0 -no-reboot  -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:07:83 -netdev tap,id=net0,ifname=net0,script=no,downscript=no,queues=2,vhost=off -chardev socket,id=char1,path=/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet1 -object memory-backend-file,id=mem,size=256M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem
@mcastelino
Copy link
Author

mcastelino commented Mar 10, 2017

Working command line using legacy BIOS and bzimage.
We need legacy BIOS as EFI firmware does not seem to support NVDIMM.
And we need bzImage to use legacy BIOS.

sudo qemu-lite-system-x86_64 \
-machine pc,accel=kvm,kernel_irqchip,nvdimm  \
-cpu host \
-m 256,maxmem=1G,slots=2 \
-smp 2 \
-no-user-config \
-nodefaults \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-kernel vmlinuz-4.9.4-53.container \
-append 'reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1  init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false' \
-device virtio-serial-pci,id=virtio-serial0 \
-chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 \
-nographic \
-object memory-backend-file,id=mem0,share,mem-path=./2-clear-13280-containers.img,size=235929600 \
-device nvdimm,memdev=mem0,id=nv0 \
-no-reboot \
-chardev socket,id=char1,path=/run/openvswitch/vhost-user2 \
-netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce \
-device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet1 \
-object memory-backend-file,id=mem,size=256M,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment