egernst/Debug CC QEMU pc-lite.md

## Debug CC QEMU pc-lite.md

      
    Raw
  

              Debug CC QEMU pc-lite.md
            
          
    Debug the kernel and rootfs for Clear Containers with pc lite

sudo qemu-lite-system-x86_64 \
  -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
  -cpu host -m 256,maxmem=1G,slots=2 \
  -smp 2 -no-user-config -nodefaults \
  -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard \
  -kernel ./vmlinux-4.9.4-53.container \
  -append "reboot=k panic=1 rw tsc=reliable no_timer_check \
     noreplace-smp root=/dev/pmem0p1  init=/usr/lib/systemd/systemd \
     initcall_debug rootfstype=ext4 rootflags=dax,data=ordered \
     dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 \
     single iommu=false" \
   -device virtio-serial-pci,id=virtio-serial0 \
   -chardev pty,id=charconsole0 \
   -device virtconsole,chardev=charconsole0,id=console0 \
   -nographic \
   -object memory-backend-file,id=mem0,mem-path=./clear-13280-containers.img,size=235929600 \
   -device nvdimm,memdev=mem0,id=nv0 \
   -no-reboot \
   -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:07:83 \
   -netdev tap,id=net0,ifname=net0,script=no,downscript=no,queues=2,vhost=off

You will see an output such as
char device redirected to /dev/pts/4 (label charconsole0)

Now connect to it
 minicom -D /dev/pts/4

Installation of OVS and DPDK

There is tight coupling between OVS and DPDK versioning, and what is available via distributions is often very out of date.  To install a working OVS-DPDK setup for your (ubuntu) host system:
sudo apt-get install autoconf automake kernel-common \
    libpcap-dev libtool python python-six 
git clone http://dpdk.org/git/dpdk -b v16.11 $HOME/dpdk
cd $HOME/dpdk
export RTE_SDK=$(pwd)
export RTE_TARGET=x86_64-native-linuxapp-gcc
make -j install T=$RTE_TARGET DESTDIR=install EXTRA_CFLAGS='-g'
sudo cp -f install/lib/lib* /lib64/
export DPDK_BUILD_DIR=x86_64-native-linuxapp-gcc
git clone https://github.com/openvswitch/ovs -b branch-2.6 $HOME/ovs
cd $HOME/ovs
./boot.sh
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var \
    --with-dpdk=$RTE_SDK/$DPDK_BUILD_DIR  --disable-ssl --with-debug  CFLAGS='-g'
sudo -E make install -j

Setting up ovs-dpdk

sudo mkdir -p /var/run/openvswitch
sudo killall ovsdb-server ovs-vswitchd
rm -f /var/run/openvswitch/vhost-user*
sudo rm -f /etc/openvswitch/conf.db

export DB_SOCK=/var/run/openvswitch/db.sock
sudo -E ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema
sudo -E ovsdb-server --remote=punix:$DB_SOCK --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach

sudo -E ovs-vsctl --no-wait init
sudo -E ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x3
sudo -E ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024
sudo -E ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

sudo sysctl -w vm.nr_hugepages=1024
sudo mount -t hugetlbfs -o pagesize=2048k none /dev/hugepages

sudo -E ovs-vswitchd unix:$DB_SOCK --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log
ps -ae | grep ovs

Setting up the bridge and interfaces

sudo ovs-vsctl add-br ovsbr -- set bridge ovsbr datapath_type=netdev
sudo ovs-vsctl add-port ovsbr vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
sudo ovs-vsctl add-port ovsbr vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser
sudo ovs-vsctl show

Testing DPDK

Launch First VM

sudo qemu-lite-system-x86_64 \
  -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
  -cpu host -m 256,maxmem=1G,slots=2 \
  -smp 2 \
  -no-user-config \
  -nodefaults \
  -rtc base=utc,driftfix=slew \
  -global kvm-pit.lost_tick_policy=discard \
  -kernel ./vmlinux-4.9.4-53.container.1 \
  -append 'reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1  init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false' \
  -device virtio-serial-pci,id=virtio-serial0 \
  -chardev pty,id=charconsole0 \
  -device virtconsole,chardev=charconsole0,id=console0 \
  -nographic \
  -object memory-backend-file,id=mem0,mem-path=./1-clear-13280-containers.img,size=235929600 \
  -device nvdimm,memdev=mem0,id=nv0 \
  -no-reboot \
  -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:07:83 \
  -netdev tap,id=net0,ifname=net0,script=no,downscript=no,queues=2,vhost=off \
  -chardev socket,id=char1,path=/run/openvswitch/vhost-user1 \
  -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce \
  -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 \
  -object memory-backend-file,id=mem,size=256M,mem-path=/dev/hugepages,share=on \
  -numa node,memdev=mem

Launch Second VM

sudo qemu-lite-system-x86_64 \
  -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
  -cpu host \
  -m 256,maxmem=1G,slots=2 \
  -smp 2 \
  -no-user-config \
  -nodefaults \
  -rtc base=utc,driftfix=slew \
  -global kvm-pit.lost_tick_policy=discard \
  -kernel ./vmlinux-4.9.4-53.container.2 \
  -append 'reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1  init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false' \
  -device virtio-serial-pci,id=virtio-serial0 \
  -chardev pty,id=charconsole0 \
  -device virtconsole,chardev=charconsole0,id=console0 \
  -nographic \
  -object memory-backend-file,id=mem0,mem-path=./2-clear-13280-containers.img,size=235929600 \
  -device nvdimm,memdev=mem0,id=nv0 \
  -no-reboot \
  -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:07:83 \
  -netdev tap,id=net0,ifname=net0,script=no,downscript=no,queues=2,vhost=off \
  -chardev socket,id=char1,path=/run/openvswitch/vhost-user2 \
  -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce \
  -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet1 \
  -object memory-backend-file,id=mem,size=256M,mem-path=/dev/hugepages,share=on \
  -numa node,memdev=mem