demikernel-catnip-setup.md

Replication Steps

These steps will need to be done on two different machines on the same network, one as a client and another as a server.

1. VM: Install Ubuntu 24.04 on VirtualBox (other hypervisors may work but local Hyper-V won't support DPDK)
   Baremetal: on Cloudlab, follow the section on requesting resources here to acquire two nodes that have the correct hardware
2. (VM only) Configure the VM with at least 4GB RAM, 8 vCPUs, PAE/NX enabled, Nested VT-x, Nested paging, and two network adapters (both bridged, using the virtio-net driver)
3. Follow the steps here: https://github.com/microsoft/demikernel/blob/dev/doc/setup.md to build DPDK and set up Demikernel. You may need to apply this diff: https://gist.github.com/jdabtieu/0b947ae141eb82bb3188e056c1c65d8e and rebuild if an issue with RSS_HF shows up later.
4. For the configuration file: Run ip addr to grab the MAC address, IP address, and interface name. Run lspci to get the PCI ID of the network adaptor. Fill everything out. arp_table should also contain an entry for both devices
5. Follow the steps here: https://github.com/microsoft/demikernel/blob/dev/doc/building.md. For catnap, just make and make install and you're done. For catnip, make LIBOS=catnip but you may need to also add the parameter PKG_CONFIG_PATH=$HOME/lib/x86_64-linux-gnu/pkgconfig
6. switch to a root prompt
7. Set the $HOME variable to the home directory of the non-root user.
8. export LD_LIBRARY_PATH=$HOME/lib/x86_64-linux-gnu/dpdk/pmds-23.0:$HOME/lib/x86_64-linux-gnu
9. export LIBOS=catnip
10. export CONFIG_PATH to point to the configuration file
11. (VM only) Clone https://github.com/emmericp/dpdk and follow these instructions to enable no-IOMMU vfio
12. (VM only) Run sudo python3 dpdk-devbind.py --force --bind=vfio-pci 00:03.0, replacing it with the PCI ID of the vNIC
13. On the server, run bin/examples/rust/tcp-echo.elf --address <SERVER_IP>:56789 --peer server --log 1
14. On the client, run bin/examples/rust/tcp-echo.elf --address <SERVER_IP>:56789 --peer client --nclients 1 --nthreads 1 --nrequests 65536 --log 1 --bufsize 64 --run-mode sequential
15. This will then print statistics showing average latencies every second. This data can then be analyzed.

We found that using realtime scheduling and CPU pinning had limited effect on p50 statistics, but may make the tail latencies a little more consistent. If desired, the echo programs can be run under SCHED_RR using chrt, with a nice value of -20 and on a physical core with no other processes scheduled with taskset (and disabling the corresponding core in systemd for other processes). RT scheduling and CPU pinning?