The addition of new features and new commits to the kernel and userland may often cause regression on certain platforms or cause performance degradation issues. If these regressions and performance degradation issues can be tracked with date/time and commits, it would be a lot easier to identify what changes caused it, to debug them and report to the specific contributor.
The Phoronix Test Suite (PTS) is an open-source, cross-platform automated testing/benchmarking software for Linux as well as BSD operating systems. PTS allows the creation of new tests using simple XML files and shell scripts, and integrates with revision control systems for per-commit regression testing, and provides reproducible benchmarking results and record their history also.
In order to tackle the issue, I integrated the latest stable Phoronix Test Suite (PTS) with NetBSD by porting and fixing the available test-profiles to NetBSD and automated the testing with Anita in this project.
- Name: Apurva Nandan
- Project: Benchmark NetBSD
- Organisation: The NetBSD Foundation
- Mentor: leot, cryo, ryoshu
- IRC nick: apurvanandan
The phase-wise progress made in GSoC 2020 is described here onwards:
To get initial hands on experience with NetBSD and pkgsrc, the package manager for NetBSD, I worked on importing/upgrading pkgsrc packages, developing kernel module, and patching NetBSD-current source.
- Add new example kernel module: current_time
- devel/afl++: import afl++-2.62c
- cppcheck: update to 1.90
- fio: update to 3.19.
- Add ppoll(): a compatibility wrapper around pollts(2)
In the first phase, I upgraded the Phoronix Test Suite from version 8.8nb1 to the stable version 9.6.1 in pkgsrc-wip and is available as wip/phoronix-test-suite.
I documented test profiles based on its installation, execution and requirement on NetBSD using x86_64 QEMU VM and have maintained the status in this sheet:
pkgsrc-wip commits:
- wip/phoronix-test-suite: import phoronix-test-suite-8.8.1nb1
- wip/phoronix-test-suite: update phoronix-test-suite to 9.6.1
- Added required PHP dependencies & removed TODO file
Test-profile commits:
- Fixed build error on NetBSD for apache-siege-1.0.4
- Fixed build error in coremark-1.0.0
- Fixed mbw-1.0.0 tests for NetBSD
- Removed pnglibconf.h not found build error in brl-cad--1.1.0
- Fixed build error in t-test1-1.0.1 due to missing memalign() in NetBSD
- Added patch for apache-siege-1.0.4
- Patch for mempcpy to memcpy conversion in mbw-1.0.0
- Fixed build error in t-test1-1.0.1 due to missing memalign() in NetBSD
In this phase, I completed the automation of the regression suite made using the Phoronix Test Suite and its integration with Anita.
This was performed in order to provide a web interface from where the user can easily automate NetBSD regression test based on PTS for actual test execution and benchmarking and Anita for automated setting up of Phoromatic clients on QEMU VMs with the latest NetBSD-current binaries.
This all was achieved using the following script as a pre-run script in the test-schedules with pts/idle-1.2.0 as the test-profile on Phoromatic server:
https://gist.github.com/apurvanandan1997/48b54402db1df3723920735f85fc7934
The successful Phoromtic-Anita integration and demonstration for reproducing them have been screen recorded in this video:
In the third phase, I upgraded to the latest stable version Phoronix Test Suite 9.8.0 in pkgsrc-wip and created patches for more test-profiles to resolve their installation and runtime errors on NetBSD.
pkgsrc-wip commits:
Test-profile commits:
- build-ffmpeg-1.0.1: Replace the make -> gmake for compatibility with NetBSD
- mafft-1.5.0: Replaced the make -> gmake for compatibility with NetBSD
- compilebench-1.0.2: Replaced python2 with NetBSD style naming python2.7
- mafft-1.5.0: Patch for replacing /bin/bash interpreter with /usr/pkg/bin/bash
I had been assigned a remote testing machine having Intel 6138 dual processor, 40 cores, 80 threads, 192GB of RAM, I spent time reproducing my automated framework i.e. Phoromatic-Anita Integration on the machine.
I also performed rigorous benchmarking on the remote machine having NetBSD-9 amd64 native installation by running 50 test-profiles on a remote machine and uploaded the results to OpenBenchmarking.org:
I have in-depth documented my phase-wise progress, achievements made, and ways to reproduce my results in the following blogs (third blog to be published soon):
- GSoC Reports: Benchmarking NetBSD, first evaluation report
- GSoC Reports: Benchmarking NetBSD, second evaluation report
- GSoC Reports: Benchmarking NetBSD, third evaluation report (to be published soon)
My end goal of the project that is to integrate Phoronix Test Suite with NetBSD and Anita is complete and its deployment on benchmark.NetBSD.org will be continued to be worked on with the coordination of moderators and merging the wip of Phoronix Test Suite 9.8.0 will be done by the pkgsrc maintainers in next days. I want to thank my mentors and the NetBSD community without whose constant support I wouldn't have achieved the goals.