Writing benchmarks is not easy. Nonius simplifies certain aspects but you'll always need to take care about various aspects. Understanding a few things about the way nonius runs your code will be very helpful when writing your benchmarks.
First off, let's go over some terminology that will be used throughout.
- User code: user code is the code that the user provides to be measured.
- Run: one run is one execution of the user code.
- Sample: one sample is one data point obtained through measurement of the time it takes to perform a certain number of runs. One sample can consist of more than one run if the clock available does not have enough resolution to accurately measure a single run. All samples for a given benchmark execution are obtained with the same number of runs.
Now I can explain how a benchmark is run in nonius. There are three main steps, though the first does not need to be repeated if there are more than one benchmark to run.
-
Environmental probe: before any benchmarks can be run, the clock's resolution is estimated. A few other environmental artifacts are also estimated at this point, like the cost of calling the clock function, but they almost never have any impact in the results.
-
Estimation: the user code is executed a few times to obtain an estimate of the amount of runs that should be in each sample. This also has the potential effect of bringing relevant code and data into the caches before the benchmark starts.
-
Measurement: all the samples are collected sequentially by performing the number of runs estimated in the previous step for each sample.
This already leads to one important rule for writing benchmarks for nonius: the benchmarks must be reentrant. The user code will be executed several times, and the number of times it will be executed during the estimation step cannot be known beforehand since it depends on the time it takes to execute the code. User code that cannot be executed repeatedly will lead to bogus results or crashes.
Sometimes the optimizer will optimize away the very code that you want to
measure. There are several ways to use results that will prevent the optimiser
from removing them. You can use the volatile keyword, or you can output the
value to standard output or to a file, which require the program to actually
generate the value somehow.
Nonius adds a third option. The values returned by any function provided as user
code are guaranteed to be evaluated and not optimised out. This means that if
your user code consists of computing a certain value, you don't need to bother
with using volatile or forcing output. Just return it from the function.
That helps in keeping the code in a natural fashion.
To be added: what's required, what's optional, actual examples