data.csv

rmd.nix

{ pkgs ? (import <nixpkgs> {})}:

with pkgs;

let 
  template = pkgs.writeText "template.Rmd" ''

# iperf-1500 inter-branch comparison

This [Rmarkdown](http://rmarkdown.rstudio.com/) document presents
comparative `iperf-1500` benchmark scores for a few branches:

- `master` is the v2016.04 release.
- `next` is the first release candidate for `v2016.05`
- `next1024` is the second release candiate for `v2016.05`.
- `newpkt` is a variation on the `struct packet` layout.
- `newcksum` is with a new checksum implementation merged.

These results were measured by manually running the `iperf-1500`
benchmark on each branch 50 times. This was done because SnabbBot had
suggested the results were suspicious.

This is the R code to load the data for processing:

```{r}
library("ggplot2")
d <- read.csv(file="${./data.csv}", head=TRUE, sep=",")
df <- as.data.frame(d)
```

## Visual comparison

Here is a visual summary of the performance of each tested branch. There are three visualizations overlayed for each branch:

- A black dot marks the average ([arithmetic mean](https://en.wikipedia.org/wiki/Arithmetic_mean)).
- A [scatter plot](https://en.wikipedia.org/wiki/Scatter_plot) shows each individual result.
- A [violin plot](https://en.wikipedia.org/wiki/Violin_plot) suggests the distribution of results and how common each result is.

The scale has been chosen to suit the specific benchmark. This is running `iperf` between two VMs with MTU=1500 on a 1.8GHz Ivy Bridge CPU. The ideal results would be consistently above 10 Gbps.

```{r}
p <- ggplot(df, aes(branch, gbps))
p <- p + expand_limits(y=0) + expand_limits(y=10)
p <- p + ggtitle("iperf MTU=1500")
p <- p + geom_violin()
p <- p + geom_jitter(color="blue", alpha=0.5)
p <- p + stat_summary(fun.data=mean_se)
p <- p + scale_y_continuous(breaks = seq(from=0, to=10, by=1), minor_breaks = NULL)
p
```

One interpretation of these results:

1. The results are far below the target of 10 Gbps. This is something that we have accepted for slow Ivy Bridge processors but it is very striking.
2. The results vary for unknown reasons. Why do we not see the same result for the same code every time?
3. The `next` branch seems to have a performance limit below the `master` branch. This does not seem to be true of the `next1024` branch (though the variation makes it hard to be entirely certain).

## Summary by branch

```{r}
by(df, df["branch"], summary)
```

## Tukey's Test

This [Tukey's Test](https://en.wikipedia.org/wiki/Tukey%27s_range_test) compares the averages for the results and provides a 95% confidence interval to account for variation.

```{r}
TukeyHSD(aov(gbps ~ branch, data=df))
```

'';
in {
  sample = pkgs.runCommand "rmarkdown-example" { buildInputs = [ rPackages.rmarkdown rPackages.ggplot2 R pandoc which ]; } ''
    cp ${template} template.Rmd
   
    echo "library(rmarkdown);render('template.Rmd')" | R --no-save
    mkdir -p $out/nix-support
    cp template.html $out
    echo "file HTML $out/template.html"  >> $out/nix-support/hydra-build-products;
  '';
}