This document contains instructions for the artefact evaluation. The draft paper is available at kcsrk.info/papers/pldi18-memory.pdf.
The main evaluation in the paper is the overhead of compiling our proposed
memory model to relaxed architectures such as ARM and POWER (Fig 7 on page 10).
To this end, we have set up an ARM (aarch64) and POWER (ppc64) machine and
have provided the ssh keys (pldi-aec-rsa) to access the machine.
Copy the key file to your machine and change its permission to
chmod 600 pldi-aec-rsa
Then you can log into the aarch64 machine as:
ssh -i pldi-aec-rsa pldi18aec@arm64.bench.ocamllabs.io
and the ppc64 machine as:
ssh -i pldi-aec-rsa pldi18aec@ppc64le.dev.ocamllabs.io
This section describes the content and organisation of the benchmarks. You may
skip this section and move to the next. In the rest of this section, I will
assume that you are logged into the aarch64 machine. This instructions for
benchmarking on ppc64, unless explicitly noted otherwise.
In the home directory, you should have 2 directories:
$ ls
logs ocamlbench-scripts
We will use ocamlbench-scripts/run-bench.sh to run the benchmarks. The
benchmark results are stored in the logs directory.
The compiler variants we will test can be listed with the opam switch command
as follows:
$ opam switch
# switch compiler description
4.07.0+trunk+bench ocaml-variants.4.07.0+trunk 4.07.0+trunk+bench
4.07.0+trunk+branch-after-load-aarch64+bench ocaml-variants.4.07.0+trunk+branch-after-load-aarch64 4.07.0+trunk+branch-after-load-aarch64+bench
4.07.0+trunk+dmb-before-store-aarch64+bench ocaml-variants.4.07.0+trunk+dmb-before-store-aarch64 4.07.0+trunk+dmb-before-store-aarch64+bench
4.07.0+trunk+profile-reads-and-writes+bench ocaml-variants.4.07.0+trunk+profile-reads-and-writes 4.07.0+trunk+profile-reads-and-writes+bench
4.07.0+trunk+sra-aarch64+bench ocaml-variants.4.07.0+trunk+sra 4.07.0+trunk+sra-aarch64+bench
default ocaml-system.4.02.3 default
operf ocaml-system.4.02.3 operf
Of these, the compiler switch 4.07.0+trunk+profile-reads-and-writes+bench is
used to generate the benchmark characteristics graph in Fig 7(a). The other
switches starting with 4.07.0+ are the variants evaluated in the graph in Fig
7(b).
All the information about the compiler variants and benchmarks are found from a
special opam remote named benches:
$ opam repo -v
[NOTE] Use '--all' to see all configured repositories independently of what is selected in the current switch
=-=- Repository configuration for switch operf =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
1 benches git+https://github.com/kayceesrk/ocamlbench-repo#aarch64
2 default https://opam.ocaml.org/2.0
You can browse the benchmarks in the directory packages in
ocamlbench-repo.
The various compiler variants used are listed in packages/ocaml-variants. The
opam files have information about the location of the compilers. The compiler
variants are:
- trunk
- branch after load (BAL) aarch64
- "dmb lb" before store (FBS) aarch64
- strong release/acquire (SRA) aarch64
- branch after load (BAL) ppc64
- "lwsync" before store (FBS) ppc64
- strong release/acquire (SRA) ppc64
- profile reads and writes
After you have logged into aarch64 or ppc64 machine, make sure you are in the operf switch.
$ opam switch operf
Start a tmux session as the benchmarks take around 4 hours to run.
$ tmux
If the ssh session gets terminated, you can come back and attach to the same
session with a tmux attach. Initiate the benchmarking script,
$ cd ocamlbench-scripts
$ bash ./run-bench.sh --nowait
Once you have finished running the experiments, you can generate the graphs.
There are 3 python scripts make_fig_7a.py, make_fig_7b.py and
make_fig_7c.py in the ocamlbench-scripts directory in order to generate the
sub-figures in Fig 7. make_fig_7a.py and make_fig_7b.py need to be run on
aarch64 machine and make_fig_7c.py on ppc64 machine.
$ python make_fig_7a.py
Writing 'fig_7a.pdf'
and similarly for other figures. scp the pdfs to your local machine to view them.