Skip to content

Instantly share code, notes, and snippets.

Last active February 15, 2016 12:18
  • Star 4 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
Reproducing (un)reproducibility results

edit: see

Not deserving of a full post, but nonetheless worth writing about: @ongardie, @aalevy, and a few others on Twitter were surprised by the number of papers that were flagged as "not reproducible" according to the recent study at Digging deeper, it appeared that 1.) "code builds" is the standard for reproducibility in this study and that 2.) many broken builds were the result of missing dependencies on the researchers' systems.

I tried reproducing a few of the authors' "unreproducible" results. It's hard to vet 600+ research code repositories, but, with a little effort (< ~10 minutes each?), I was able to get all of the following to actually build (on Ubuntu 13.10). This doesn't inspire confidence in the reproducibility of the study results.


tar -xzf postgresql*
cd postgresql-9.2.7/; ./configure; make; sudo make install
# hack for paths
sudo chown -R ubuntu /usr/local/pgsql
cd ..; wget; tar -xvf bismarck.tar.gz; cd bismarck;
# change path per
sed -i 's,/home/bismarckvm/Desktop/dependencies/postgresql,/usr/local/pgsql,' bismarck.path;
source ./bismarck.path; make
git clone git://
# dropped libboost version, added libzookeeper-mt-dev
sudo aptitude install build-essential git-core doxygen libboost-all-dev libpcre3-dev protobuf-compiler libprotobuf-dev libcrypto++-dev libevent-dev libzookeeper-mt-dev
# hack for zk path on ubuntu
sudo ln -s /usr/lib/x86_64-linux-gnu/libzookeeper_mt.a  /usr/local/lib/libzookeeper_mt.a
cd ramcloud; make
# if compiler warnings cause compilation to halt, remove any references to -Werr from Makefile
  • "Probabilistically Bounded Staleness for Practical Partial Quorums" from VLDB 2012
    • Reported build behavior in report: "the implementation works"
    • Probable cause of build error: no idea why this was marked as not building since it's a web demo (though we do have publicly linked patches and not one but two GitHub repos for our actual code)
    • Reproducible? no, unclear why this was marked as "Build Fails". Moreover, the authors didn't attempt to build the other code or email me (or the other authors). FWIW, our Java repo builds:
sudo apt-get install ant openjdk-7-jdk
git clone
cd cassandra-pbs; ant
  • "Reusing debugging knowledge via trace-based bug search" from OOSPLA 2012

  • Reported build behavior in report: "Not sure about the syntax, hence I was getting errors everytime I tried running a query. Otherwise, the tool looks good. Benefit of Doubt can be given to the developer that it is working."

  • Reproducible? not really--by the description in the report, this should have been marked as "Builds". I'm not sure why it's marked as "Build Fails".

Copy link

IMO, the config/setting of a research prototype could come as pre-configured, loaded with data, and ready-to-run, Vagrant or Docker images so that a third party can readily reproduce experiments, besides evaluating the code and playing with its data.

The use of Vagrant (or Docker) will not solve this problem completely, but will definitely be a step forward. I know of the stress surrounding the writing of a paper, and sometimes it will not be possible to config this image before submitting the paper, but we can even simulate a Cassandra cluster on Docker so why not resort to this kind of tool?

It will make the simulation of many nodes on cloud systems even more easier! Maybe, even going as far as we having an open repository of research Virtualbox/VMWare images on the web, with bibtex reference to their respective papers. ;)

Otherwise, it's very good that you did a counter proof of this paper. :)

Copy link

camild commented Mar 20, 2014

Good job Peter!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment