git bisect to find first commit on R-devel that results in test failure

README.md

Test failure on Pool package

This repository contains information to reproduce a test failure on the pool package.

First, clone this gist and enter the directory. Then build the Docker image:

docker build -t pooltest .

Then start up a container and run RD CMD check on the package (RD invokes R-devel):

docker run --rm -ti pooltest /bin/bash

RD CMD INSTALL /pool_0.1.0.tar.gz --install-tests
cd /usr/local/lib/R/site-library/pool/tests
RD -e 'source("testthat.R")'

Part of the output looks like this:

20170404-200350.557210-18 20170404-200400.547020-17 20170404-200530.546788-16 
20170404-200350.557210-18 20170404-200400.547020-17 20170404-200530.546788-16 ----
20170404-200350.557210-18 20170404-200400.547020-17 20170404-200530.546788-16 
20170404-200350.557210-18 20170404-200400.547020-17 20170404-200530.546788-16 ----
20170404-200400.547020-17 20170404-200530.546788-16 
20170404-200400.547020-17 20170404-200530.546788-16 ----
20170404-200530.546788-16 
20170404-200530.546788-16 ----
1. Failure: pool scheduler: schedules things in the right order (@test-scheduling.R#24) 
`results` not identical to 1:3.
Lengths differ: 4 vs 3


===== Results:  1 2 3 1 =====

Using git bisect on R-source

This repository also contains scripts to perform a git bisect on R itself, to find the first commit where pool's tests fail.

First, clone this gist, as well as the r-source repository. They can be in sibling directories. I named the directory for this gist pool-test.

Next, modify test-script.sh so that R_SOURCE_DIR points to the absolute path of your r-source directory. For example:

R_SOURCE_DIR=/Users/winston/nobackup/r-source

From this directory, build the Docker image:

docker build -t pooltest .

On the host, in the r-source directory, run the git bisect. Make sure that the path to test-script.sh below points to this directory.

cd ../r-source
git bisect start trunk a1ccebfa8
git bisect run ../pool-test/test-script.sh

(Note that a1ccebfa8 is the commit for tags/R-3-3-branch. I did a shallow checkout and didn't get the tags/ for some reason.)

This procedure prints out a lot of stuff, so it can be useful to write the output to a file:

mkdir ../pool-test/logs
git bisect run ../pool-test/test-script.sh > ../pool-test/logs/run_log.txt 2>&1

Then in another terminal you can monitor the output with:

tail -f bisect_log.txt

At the end, you can save the bisect logs with:

git bisect log > ../pool-test/logs/bisect_log.txt

build-r-source.sh

#!/bin/bash

set -e

cd /r-source

# tools/rsync-recommended
./configure --without-recommended-packages
(cd doc/manual && make -j4 front-matter html-non-svn)

echo -n 'Revision: ' > SVN-REVISION
git log --format=%B -n 1 \
  | grep "^git-svn-id"    \
  | sed -E 's/^git-svn-id: https:\/\/svn.r-project.org\/R\/[^@]*@([0-9]+).*$/\1/' \
  >> SVN-REVISION
echo -n 'Last Changed Date: ' >>  SVN-REVISION
git log -1 --pretty=format:"%ad" --date=iso | cut -d' ' -f1 >> SVN-REVISION

make -j4
make install
make distclean

RD -e "install.packages(c('DBI', 'R6', 'testthat'), repos = c(CRAN='https://cloud.r-project.org'), Ncpus=4)"

RD CMD INSTALL /pool_0.1.0.tar.gz --install-tests

Dockerfile

FROM rocker/r-devel

ARG DEBIAN_FRONTED=noninteractive
RUN apt-get update && \
    apt-get install -y git libssl-dev libcurl4-openssl-dev libxml2-dev
RUN RD -e "install.packages(c('DBI', 'R6', 'testthat'), Ncpus=4)"

COPY build-r-source.sh /
COPY test-build.sh /
COPY pool_0.1.0.tar.gz /
COPY test-pool.R /

pool_0.1.0.tar.gz

test-build.sh

#!/bin/bash

set -e

# Build R and install needed packages
./build-r-source.sh || exit 125 # git bisect skip if can't build R

# Run tests
cd /usr/local/lib64/R/library/pool/tests
RD -e 'source("testthat.R")'

test-script.sh

#!/bin/bash

set -e

R_SOURCE_DIR=/Users/winston/nobackup/r-source

docker run --rm -ti -v $R_SOURCE_DIR:/r-source --name pooltest pooltest /test-build.sh

Logs from running this are at https://gist.github.com/wch/4ea05e99997b2d56743cd64ac51b9da1

The first bad commit is wch/r-source@7c950ac