Verify your git-bisect run script

README.md

When using git bisect run ./test.sh it's not uncommon to get the condition in test.sh wrong. This can cause a lot of wasted time as you only notice the mistake hours later when bisect finishes and points at the wrong commit.

This Python script runs the test script against the commits you already marked as good or bad in the bisect session (either with 'git bisect good/bad' or as part of the 'git bisect start' command) and ensures your test script returns the correct status code.

$ git bisect start v12.0 v11.0
Bisecting: 9035 revisions left to test after this (roughly 13 steps)

$ git-bisect-check ./test.sh
Commit 403926033d001b5279df37cbbe5287b7c7c267fa should be bad, checking...
HEAD is now at 403926033d00
Testing...
Commit 403926033d001b5279df37cbbe5287b7c7c267fa marked as bad but script returns 0 (is the script condition backwards?)

# oops
$ vim test.sh # fix it!

$ git-bisect-check ./test.sh
Commit 403926033d001b5279df37cbbe5287b7c7c267fa should be bad, checking...
HEAD is now at 403926033d00
Testing...
Commit a110e6b9a361653a042e3f5dfbac4c6105693789 should be good, checking...
Previous HEAD position was 403926033d00
HEAD is now at a110e6b9a361
Testing...
All good!

# check passed, we can do the actual bisect now
$ git bisect run ./test.sh

git-bisect-check.py

#!/usr/bin/python3

import subprocess
import re
import sys

if len(sys.argv) < 2:
    print("Usage: git-bisect-check runscript [args]", file=sys.stderr)
    sys.exit(1)

output = subprocess.check_output(["git", "bisect", "log"])
commits = []

for line in output.split(b"\n"):
    line=line.strip().decode('utf8')
    m = re.match("# (bad|good): \[([0-9a-f]+)\]", line)
    if m:
        commits.append((m.group(1), m.group(2)))

retval = 0

for t, c in commits:
    print(f"Commit {c} should be {t}, checking...")
    subprocess.check_call(["git", "checkout", c])
    result = subprocess.run(sys.argv[1:])
    if result.returncode == 125:
        print("???")
        retval = 2
    elif result.returncode == 0 and t == 'bad':
        print(f"Commit {c} marked as bad but script returns 0 (is the script condition backwards?)")
        sys.exit(1)
    elif result.returncode != 0 and t == 'good':
        print(f"Commit {c} marked as good but script returns non-zero (is the script condition backwards?)")
        sys.exit(1)

if retval == 0:
    print("All good!")

sys.exit(retval)