breakdown of intelligent test runner

gistfile1.md

from a user perspective, we want something that can run in the context of a cabal project.

it should have some idea of a hierarchy of tests (could be from tasty or hspec, i wouldn't bother trying to make it general). when it starts up, it ought to try to run all the tests, noting which ones fail (and also which ones fail to compile: ideally, you would be able to achieve local progress even if there are compile errors in other parts of the program, though it's not essential.)

when a test is failing, it should scope down to that file, and keep running that test on file change until it works. when it succeeds, the whole test suite one layer up should run, and so on up to the root of the test tree.

When a test file changes, that's easy - we just run it. When another file changes, we need to trace back all the test files that are affected by it, and add them to the list of tests to be run.

expected pain points:

• inotify equiv on mac - https://github.com/nkpart/fsevents-api maybe? would be nice to have a general layer.
• convincing cabal to give you the dependency tree of what needs to recompiled. would be nice to avoid a full "cabal build" on every run, as it can be slow.

((inotifywait -m -r .  -e CLOSE_WRITE |grep --line-buffered '\.hs$'   | grep -v --line-buffered flymake | collapse | while read x; do echo ":reload\n:main --rerun" ; done) & export IJOB=$!  ;  cat -;  pkill -TERM -P $IJOB)  | cabal repl tests

this one also gives you the ability to interact with GHCi at the prompt. Control-d will exit cleanly, Control-c will most definitely not.

For the sake of learning, I thought I'd have a go at implementing the equivalent of your bash pipeline in Haskell and I'd be grateful to get some recommendations on how this could be improved.

https://gist.github.com/schristo/710584efcfbdbb310454

This allows for something like the following, where bar.foo was created and then removed:

~/b/watcher ❯❯❯ ./watcher '**/*.foo'
+ /Users/scott.christopher/build/watcher/bar.foo
- /Users/scott.christopher/build/watcher/bar.foo

Or carrying on with your example, the following should be possible:

# watcher defaults to watching **/*.hs
./watcher | while read; do echo ":reload\n:main --rerun"; done | cabal repl tests

I thought this could be modified further to accept an FilePath -> IO () handler to allow running tests directly rather than just printing the paths to stdout. Given this would be evaluated whenever a FSNotify event is fired, I'm not sure how I would model the strategy to prevent executing the handler concurrently. One solution would be to killed off the currently running tests using a combination forkIO and killThread and then restart them. Alternatively, the running tests could block and prevent any further tests from executing until finished. Could/should this potentially be modelled using STM?

In other news, I also discovered http://hackage.haskell.org/package/tasty-rerun, which configures Tasty to execute tests based on the outcome of the previous test run (i.e. only execute previously failing tests), which may help close the gap of outstanding functionality.