breakdown of intelligent test runner

gistfile1.md

from a user perspective, we want something that can run in the context of a cabal project.

it should have some idea of a hierarchy of tests (could be from tasty or hspec, i wouldn't bother trying to make it general). when it starts up, it ought to try to run all the tests, noting which ones fail (and also which ones fail to compile: ideally, you would be able to achieve local progress even if there are compile errors in other parts of the program, though it's not essential.)

when a test is failing, it should scope down to that file, and keep running that test on file change until it works. when it succeeds, the whole test suite one layer up should run, and so on up to the root of the test tree.

When a test file changes, that's easy - we just run it. When another file changes, we need to trace back all the test files that are affected by it, and add them to the list of tests to be run.

expected pain points:

• inotify equiv on mac - https://github.com/nkpart/fsevents-api maybe? would be nice to have a general layer.
• convincing cabal to give you the dependency tree of what needs to recompiled. would be nice to avoid a full "cabal build" on every run, as it can be slow.

wait, https://hackage.haskell.org/package/fsnotify seems a good cross-platform one (thanks @nkpart)

Yep, fsnotify is great. It's the one I have in my notes here. Not finding anything in the way of usable code though... mostly only me arguing with myself in a text file about a few things that I liked from sbt.

dump it here then, good to know what's already been tried.

OK, I found the only working part I wrote. It watches a filepath and runs "cabal build" on changes. I had started work on a small module to either whitelist or blacklist files to watch for (right now it's far too eager; if emacs saves a file in-progress everything recompiles), and then I moved to cabal integration and didn't get anywhere useful.

Integrating directly with cabal sounds like the smartest answer, but I haven't had the time to work through what that would even look like. An alternative I was working on was a basic config file that would let you specify what command to use for tests, and then a haskeline prompt that would let you either watch for changes and recompile or watch for changes and run tests. Being able to specify tests in a more granular format like @mwotton mentioned would be great too.

import System.Cmd
import System.FSNotify
import System.Directory
import System.Environment
import Control.Concurrent

import Prelude hiding (FilePath)

import Filesystem.Path
import Filesystem.Path.CurrentOS (decodeString)


-- calls cabal build to recompile, but isn't aware of cwd or cabal
recompile :: Event -> IO ()
recompile ev = print ev >> system "cabal build" >> return ()  -- ignore exit code for now

-- a basic watcher to recompile on changes to anything not in dist
-- (needs to be more sophisticated in terms of what it watches/ignores)
watcher :: WatchManager -> FilePath -> IO ()
watcher manager fp = do
  putStrLn "Watching!"
  watchTree manager fp notDist recompile

-- make sure we ignore events affecting the dist/ directory
notDist :: Event -> Bool
notDist ev = decodeString "dist" `notElem` splitDirectories (path ev)
  -- an example of what we can pattern match on, even though we return `fp`
  -- regardless.
  where path (Added    fp _) = fp
        path (Modified fp _) = fp
        path (Removed  fp _) = fp

-- set current directory was used to accommodate hsenv, but is very fragile
main = do
  [filepath] <- getArgs
  let workingDir = decodeString filepath
  setCurrentDirectory filepath
  manager <- startManagerConf (Debounce 0.1)
  watcher manager workingDir
  putStrLn "hit enter to stop"
  getLine
  stopManager manager

https://github.com/supki/guard-haskell could be mined too (ping @supki)

so, here's what's working for me with hspec right now:

i have a test suite called "tests", and i run it under cabal repl.

cat <(echo "hi") <(inotifywait -m -r .  -e CLOSE_WRITE |grep --line-buffered '\.hs$'   | grep -v --line-buffered flymake) | collapse | while read x; do 
  echo ":reload\n:main --rerun" ; 
done | cabal repl tests

collapse.hs is a little utility I wrote:

orb ➜  ~/projects/meanpath/spider/lambdaspider git:(pipes) ✗ cat src/collapse.hs                              sandboxed 
import           Control.Monad
import           System.IO

main :: IO ()
main = do
  hSetBuffering stdout LineBuffering
  forever $ do
    getLine >> emptyStdin
    putStrLn "x"

emptyStdin :: IO ()
emptyStdin = do
  res <- hReady stdin
  when res $ getLine >> void emptyStdin

... this seems to fulfil my needs at the moment. Deficiencies: it runs everything that failed, but i think I can adapt my workflow to only have one failing test at a time. It also relies on inotifywait, which only exists on linux.

Still, it serves as a fairly clear & easy starting point...

((inotifywait -m -r .  -e CLOSE_WRITE |grep --line-buffered '\.hs$'   | grep -v --line-buffered flymake | collapse | while read x; do echo ":reload\n:main --rerun" ; done) & export IJOB=$!  ;  cat -;  pkill -TERM -P $IJOB)  | cabal repl tests

this one also gives you the ability to interact with GHCi at the prompt. Control-d will exit cleanly, Control-c will most definitely not.

For the sake of learning, I thought I'd have a go at implementing the equivalent of your bash pipeline in Haskell and I'd be grateful to get some recommendations on how this could be improved.

https://gist.github.com/schristo/710584efcfbdbb310454

This allows for something like the following, where bar.foo was created and then removed:

~/b/watcher ❯❯❯ ./watcher '**/*.foo'
+ /Users/scott.christopher/build/watcher/bar.foo
- /Users/scott.christopher/build/watcher/bar.foo

Or carrying on with your example, the following should be possible:

# watcher defaults to watching **/*.hs
./watcher | while read; do echo ":reload\n:main --rerun"; done | cabal repl tests

I thought this could be modified further to accept an FilePath -> IO () handler to allow running tests directly rather than just printing the paths to stdout. Given this would be evaluated whenever a FSNotify event is fired, I'm not sure how I would model the strategy to prevent executing the handler concurrently. One solution would be to killed off the currently running tests using a combination forkIO and killThread and then restart them. Alternatively, the running tests could block and prevent any further tests from executing until finished. Could/should this potentially be modelled using STM?

In other news, I also discovered http://hackage.haskell.org/package/tasty-rerun, which configures Tasty to execute tests based on the outcome of the previous test run (i.e. only execute previously failing tests), which may help close the gap of outstanding functionality.