After using vim and neovim for almost a year now I've come to the point when frustration I had with running test with vim made me think about ideal workflow for running tests (at least in command line, but some rules apply to GUI).
Below you can read my thoughts about ideal workflow, a summary of problems with current workflows (at least that I know of, but I think I've tried them all) and my own new (well, at least I think it's new) solution to them.
I won't consider a case, when your IDE can automatically run approriate test on each code change in background and show you the errors when there are any. The idea is nice, but not very reliable in dynamic languages, and I wouldn't want to be constantly reminded that there is a failure when I'm writing new code and already seen that test failed (in fact when I use TDD I know that it would fail in the first place). One use case when it's actually useful is when it helps you to detect that you've accidentally broken something, but this can be done automatically at commit.
So anyway.
- You need the ability to quickly change test runner command. For example suppose you running a test suite and more than one tests is failing. It's inconvenient to see all errors at once (even in gui) and you would probably want to rerun test in such a way that you can deal with one error at a time.
- After the test is done executing there are two cases. Either all tests are
green or not.
- If there are no errors I'd like to see what tests actually have been executed, so that I would be sure that I've tested the right thing. This may not be required for you. After I've seen what tests have been executed I don't need to see the test runner window and therefore I should be putted back to vim in fullscreen mode.
- If there are errors you need to be able to see both errors and code at the same time. Ideally you should be able to jump automatically between the errors in the code quickly.
- You need a cheap way to repeat the last test, because that's what you do most of the time.
- You should be able to quickly build command that automatically executes last test, test suite, nearest test etc.
- You should be able to quickly jump between code and test results.
- If your test runner can show you useful information about what exactly have gone wrong using colours you should see the colour.
The first option doesn't require any specific plugin, terminal application, settings or anything. It's just using the shell ability to put process in background and foreground. So you can put vim to background, run the command and switch between vim and command line back and forth.
Summary:
- You need to type command manually at least the first time.
- If there are no errors you wouldn't be placed back into code
automatically.
- You need 3 keystrokes to switch from command line to vim ("fg Enter")
- You need 4 keystrokes to rerun the test "C-z Up Up" because the last
command would be "fg".
- You can't see errors and code at the same time.
- There isn't any automatic parsing of test results and no quick jumping
between errors.
- You can see the colour.
The second options also doesn't require anything. Because vim can execute arbitrary shell command you can just enter vim command line with :!{your test runner command}.
Summary:
- You need to type command manually at least the first time.
- If there are no errors you wouldn't be placed back into code
automatically.
- You can't see errors and code at the same time.
- In fact you can't see errors more than one time (AFAIK)
- You can't see the colour.
I've personally stopped using this approaches after about a week of using vim. Because after using intellij for a couple of years I coulnd't cope with this level of inefficiency and it was more productive to use vim-emulator plugin inside intellij than the vim itself.
My next option was to use neovim's terminal emulator.
Summary:
- You can switch between errors and code as fast as you can between any vim
windows (requires some neovim terminal configuration)
- You can see both errors and the code at the same time. If I recall
correctly this made me actually switch from intellij entirely, because
running tests was the last thing that made me use intellij at that point.
- You still need to parse the error output yourself.
- Neovim terminal emulator doesn't support true colour, so if your test
runner (or in my case debugger) using true colour escape codes (because
it's a 21 century you know) you wouldn't see any colour at all.
- To escape from terminal mode you need to press "<C-\><C-n>" and you would
probably remap it like so ":tnoremap <Esc> <C-\><C-n>". Which makes this
impossible to use vi-bindings (which is despite what a lot of people say
is very handy)
- If there are no errors you wouldn't be placed back into code
automatically.
- You can see the colour, but not true colour.
After that I've started to use vim-test plugin. The main advantage it has is ability to automatically run nearest test, last test, test suite etc, so you don't need to manually type any command and it can run the tests via different "strategies" like neovim terminal emulator, tmux split etc. I've used it with neovim strategy (tried others as well).
There are couple of problems with this approach/plugin: - Again, you need to parse errors yourself if you're not using "dispatch" strategy. But more on dispatch later. - Test runner options are fixed (but customizable). By that I mean that if you want to run specific test with different options, you would have to change test runner options and than back and it's easier to just type the command by hand in that case. - Plugin specific problem is that with "neovim" strategy it doesn't returns you to the last pane (which I think is a bug) and "WindowEnter" autocmd isn't issued when you're back to the code buffer. - If there are no errors you wouldn't be placed back into code automatically. - And of course all the problems of neovim terminal emulator
Next group of options is make (vim command) related options (make, dispatch and other plugins that use dispatch)
You see, vim has options to define what command to run when you issuing make command for each file type. The idea here is that you can type ":make" and vim would execute the command you want with options you want, parse it and populate quickfix list with results, so that you can jump between errors fast. The idea is great, but it has one big flaw, vim can't display multiline errors in quickfix list. Which means that if your errors are multiline, you actually can't jump fast to the next error, because the next line in quickfix window would be the next line of the same error. The dispatch plugin is only adding ability to run commands asynchronously to that.
Summary: - You can't quickly change test runner options. - You can't jump quickly between the errors if they are multiline. - You kind of can see the code and the errors, but only with brief messages and without colouring. - If there are no errors you wouldn't be placed back into code automatically.
After realizing that neovim terminal can't show true colour, which I use in pdb++ (python command line debugger) I've decided to run tests manually in tmux split and I've quickly realized that there is a very clear pattern in my workflow.
- I create the split if it wasn't created previously.
- If I run new test, then I type the command by hand.
- If Im rerunning test I press "Up Enter" to repeat last command.
- I stare at the output and use tmux copy-mode (because of the vim-bindings) to scroll.
- I find the error and use keybindings jump to the first tmux pane (it's always vim there)
If there's a pattern it can be automated, right? So that's what I did.
First I've created a shell function that stores id's of panes where I'd like to repeat last command. And the function that repeats the last command in marked panes and selects the last one.
mark_pane() {
if [ -n "$TMUX_PANE" ]; then
sed -i "/$TMUX_PANE/d" /tmp/marked_panes 2>/dev/null
echo ${TMUX_PANE} >> /tmp/marked_panes
fi
}
alias m="mark_pane"
alias unm="rm /tmp/marked_panes 2>/dev/null"And the script to repeat last commands
#! /usr/bin/env zsh
local last_pane=0
for i in $(< /tmp/marked_panes); do
tmux send -t.$i up Enter;
last_pane=$i
done
tmux select-pane -t.$last_paneAnd I've bounded "prefix r" (r for repeat) to execute repeat_last_commands in
tmux
bind-key r run-shell -b repeat_last_commands
So that gives ability to easily rerun last test, but what about other requirements?
After that I've written this
#! /usr/bin/env zsh
setopt pipefail
comm=$1
shift
case $comm in
pytest)
force_color="--color=yes"
;;
*)
force_color=""
esac
$comm $force_color $@ | tee /tmp/out
if [[ $? -eq 0 ]]; then
sleep 2
tmux select-pane -t.0 \; resize-pane -Z
else
line_count=$(wc -l /tmp/out | awk '{print $1}')
if [[ $line_count -gt $LINES ]]; then
grep -q -e '(Pdb\(++\)\?)' /tmp/out
if [[ $? -eq 1 ]]; then
less +G /tmp/out
fi
fi
tmux select-pane -t.0
fi
unsetopt pipefailI prepend this command to the test runner like for example wrapper pytest -x
and it does the following:
- If tests are green it would show me tmux split pane with tests output for
2 second and then bring me back into vim pane in zoom (so I would only see
vim pane).
- If there are errors and there wasn't breakpoint in my code it would check
if pane size is enough to show tests report. If not it would show them
with less (again because of vi-bindings).
- If there was breakpoint in the code or pane size contains all test report
or after I quit less it would bring me right back into zoomed vim pane.
So If there are errors I can easily scroll with vim bindings, search text in test report, switch between test results and code via tmux keybindings, jump between errors by typing :e +{some line numer} {file_with_error}. And when I'm done I'm pressing "prefix s" (s for switch) to switch in zoomed vim.
One thing that isn't working with this approach is parsing test results and jumping quickly between errors, but it isn't really working in any approach.
Another thing is that you need to type new test command by hand which isn't required in vim-test and similar plugins. You can actually make my approach work with vim-test, because it's just using tmux, but right now I don't think it's worth it. I rerun the same test 99% of the time. So I've actually deleted vim-test plugin.
What I also like about this approach is that it's a more general solution. So for example it can be used to run some manual tests, where you need for example to start some server and some client. To kill them both I use this function bounded to "prefix k" (k for kill of course).
kill_in_marked_panes() {
for i in $(< /tmp/marked_panes); do
tmux send -t.$i C-c \;
done
}You can see all this in action here
If you like this approach checkout my dotfiles