ycm vim tests

test.md

Quick Start

## Running the tests

You have to be on MacOS or Linux. Sorry, Windows testing is not supported.

• Ensure you have at least the Vim vresion in YCM_VIM_VERSION (in azure-pipelines.yml)
• Ensure ycmd is compiled and clangd is enabled install.py --clangd-completer
• Install the test python deps (pip install -r python/test_requirements.txt)
• Run ./run_vim_tests

Overview

The test framework is based on the "new style" Vim tests. These are the tests that are used to test Vim itself. There is good info on this in :help testing.

There's also some useful info in Vim's test readme.

In short, the test framework runs Vim, sources the test script, then executes all of the test functions. Test functions should use the vim built-in assert functions to report errors (see :help new-style-testing) and should attempt to reset any changes they make at the end of the function.

You can add set-up and tear-down functions, and can skip tests by throwing a message starting with the word 'Skipped' (e.g. throw "SKipped: <message>").

Test Framework

The test framework has the following components:

• A vim 'plugin' (in test/lib) containing the framework itself, comprising:
  • run_test.vim, which wraps the test functions and executes them, reporting failures.
  • Some basic support functions in plugin/shared.vim (from Vim)
  • Some screendump support functions in plugin/screendump.vim (from Vim)
  • Some YCM-specific autoloaded functions in autoload/youcompleteme/test/*
• A script to run the tests, including specific test script and function
• The actual test scripts in test/*.test.vim
• CI integration for azure.

Test Scripts

The basic structure of a test is as follows:

function! SetUp()
  " ... set g:ycm_* options here...
  call youcompleteme#test#setup#SetUp()
endfunction

function! ClearDown()
  call youcompleteme#test#setup#CleanUp()
endfunction

function! Test_MyTest()
  " test goes here, e.g.
  aseert_true( pyxeval( 'ycm_state.ServerIsHealthy()' ) )
endfunction

function! Test_MyOtherTest()
  assert_false( 0 )
endfunction

Test scripts are placed in src/test and are named *.test.vim. Each test script can contain any number of individual tests, each of which is a Vim function named Test_<test name>. Test functions are run in arbitrary order, so must not rely on each other.

Each test script is a fixture, but setup and teardown is done for each and every test. Global (one-time) setup can be done at script level, but this is not recommended.

Set up and tear down functions are run before and after tests. You can define one for the whole script, which is run before every test, and a per-test setup/tear down function which is run before both the global setup function and the test function.

To explain, for each function in the script named Test_<test name>, run_test.vim does the following:

• If there is a function named SetUp_Test_<test name>, call it
• If there is a function named SetUp, call it
• Call Test_<test name>
• If there is a function named TearDown, call it
• If there is a function named TearDown_Test_<test name>, call it

If any of these functions raises an exception, it is added to v:errors. The exception is if the test is called Test_nocatch_<test name>, in which case, exceptions are not caught be the test and should be handled by the test function itself.

If at the end of this, the v:errors list is non-empty, the test test name is marked as failed.

If a test fails, run_test.vim attempts to print out all of the log files that YCM's ycm_state object knows about.

The test plugin

The "plguin" provides a handful of things, some of which were simply ported from Vim's test framework, and some were writted specifically for YCM.

Ported from Vim

These are general purpose functions which are commonly used:

• WaitForAssert: This one is the most useful. It takes a callable (usually a lambda) and waits for it to return 0, but allows the Vim event loop to run in betwen calls. This is key to ensuring that the YCM code can execute while the script is actively trying to test it. NOTE: It waits for the function to return 0, *NOT for the assert to be true/v:errors to be empty!

YCM-specific

The autoload functions perform some useful common YCM-sepcific stuff such as setup and teardown, and will likely be built out over time as the suite increases in size and complexity.

Tips and tricks

Things that you need to know to write tests effectively:

• Don't forget to :%bwipeout! at the end of each test function.

• Understand the arguments to feedkeys. Importantly, if you want it to behave the way you think it should, use feedkey( "...", 'xt' ). This makes it wait for the input to be actually read before returning, which is important for tests. See :help feedkeys for the other options.

• Remember that test scripts a vim functions. I know that sounds obvious, but things like "insert mode completions" are hard to test with functions which are typically not invoked in insert mode. In order to actually do anything in insert mode, you need to do the following:

  • If you want TextChangedI to fire, call test_override( 'char_avail', 1)

  • Normally, feedkeys would exit insert mode if you enter it. Tell it not to by passing the ! flag.

  • Now that you've left Vim in insert mode, your test will hang forever unless you exit insert mode, so define a function and call it via a timer or other async callback which performs the actual asserts, and ends by calling feedkeys( "\<ESC>" ) to exit insert mode.

  • Check completion.test.vim for an example.

• Remember that the assert* functions don't throw exceptions. They return 0 on success, and return nonzero on failure, populating v:errors.

• Throwing exceptions in tests does fail the test, but this is not recommended because it skips the (local) teardown code.

Coverage

The test suite supports covimerage coverage testing. Just set the COVERAGE environment variable when running run_vim_tests.