A discussion about formal unit testing in notebooks

2021-testing-in-notebooks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ccfea32",
   "metadata": {},
   "source": [
    "# formal interactive notebook testing\n",
    "\n",
    "This post is for notebook authors who need to ship software, and anyone who wants to start learning more about testing in python. We'll focus on Python's standard library unit testing tools interactively from the notebook. There are many flavors of testing, but our focus remains on unit testing providing quanititative metrics about the fitness of program.  Readers will leave understanding how to prototype and simulate writing unit tests in interactive notebooks.\n",
    "\n",
    "\n",
    "It turns out notebooks can be valuable testing because. A common motivation for using notebooks is to _test an idea_. Without formal conventions, notebooks results in scatter-shot code that informally verifies an idea. In this post, we discuss how to mature informal notebooks into formal unit test conventions. With practice, an effective use of the notebook is to compose code and formal tests that can be moved into your project's module and testing suite; you do have tests don't you?\n",
    "\n",
    "## why are formal tests valuable\n",
    "\n",
    "Tests are investments, and testing over time measures the return on investment. Testing promotes:\n",
    "\n",
    "* longevity of ideas\n",
    "* protection from upstream changes\n",
    "* value to you and consumers of your software\n",
    "* health metrics when used in continuous deployment\n",
    "\n",
    "> learn more about the motivation for testing [The Hitchhiker's Guide to Python - Testing your code][hitchhiker]\n",
    "\n",
    "[hitchhiker]: https://docs.python-guide.org/writing/tests/ \"a very good resource for learning about testing for python programmers\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d2b4ed3",
   "metadata": {},
   "source": [
    "## testing is standard\n",
    "\n",
    "Most programming languages come with unit testing abilities that allow authors to make formal assertions about the expectations of their code. In python, [`doctest`] and [`unittest`] are builtin libraries that enable testing; meanwhile, `pytest` is the choice of popular projects in the broader python community. You will not need extra dependecies besides a notebook interface and python to apply the ideas from this post.\n",
    "\n",
    "> we will not discuss testing notebooks in `pytest` in this document, but if you want to read ahead you can look at [`nbval`], [`importnb`], or [`testbook`] for different flavors of notebook testing.\n",
    "\n",
    "[`doctest`]: https://docs.python.org/3/library/doctest.html \"the python standard library documentation for doctest\"\n",
    "[`unittest`]: https://docs.python.org/3/library/unittest.html \"the python standard library documentation for unittest\"\n",
    "[`pytest`]: https://docs.pytest.org/ \"the python framework for running tests\"\n",
    "[`nbval`]: https://nbval.readthedocs.io/en/latest/ \"a pytest extension for testing notebook inputs and outputs by excluding notebooks\"\n",
    "[`importnb`]: https://github.com/deathbeds/importnb \"a pytest extension for extracting tests for notebooks as modules\"\n",
    "\n",
    "[`testbook`]: https://github.com/nteract/testbook \"testbook is a unit testing framework extension for testing code in notebooks\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9a6e77e",
   "metadata": {},
   "source": [
    "## python `doctest`\n",
    "\n",
    "Documentation driven testing was [introduced into python in 1999][ddt]. It introduced the ability to combine code and narrative into docstrings following a long lineage of literate programming concepts.\n",
    "\n",
    "[ddt]: https://groups.google.com/g/comp.lang.python/c/DfzH5Nrt05E/m/Yyd3s7fPVxwJ"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "874e3649",
   "metadata": {},
   "source": [
    "### anatomy of a `doctest`\n",
    "\n",
    "the goal of a `doctest` is to compare the execution of a `line_of_code` with an `expected_result`, when a `doctest` is executed it executes the `line_of_code` and generates a `test_result`. When the `test_result` and the `expected_result` are the same a test has passed, otherwise a test has failed.\n",
    "\n",
    "Each `doctest` input begins with `>>>` and continues with `...`; an expected output, if it exists, immediately follows a line beginning with either `>>>` or `...`. The list below illustrates the basic forms of `doctest`s in pseudocode representation.\n",
    "\n",
    "* a `doctest` with a single line of code and an expected result\n",
    "\n",
    "    ```pycon\n",
    "    >>> {line_of_code}\n",
    "    {expected_result}\n",
    "    ```\n",
    "\n",
    "* a `doctest` with multiple lines of code and an expected result\n",
    "\n",
    "    ```pycon\n",
    "    >>> {line_of_code}\n",
    "    ... {line_of_code}\n",
    "    ... line_of_code\n",
    "    {expected_result}\n",
    "    ```\n",
    "\n",
    "* a `doctest` with multiple lines of code that prints no output\n",
    "\n",
    "    ```pycon\n",
    "    >>> {line_of_code}\n",
    "    ... {line_of_code}\n",
    "    ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b23e9c03",
   "metadata": {},
   "outputs": [],
   "source": [
    "    import doctest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "646f7cfa",
   "metadata": {},
   "source": [
    "We like `doctest` because it is the easiest way to run tests in notebooks. Below is a concrete example of a `doctest`:\n",
    "\n",
    "* line 4-6 represent the `line_of_code`\n",
    "* line 7 is the `expected_result`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5b5c9a2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "    def a_simple_function_with_a_doctest(x):\n",
    "        \"\"\"this function turns every input into its string representation.\n",
    "\n",
    "        >>> a_simple_function_with_a_doctest(\n",
    "        ...      1\n",
    "        ... )\n",
    "        '1'\n",
    "        \"\"\"\n",
    "        return str(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7886786d",
   "metadata": {},
   "source": [
    "The easy invocation of a `doctest` in the notebook is what makes it the easiest test tool to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8e139fab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TestResults(failed=0, attempted=1)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    doctest.testmod()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e987081",
   "metadata": {},
   "source": [
    "When we invoke `doctest.testmod` it will look for all the functions and classes in our module that have docstrings, and `doctest` examples. As a result, the previous example finds one test. The test results summarize the test success and failures. If the `test_result`, the execution of the `line_of_code`, matches the `expected_result` our tests succeed, otherwise they fail. When test fail, we'll want to inspect both our tests and source code to discover to the source of the failure.\n",
    "\n",
    "\n",
    "> learn more about [`doctest` discovery]\n",
    "\n",
    "[`doctest` discovery]: https://docs.python.org/3/library/doctest.html#which-docstrings-are-examined \"a section about how doctest discovers tests\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8966a92f",
   "metadata": {},
   "source": [
    "Here we add a new class with another `doctest`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9b5d0bb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "    class ASimpleClassWithADoctest(str):\n",
    "        \"\"\"this type turns every input into its string type.\n",
    "\n",
    "        >>> ASimpleClassWithADoctest(1)\n",
    "        '1'\n",
    "        \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfec454f",
   "metadata": {},
   "source": [
    "When we re-run the `doctest`s we notice that another test was discovered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "beb57e41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TestResults(failed=0, attempted=2)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    doctest.testmod()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53ddaa6d",
   "metadata": {},
   "source": [
    "There is one more way that `doctest` discovers tests, which is through the `__test__` variable. We can make a `__test__`  dictionary with keys that name the tests, and the values are objects holding `doctest` syntaxes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "77419445",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a_test_without_a_function': '>>> assert a_simple_function_with_a_doctest(1) == ASimpleClassWithADoctest(1)'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    __test__ = dict(\n",
    "        a_test_without_a_function=\"\"\">>> assert a_simple_function_with_a_doctest(1) == ASimpleClassWithADoctest(1)\"\"\"\n",
    "    ); __test__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4472eab5",
   "metadata": {},
   "source": [
    "Now the `doctest` finds three tests."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a495a616",
   "metadata": {},
   "source": [
    "## `unittest` when `doctest` isn't enough"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "102c86e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "    import unittest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a60a48f",
   "metadata": {},
   "source": [
    "`doctest` are the easiest to invoke, but they can be difficult write for some tests you may wish to run. Python provides the alternative `unittest` library that allows authors to write tests in pure python, rather than strings. This approach to writing tests with be more familiar to new python learners.\n",
    "\n",
    "`doctest` relies on the comparison of an `expected_result` and a `test_result`. meanwhile, `unittest` provides an extended interface for comparing items using their [list of assertion methods].\n",
    "\n",
    "[list of assertion methods]: https://docs.python.org/3/library/unittest.html#assert-methods \"A list of assertion method in unittest test cases.\"\n",
    "\n",
    "> learn more about the relationship between `doctest` and `unittest` [The Hitchhiker's Guide to Python - Testing your code][hitchhiker]\n",
    "\n",
    "[hitchhiker]: https://docs.python-guide.org/writing/tests/ \"blessed documentation comparing unit test approaches\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17d7fc12",
   "metadata": {},
   "source": [
    "python unit tests subclass the `unittest.TestCase` type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f1131e04",
   "metadata": {},
   "outputs": [],
   "source": [
    "    class UnitTests(unittest.TestCase):\n",
    "        def test_simple_methods(self):\n",
    "            pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee70f854",
   "metadata": {},
   "source": [
    "Running `unittest` in the notebook requires some keyword arguments that we'll wrap in a function to facilitate our discussion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "32682478",
   "metadata": {},
   "outputs": [],
   "source": [
    "    def run_unittest():\n",
    "        unittest.main(argv=[\"\"], exit=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03a40098",
   "metadata": {},
   "source": [
    "When we invoke `run_unittest` we notice our test is discovered,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "58f67272",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.001s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "run_unittest()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32e3862",
   "metadata": {},
   "source": [
    "We've already written three other `doctest`s though.  Wouldn't it be nice to include our `doctest`s in the test suite too?! This integration is not immediately apparent until you dig deep into the `doctest` documentation to find the [`unittest` interface]; here, we find a special `load_tests` function. The `load_tests` function is where we can add our `doctest`s to our `unittest`ing suite as illustrated to code block below.\n",
    "\n",
    "[`unittest` interface]: https://docs.python.org/3/library/doctest.html#unittest-api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2fc1bebb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_tests(loader, tests, ignore):\n",
    "    tests.addTests(doctest.DocTestSuite(__import__(__name__)))\n",
    "    return tests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a3e3da5",
   "metadata": {},
   "source": [
    "Now `run_unittest` discovers 4 tests, including our `doctest` defined earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "29fc7761",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "....\n",
      "----------------------------------------------------------------------\n",
      "Ran 4 tests in 0.008s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "run_unittest()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac33d856",
   "metadata": {},
   "source": [
    "It may appear that we are using multiple testing forms by combining `doctest` and `unittest`. It turns out that a `doctest` is a special test case that compares the output of string values. With experience, authors will find that some tests are easier to write as `doctest` and others are easier as `unittest` assertions. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30ac68c5",
   "metadata": {},
   "source": [
    "## restart and run all, or it didn't happen\n",
    "\n",
    "Now you see how to simulate running formal tests against your interactive notebook code. The notebook is a means, not an ends. Now it is time to copy your module code and new tests into your project!\n",
    "\n",
    "For posterity though, it is helpful that a notebook can restart and run all. Better yet, your notebook can restart and run all... with testing in the last cell! And, this specific document abides as the prior code cell is the last cell, and a test! We now have a little more confidence that this notebook could work in the future, or at least verify that it still works."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa8abca5",
   "metadata": {},
   "source": [
    "When you get to the last cell with no errors, it is time to celebrate a notebook well written."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5be80d71",
   "metadata": {},
   "source": [
    "## conclusion\n",
    "\n",
    "* we can simulate unit testing in the notebook.\n",
    "* `doctest`s run tests in strings and docstrings .\n",
    "* `unittest`s run tests on objects and may include `doctest`.\n",
    "\n",
    "Testing is a good practice. It helps formalize the scientific method when code is involved. Being able to simulate tests in the notebook, `doctest` and `unittest` help expedite the test writing process by taking advantage of the rich interactive features of notebooks. When we write tests we record ideas so that our future selves can thank us for our past practice."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}