public
Last active

Unittest2 test discovery and real dotted-path named test selection for Django

  • Download Gist
runner.py
Python
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
"""
An alternative Django ``TEST_RUNNER`` which uses unittest2 test discovery from
a base path specified in settings, rather than requiring all tests to be in
``tests`` module of an app.
 
If you just run ``./manage.py test``, it'll discover and run all tests
underneath the ``TEST_DISCOVERY_ROOT`` setting (a path). If you run
``./manage.py test full.dotted.path.to.test_module``, it'll run the tests in
that module (you can also pass multiple modules).
 
And (new in this updated version), if you give it a single dotted path to a
package, and that package does not itself directly contain any tests, it'll do
test discovery in all sub-modules of that package.
 
This code doesn't modify the default unittest2 test discovery behavior, which
only searches for tests in files named "test*.py".
 
"""
from django.conf import settings
from django.test import TestCase
from django.test.simple import DjangoTestSuiteRunner, reorder_suite
from django.utils.importlib import import_module
from django.utils.unittest.loader import defaultTestLoader
 
 
 
class DiscoveryRunner(DjangoTestSuiteRunner):
"""A test suite runner that uses unittest2 test discovery."""
def build_suite(self, test_labels, extra_tests=None, **kwargs):
suite = None
discovery_root = settings.TEST_DISCOVERY_ROOT
 
if test_labels:
suite = defaultTestLoader.loadTestsFromNames(test_labels)
# if single named module has no tests, do discovery within it
if not suite.countTestCases() and len(test_labels) == 1:
suite = None
discovery_root = import_module(test_labels[0]).__path__[0]
 
if suite is None:
suite = defaultTestLoader.discover(
discovery_root,
top_level_dir=settings.BASE_PATH,
)
 
if extra_tests:
for test in extra_tests:
suite.addTest(test)
 
return reorder_suite(suite, (TestCase,))
settings.py
Python
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
"""
You need the ``BASE_PATH`` and ``TEST_DISCOVERY_ROOT`` settings in order for
this test runner to work.
 
``BASE_PATH`` should be the directory containing your top-level package(s); in
other words, the directory that should be on ``sys.path`` for your code to
import. This is the directory containing ``manage.py`` in the new Django 1.4
project layout.
 
``TEST_DISCOVERY_ROOT`` should be the root directory to discover tests
within. You could make this the same as ``BASE_PATH`` if you want tests to be
discovered anywhere in your project. If you want tests to only be discovered
within, say, a top-level ``tests`` directory, you'd set ``TEST_DISCOVERY_ROOT``
as shown below.
 
And you need to point the ``TEST_RUNNER`` setting to the ``DiscoveryRunner``
class above.
 
"""
import os.path
 
# This is correct for the Django 1.4-style project layout; for the old-style
# project layout with ``settings.py`` and ``manage.py`` in the same directory,
# you'd want to only call ``os.path.dirname`` once.
BASE_PATH = os.path.dirname(os.path.dirname(__file__))
 
# This would be if you put all your tests within a top-level "tests" package.
TEST_DISCOVERY_ROOT = os.path.join(BASE_PATH, "tests")
 
# This assumes you place the above ``DiscoveryRunner`` in ``tests/runner.py``.
TEST_RUNNER = "tests.runner.DiscoveryRunner"

This should probably say that you have to enable it like this in the settings:

# The name of the class to use to run the test suite
TEST_RUNNER = 'dotted.path.to.DiscoveryDjangoTestSuiteRunner'

I need to blog about the version of this Gisle Aisl showed me at PyCon 2011. It's in https://github.com/opencomparison/opencomparison and is just plain AWESOME.

Right, thanks jezdez. Mentioned in the tweet that pointed to this, but it should be mentioned here. You also need to set TEST_DISCOVERY_ROOT to point to the root path for test discovery, when no labels are given on the command line.

@pydanny - you talking about this? https://github.com/opencomparison/opencomparison/blob/master/testrunner.py

It looks like that just defaults to a settings-defined set of project apps instead of all installed apps, yeah?

One issue with this that I ran into is when specifying app_labels. The problem is that defaultTestLoader.loadTestsFromNames expects the given name to be a dotted path to the tests.py files. It loads <app>.__init__.py and then calls dir on that module, looking for subclasses of unittest.TestCase. If you look at Django's build_suite code, it does a bunch more work to build up the test from an app label by also looking in the tests module.

I fixed it by just calling the parent build_suite in the case when there are app labels specified. Here's my gist

@streeter - Indeed. That's not a bug, that's a feature :-) Selecting tests by full dotted path rather than by app label is part of the purpose of this alternative runner. It is more flexible and gives more control. For one thing, it allows keeping tests in any Python package/module, not just in an installed app. And if tests are split into multiple modules within a tests/ package in an app, it allows running just the tests in one module, rather than forcing you to either run all the tests in an app, or just a single TestCase. And, of course, it's the way every other Python test runner works :-)

In a broader sense, I consider app-labels to be a mistake in Django's design in the first place. They take a functional and well-known namespace, the Python module namespace, and attempt to "flatten" it. This just causes needless collisions (inability to have both django.contrib.auth and my.own.auth) and doesn't really offer any benefits.

There is discussion ongoing to perhaps make something along these lines the default test runner in Django 1.4 (with a few more options added to allow selecting tests by app label for those who prefer that). You may want to take a look and weigh in if you're interested: https://code.djangoproject.com/ticket/17365

Ah, interesting. I hadn't thought that was a feature :) However, it definitely makes it more flexible at the expense of a bit of extra typing. Though when I'm typing individual app labels, I'm also using my bash history more often. I'll check out the 1.4 ticket and think about it a bit more.

@carljm - yup. And the delineation betweeen APP types is really clear. It would be nice if incorporating this sort of thing into Django could be done, or if a command-line flag could be set.

@carljm - I'm trying to split out tests into separate files as per your PyCon 2012 talk, but I'm not able to get this working in Django 1.3.1. All of the following command find zero tests: ./manage.py test myapp and ./manage.py test myapp.tests. But ./manage.py test myapp.tests.user_test does find them. ./manage.py test actually brings up are error, it's trying to look in ../tests, outside my project directory.

My tests are in myproject/myapp/tests/user_test.py, and I have the following in my settings.py file:

import os.path
BASE_PATH = os.path.dirname(os.path.dirname(__file__))
TEST_DISCOVERY_ROOT = os.path.join(BASE_PATH, "tests")
TEST_RUNNER = "helpers.DiscoveryRunner"

@chase-seibert Probably your layout is the old-style project layout (with manage.py and settings.py adjacent), in which case there may be one too many os.path.dirname calls in that BASE_PATH setting? I'd just debug what BASE_PATH and TEST_DISCOVERY_ROOT are getting set to, and whether it's what you expect.

The other issue is a known limitation of the simplified runner code; it either does full discovery, or runs exactly the modules you give it; it doesn't do partial discovery within a given module/package. The updated version of this gist I just posted (which was at the /code link in my talk slides, but didn't fit on a slide itself :-) adds in the partial-discovery feature.

This could easily be further improved to be able to do partial discovery within multiple given packages.

Thanks for the help, Carl. I've tried creating a new project under Django 1.4c1. I've uploaded it here: http://dl.dropbox.com/u/422013/testsite.tar.gz

(virtualenv)chase@chase-mint ~/testsite $ ./manage.py test
BASE_PATH:  /home/chase/testsite
TEST_DISCOVERY_ROOT:  /home/chase/testsite/tests
Creating test database for alias 'default'...

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Destroying test database for alias 'default'...
(virtualenv)chase@chase-mint ~/testsite $ ./manage.py test tests
BASE_PATH:  /home/chase/testsite
TEST_DISCOVERY_ROOT:  /home/chase/testsite/tests
Creating test database for alias 'default'...
TestCase1
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
Destroying test database for alias 'default'...
(virtualenv)chase@chase-mint ~/testsite $ ./manage.py test tests.more
BASE_PATH:  /home/chase/testsite
TEST_DISCOVERY_ROOT:  /home/chase/testsite/tests
Creating test database for alias 'default'...
TestCase2
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
Destroying test database for alias 'default'...

I'm expecting the first and second commands to find both tests, the one in test/__init__.py and the one in test/more.py. Do you have a whole project you can share where it's working?

@chase-seibert Ah, I didn't think to document how unittest2 test discovery itself works. By default, only files named test*.py are searched for tests. You can change this by passing the pattern argument to the discover method (it takes a shell glob, the default value is "test*.py"). It would be easy enough to modify the above runner to respect a TEST_DISCOVERY_PATTERN setting.

Firstly, thanks for this. When I'm more hip I might use nose or something but this works perfectly.

Secondly there is a tiny mismatch between runner.py and settings.py. DiscoveryRunner in settings and DiscoveryDjangoTestSuiteRunner in runner.py. Tripped me up for a few seconds.

Thanks @mallison, fixed.

Nose (with django-nose, so you don't lose the Django db setup/teardown stuff) is a great option, too. The main reason for this gist is that it's pre-testing something we're hoping to put in Django core in time for 1.5, and requiring nose isn't really an option for core.

In case anyone is interested, I've packaged this up for general consumption (apologies @carljm, I needed it quick): http://pypi.python.org/pypi/django-discover-runner

I'm stuck on how to use this with Jenkins. If I want my Jenkins server to use this plugin, then I replace the "./manage.py jenkins" command that Jenkins was running with "./manage.py test". But now I'm no longer using the django-jenkins plugin command, I don't get the XML output that I think Jenkins needs in order to display the test results. Am I being dense?

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.