Pulp Smash Pitfalls

pulp_smash_pitfalls.md

Pulp Smash Pitfalls

1 Attribute dependency & Skipping

Currently we use the pattern :

class TestCase(unittest.TestCase):

    def test_01(self):
        self.attr = 'foo'

    @skip_if(bool, 'attr', True)
    def test_02(self):
        ...

Making tests numbered _02 depending on tests numbered 01 and sequentially. This is a bad pattern for reasons:

• Not easy to select specific tests to run:

py.test test/test_module.py -k test_02_....   # always skipped

• Impossible to use IDE test debuggers

• Not easy to randomize the order of tests to run.

• Not easy to mark tests for categorization

Statement:

Tests would be independent, idempotent, self sufficient. In other words: in a test run we would be able to select 1 or n tests to run in any chosen order, each test would be responsible by a declarative design to setup all the things needed to run. (as in a fixture)

Suggestion:

• Adopt pytest:
  • fixtures solves dependency injection
  • marks solve test selection
  • going classless will allow randomization
  • Decorators can be used to skip tests based on multiple kinds of conditions

  def test_foo(thing):
      # this test will always depends on previously created fixture `thing`.
      # `thing` fixture on conftest.py can be imported from pulp-smash.fixtures
      # `thing` can be scoped to function, module, session
      # `thing` can handle its own teardown/cleanup
      # `thing` can be a list of things to cleanup
      # `thing` if session scoped can be a state-machine or shared resource

2 Config System

Currently the config system that Pulp-Smash uses is overengineered, it does a lot of validation using json-schema with an environment of distributed services that we don't actually use (and for pulp2 we will never use)

When we go to a distributed environment it will now be on kubernetes/openshift and then we will talk only to a single API and Kubernetes will route it internally, we do not need to support multiple api's right now and what we have (api and content_host) is sufficient for current needs.

It will be easier to implement some sort of separation on demand than keep complex structure

"Premature optimization is the root of al evil"

Suggestion: We need a simpler config system and at the same time a config system that translates to declarative pulp services. So only for Pulp3 I suggest.

default:
  hosts:
    api: &api
      hostname: localhost
      se_linux: true
      expose: 80
    content: &content
      hostname: otherhost
      se_linux: false
      expose: 24816
  services:
    - name: pulp-resource-manager
      systemd: true
      host: *api
    - name: pulp-wsgi
      systemd: true
      host: *api
      protocol: https
      port: 24817
      verify: false
    - ... (more services listed)
  api_client:
    timeout: 600
  cli_client:
    ssh_user: root
    ssh_key: ~/.ssh/foobar.pub
  debug_level: DEBUG

It would be now easier to have a function like pulp_smash.utils.restart('pulp-resource-manager', 'pulp-wsgi') and then the function will read all the services list to spawn the tasks.

It also solves the granularity as we now have readable and host-based configurations.

In a test run, we would like to be able to execute tests without a config file, so pulp-smash would assume some defaults (based on latest pulp-release and allow us to specify only what we need like)

# this envvars exported using toml formatting
export PULP_SMASH_HOSTS='{api.hostname="my-own-hostname.com", api.export=8080}'
export PULP_SMASH_API_CLIENT='{timeout=300}'
py.test pulpcore/tests/functional
...

No need to create a config file, if I provide the only required information which is the host, other stuff are assumed by default.

3 Constants and utils system

Right now we have in every repository a constants and a utils file, this forces us to import stuff from that specific plugin namespace like from pulp_file.tests.utils import foobar and it is a bad design.

Our tests would be independent, the plugin itself does not need to be installed on the environment, to run the tests we only need pulp-smash, test runner pytest and the test scripts.

Test scripts are located under the plugin repo for convenience, but they would be independent and in theory could be located anywhere as they would not depend on plugin code.

We need to get rid of utils and constants that stuff would be located only inside the framework pulp_smash.

the test structure would be:

plugin_name/
  plugin_code....
tests/
  conftest.py  # fixtures, setups, teardowns, pulp-smash importing
  test_foo.py
  test_bar.py

1. Our tests would be stored on the root of the repo and not be part of the package itself
2. on tests/ folder we would have only test_* scripts (which are really independent scripts, depending only on pulp_smash) and conftest.py which is the fixture bootstrap for pytest (which will also get fixtures from pulp_smash.fixtures)

That will solve the problems

• of having duplicated constants
• of having inter-plugin dependency
• of having to install the plugin package to run the tests
• We wil be able to get only the tests/ folder when building the test runner

4 Global cleanups

Pulp Smash performs global cleanups (calls to clean-orphans) and that is a bad idea and can lead to race conditions.

We want to be able to run tests in parallel, so each test must be responsible to create and clean only its isolated entities and data.

5 Pulp Smash misses categorization and tagging of test cases

Sometimes for metric gathering or specialized test runs, we need to collect some sort of tests.

Example: As a QE I want to run only CRUD test cases. As a QE I want to run only critical tagged test cases. As a manager I want to query how many issues(bz, redmine, jira) our tests are covering. As a manager I want to query how much of our tests are integration tests?

That are just examples but that would be only possible if we do a proper test tagging.

import pulp_smash as ps

@ps.skip_if_not(ps.bug_is_fixed, 'R#1234')
@ps.skip_if_not(ps.plugin_installed, 'docker')
@ps.skip_if(ps.running_on_travis)
@ps.properties(
    category='crud',
    level='critical',
    type='integration',
    covers=['R#1234', 'G#555', 'J#666']
)
def test_crud_critical_integrated_with_docker(api, cli, urls, simple_docker_repo):  # fixtures provided by `ps`
    """This test covers bla bla bla bla

    The `Arrange` step for this test is provided by the fixtures received above
    The test just need to perform the 'Act' and 'Assert'
    """
    # Act
    result = api.client.get(urls.DOCKER_FOO, params={...})
    other_thing = cli.client.run(['docker', 'subcommand', result.foo])
    # Assert
    assert other_thing in simple_docker_repo.foo.bar.baz, "debugging message here"