browser_testing_question.md

These are questions about browser testing and state.

Imagine a large suite of browser tests that intend to deeply regression test an application. Like most web apps, this involves different types of state: user permissions, expected data in a database, etc. Testing the application requires logging in as different users and/or user types to test different views into the data.

1. What are patterns for creating expected state prior to running browser tests?
2. Is it appropriate for the browser tests to communicate directly with the database of the system under test, creating state in the same manner that you would in an integration test?
3. How important is it that browser tests be decoupled from the application under test, such that the only thing required to run tests are the tests themselves, which can be pointed at any applicable URL (dev, test, staging)?

Hey Marc, good questions. Not sure if you want a response here, but I'm going to do that anyway:

1. What we often do is use an available API to create data for our tests. This is pretty much a necessity when the tests are run on a different machine than the site under test. We would never do this in production, though. Our tests against production never change any data.
2. We never communicate directly with the database from the tests, but communicating with the database via the app (e.g., via the model layer) is an option if the tests are running on the same machine as the site. That's pretty much only the case for us for a developer running the tests on their own machine, so we tend not to follow that model for our functional tests as we want them running against "common" environments, such as staging. I have seen functional tests written by the app developers, which are designed to run on their own machines that do communication with the database via the app though.
3. This is very important to us for a couple of reasons:
4. As you mention above, we do want to be able to run our suites against different URLs (dev, stage, prod).
5. We rely heavily on contributors to help us with our test suites, and most of them would find having to get the full application running locally to be a big barrier to entry. We want a contributor to be able to clone a git repo, run pip install, and start running tests immediately.

Really good set of questions, Marc. It prompted me to write a blogpost on the topic because I've been putting that off for a long time. Thanks for a good prompt to actually do it!

The gist of the blogpost: Use custom APIs, leverage your system's APIs, use baseline datasets, and never, ever share state between two tests!

Just two cents from me:

There are multiple ways to create the state of the app that you wish, if you can have majority of your tests and parts of the stack running locally on your machine, then you can write majority of non full integration tests to chip away at smaller parts of the application. Yes the orders service or the integration with 3rd party tests might not work, but at least you can test your application alone in a hermetic environment, this will provide a really consistent test results. In my opinion, the purer your application is, that is no external input, the more useful the test results are.

As for the direct communication with the DB, there are several options

1. If you are running tests locally, there is no problem with generating and loading fixtures into DB as part of the pre-build process. This way your tests know the exact state of the app every single time

2. On integration environments, i've used a smaller sub-sample of the prod DB (with customer data obfuscated), this way the MySQL dump loaded post each deploy of new build unto integration environment, and can also be parsed by your tests to know the current state of the application

3. In situations where the DB cannot be easily manipulated pre build or post deploy, such as staging, i've used the native mobile app's API to get access to some of the data on the environment, such as product listing etc. This way being most limited and READ ONLY compared to the previous options.

As for the coupling, i don't think i can give a coherent response to that question at the moment.