This document is the starting point for a Recipes test plan. The goal of this document is that we can have a brainstorm together and fill in the plan with input from the whole team. @rynowak will hold the pen for writing the final document, and @kachalwa is the approver. The work that's detailed here will feed into the individual items being driven by feature owners (functional tests are part of the definition of done).
Our goal for this discussion is to introduce the idea of a test plan, collect, and refine input.
We're doing this now because we're executing on Terraform recipes in a highly parallel well. Getting a shared understanding of the test strategy will help individuals build the right tests by doing some big-picture-thinking upfront.
We've also had a lot of churn in our design practices for tests and recently done some thinking about concepts like:
- Open-box vs Closed-box analysis
- The testing pyramid
- Defining equivalence classes
This will be an opportunity to put these ideas into practice.
In Scope: We're writing this test plan because we're working on Terraform, but you may notice the title says "Recipes" not "Terraform Recipes". Bicep or other kinds of Recipes are in scope, we'll create issues for any gaps we discover and prioritize them alongside all other work.
Out of Scope: We're not going to spend a lot of time talking about unit tests here. We have established practices for unit-tests and generally we follow them. Our overall code-coverage of Radius is very high. A test plan can usually include unit tests, but it doesn't seem the best use of our time.
Out of Scope: Non-functional testing - reliability, resource consumption, security, performance, etc. We're already doing long-haul testing that covers resource consumption and reliability based on our functional tests. We're also adding metrics to Recipes that will cover some of the performance basics. This means that we're getting a lot of non-functional testing for "free" every time we add a functional test. For now we're not going to design separate non-functional tests, we'll rely on the functional tests we're writing.
OK time to do some brainstorming
- I'm leaning on the people who know Recipes for the analysis
- I'm leaning on the people who don't know Recipes to ask good questions
Think about how recipes work. Make a mental list of the tasks that our code performs when executing a Recipe.
- Publishing recipes (Bicep only)
- Registering recipes as part of the environment
- Retrieving the set of declared parameters on the template (
rad recipe show)
PUT operation:
- Retrieving recipe metadata and configuration from the environment (in the engine)
- Metadata: which template, which driver, and parameters
- Configuration: cloud provider configuration
- Invoking the driver: (engine)
- Installing Terraform (in the driver, specific to Terraform, handled by hc-install)
- Downloading the recipe (in the driver)
- This supports multiple sources
- For Bicep we support any OCI registry, but we only test ACR (we use the ORAS library)
- For Terraform we support any Teraform source, but we only test HTTP archive format (tf get, handled by tf-exec)
- This supports multiple sources
- Downloading providers (in the driver, specific to Terraform, handled by tf-exec)
- Generating inputs to the template (in the driver)
- Merging parameters
- Build the context parameter
- Configure cloud providers with scope and credentials
- Configure state storage (in the driver, specific to Terraform)
- Executing the template (in the driver)
- Bicep: PUT on a deployment resource
- Terraform: Calling tf apply (handled by tf-exec)
- Processing template outputs (in the driver)
- Values
- Secrets
- Resource IDs (implicit resources vs explicit resource)
- Connections to resources (implementation in container RP, tested in our E2E tests)
- Bicep: reading results of deployment
- Terraform: using tf show to output the state (handled by tf-exec)
Delete operation:
- Delete controller calls the delete function on the engine
- Retrieving recipe metadata and configuration from the environment (in the engine, same as PUT)
- Delete function of driver (in the driver)
- Bicep: delete resources using resources (loop over output resources)
- Complex and underverified
- Terraform: using tf destroy (handled by tf-exec)
- Bicep: delete resources using resources (loop over output resources)
The Recipe implementation uses as "ports and adapters" design. There are multiple kinds of recipes (drivers). eg: Bicep, Terraform, Helm, etc. Which tasks are general and which are specific to each driver? We're using external dependencies like Bicep and Terraform, what are we responsible for and not responsible for?
Hint: we're always responsible for error handling
Cloud providers also use a "ports and adapters" design. What tasks do we have to perform for each cloud provider? What are we responsible for and not responsible for?
- Configuring the scope and credentials as part of the deployment (either Bicep or tf apply)
- Some error handling for misconfiguration?
- Missing cloud provider configuration?
- What if UCP is behaving badly?
- Deletion
- For Bicep deletion we all the cloud APIs directly, which means we need to handle errors and potentially retries
A few good equivalence classes emerge:
- Tasks that are part of the engine, and apply to all resources and drivers
- Tasks that are part of the driver, but are consistent for all resources
- Tasks that are required for each cloud provider (or Kubernetes) in use, usually driver specific, but are consistent for all resources
Reminder: it's not our goal to test all combinations of these classes. It's our goal to be efficient and get a lot of value from a small number of tests.
Let's divide up the world. Previously we aligned on having at least one E2E (user-scenario) test for each resource type. The E2E tests should reflect each distinct user scenario.
- E2E test for 'recipe' scenario
- E2E test for 'manual' scenario
- API validation is sufficiently covered by unit tests:
- 'recipe' scenario
- 'manual' scenario
- External failure cases
- Authoring mistakes
- Misconfiguration