Describes the different levels of testing, tools and techniques available on z-frontend for Web Apps. The RFC touches on the levels of testing, how they interact, which tools we use for each, how do we make them scale, when do we run them (locally and different CI/CD) stages. Finally, we also propose changes to our deployment process.
There is currently a big Test Gap today. This leads to lack of confidence on doing changes, slower development pace, longer deployment cadences and customer errors. This is an issue for Product Teams as well as Platform Changes. When we do a change, we want to be confident. Manual verification still important by Engineers and QE, but we need tools to help us minimize this.
There are great tools on the React ecosystem, there are so many tools that it's sometimes hard to choose the right set. Even when implemented, we need to make sure we establish patterns and additional support to use them in a scalable and consistent way.
Finally, we have a lot of problems with our Ember testing ecosystem, we need to consider those issues to make sure we don't repeat the same mistakes.
This RFC focuses mostly on z-frontend's React Apps, but there is a small section to cover Other Components With Testing Gaps that briefly touches on React Native and GraphQL (yp3/graphql and zgraphql).
For the purposes of frontend testing we have defined the following levels. These might not be the same as what you have used in the past, but we will use the following terminology at Zenefits.
- Static Analysis and Types: basic level of coverage. We use eslint, tslint, prettier and TypeScript. This is working well, documented on our Testing Guide and outside of the scope of this RFC.
- Unit Tests: test a single unit, normally a function, class or module in isolation. We use jest for this, it works well, it's documented on our Testing Guide, but we need to write more tests at this level.
- Integration Tests: test more than one unit at a time
- Component Test: this is a special type of testing level between unit and integration. We use react-testing-library, Cosmos and it's documented on our Testing Guide. These aren't widely used yet.
- Acceptance Tests: they test from the point of view of the end user, but without hitting the backend. Currently this level isn't documented, we have a WIP PR For this type of test we use Cypress which allows us to test the app and all its dependencies, while still mocking the backend. TODO: extract setup and add a few tests for existing apps
- E2E Acceptance Tests: this covers all layers, starting with the UI from the perspective of the end user just like acceptance tests, but they hit backend services and the database. Currently this isn't documented, builds on Acceptance Tests and requires more setup to work on CI.
- Deployment/activation verifications: not really your traditional test, but we will add more checks during this step. For example, to make sure the schema of an app (consumer) is compatible with the server's schema (provider).
We have separate sections for E2E and Acceptance Tests to expand on the Detailed Design.
- node.js. Tend to be faster. We use jest for Unit, Integration and Component Tests and it runs tests on node.js
- Browser tests. Tend to be slower, but reflect the actual environment where our code will be running. We use Cypress for Acceptance and E2E which uses Chrome and variants like Chromium or Chrome Canary. In the future they plan to add support for Firefox, Safari and Edge.
- Line by line, we're not doing it today. Jest (unit, integration and component) supports this. Cypress doesn't.
- Component test coverage can be obtained by running
yarn zcli componentStatus. We could extend it for apps, that could be a good indication to help us track how we're closing our gap.
TODO: complete this section. Cypress allows use to... blablabla, it's great. They suggest not to hit your backend, but YOLO, etc, etc, still doable and this is how.
When testing a new page or feature. I suggest starting with a really broad App Acceptance Test with a
coarse assertion and/or a Visual Snapshot initially set to the Mock. You now have your first failing
test. Time to start implementing a page. You add a route and a new Component for it. At this point, it's
time to go down to the next level. Write a Cosmos Component Test, use a Fixture to abstract routing, give the component the data it needs. As you work on the Component, you may find areas that are better suited
for a Unit Test, that's the point to move again to another level, leave your failing Component Test and
focus on the Unit Test. Implement that and get it to pass and move up to your failing test. Continue
iterating between Component and Unit until your Component Test is working. Add more Component Tests and at
the end check your Acceptance Test, if things are failing, it's a good opportunity to fix Component Tests.
We don't need to the Visual Snapshot to match the mock pixel by pixel, it is just a guide once we visually
inspect it, we can update our Mock with the latest snapshot.
Finally, add the seeder('seederName') decorator to the first test you wrote with the coarse assertion.
This will run the Django seeder and run the test both with Fixtures and as an E2E test.
TODO: I need to run through this process a few more times to verify all the tools.
Both Cypress and Cosmos rely on fixtures. Cosmos use them to mock a lot of different "proxies", one of
which is Apollo (GraphQL). Cypress has fixtures for Network Requests. In both cases, the fixtures are for
one particular scenario. For example, if I want to test the "Blank State" I would start with a fixture with
goals: []. For each test context, our fixtures will be slightly different.
This approach scales better than having a single Mock Server since it's harder to have a Server support all scenarios without becoming as complex as the real server. This happened with Talent.
To see this fixtures for existing pages, we have started by copying valid responses. However, it will be more scalable to rely on faker, factory and our own helpers to make it easier to create data fixtures and only vary what is relevant for each specific Test Case.
TODO: this section needs more feedback and experimentation
We are able to run all of our tests locally and run them smartly based on changes or in some cases explicitly running a specific command.
- vs-code is configured to run eslint/tslint and do TypeChecks as you type. Prettier will run and reformat your code on save. You can run
yarn eslintoryarn prettifyto run them explicitly. jest --watch --notifywill run all your Unit, Integration and Component Tests. The command works at the project level (e.g.apps/benconnect) or at the root. Edit your code or tests and jest will re-run only what changed.zcli cosmoswill launch the cosmos workspace. This is useful to Visually inspect, debug or test a component using a specific fixture.yarn cypresswill open Cypress' test selector and run your Acceptance Tests. We'll look into integrating this andjest. This will also monitor changes to your test and implementation code and re-run the selected tests. These tests are slower, so we need to select a specific test or suite to run.yarn cypress:e2esame as above, but scoped to the e2e tests.yarn verify-schemawill make sure your Queries are compatible with your current yp3 branch. TODO: consider unifying this command with others likeyarn startoryarn test
z-frontend's CI will run all the tests specified on the previous section (Locally)- yp3's CI will trigger Cypress's e2e test when running a Full Suite or an App Suite that includes a
reference to
extensions.tests.Cypress.<app-name>and make sure that themasterand the currently active version of the app work with that version of the backend. zgraphql's CI will trigger Cypress's e2e tests using the active version of yp's schema, and all the app's onmasterand their currently active versions.
In order to run e2e tests inside of z-frontend we have multiple alternatives. This requires further exploration and this is detailed as an Unresolved Question. Short-term we can ignore it and rely on yp3 running z-frontend which is an easier problem to solve.
In order to run e2e tests inside of yp3, we will write a new Python test similar to what we did for
extensions.tests.Graphql.testGraphql and for Ember tests. This requires changes in the following areas:
- During z-frontend's CI, we'll publish "test" assets to an s3 bucket.
- The python test will query the
activeversion's index.html, extract the SHA and use it to pull the test assets - Start
yarn testornpx cypresson the test assets folder - For the version on master, we need to be able to query asset information. The deployment dashboard could help with this, alternatively we could query GitHub for the latest commit in master with a successful build. Once we know the SHA we repeat step 2 and 3 with this version.
- We need to have a running version of yp3. Probably using "yp3 CI" image or just using Glue's JS as a proxy to Django's API.
TODO: how to run app's e2e integration tests for zgraphql and yp3/graphql (see Unresolved Question)
- yp3's deployment will re-trigger
extensions.tests.Cypress.<app-name>as a smoke/verification test - This will trigger schema verification to make sure that all active apps and apps in master are compatible
with the version of the schema (yp) that we're about to "deploy". In order to do this:
- We need each app to publish a subset of their schema (consumer schema). We do this by finding all the queries the app uses (structure) and merging that with the schema.json information they have (types)
- We pull all of the schemas from s3
- Run the schema verification script for each app. This will require us to have
nodeandnode_modulesavailable during this step. Alternatively, we could do this on the GraphQL server (Duplo)
NOTE: during deployment we only run E2E tests since all the other tests only verified the static assets generated during CI. Testing them again would be redundant.
- GraphQL (zgraphql) activation on Beta and Production will re-trigger Cypress' e2e tests
- App activations, will re-trigger e2e tests
- This will trigger schema verification to make sure the subset of an app's schema is compatible with the currently active schema of the backend on the environment we're activating (production/beta)
NOTE: during activation we only run E2E tests since all the other tests only verified the static assets generated during CI. Testing them again would be redundant.
- On a defined schedule we will continue to run Cypress' e2e tests
NOTE: the deployment verification checks depend on the Deployment Dashboard.
Up to now, z-frontend Web Apps have been deployed on demand. This is great and has given teams flexibility and
ownership about their releases. However, it has also caused us to work on large batch sizes, different releases cadences
between Apps and yp3, confusion about what is in production and master often being on an un-releasable state.
This in combination with poor Test Coverage, reduces the confidence from Engineers when doing changes.
Once we have better testing and deployment verification checks, we suggest activating apps at least at the same
cadence as yp3. Teams would still have the ability to release more frequently, for hotfixes, beta testing or
other purposes. Apps in maintenance mode or with known issues could opt-out of the default "activation" cadence.
This change will force the Product Teams and the UI Platform team to maintain master in a releasable state, with high level of quality.
This RFC specifies only the implementation order, the timeline depends on other priorities and will be defined outside of this RFC
- Promote all the types of tests that we can do already (Unit, Integration, Component, Static)
-
- Address concerns with fixtures for Cosmos and Cypress.
- Add more meaningful examples of Cosmos tests
- Add support for Acceptance Tests using Cypress
- Document and promote Outside in TDD Workflow
- Add E2E support locally
-
- Add E2E support during yp3's CI
- Add GraphQL E2E support
- Add schema verification checks during yp3's deployment
-
- Add E2E support during z-frontend's CI
- Add schema verification checks during z-frontend apps activations
- For GraphQL we use jest to test resolvers and verify the schema. This is run during CI by Gondor
using
extensions.tests.Graphql.testGraphql - e2e GraphQL support. This allows us to test Queries and Mutation without needing a UI.
- This works locally today.
- To run them in Gondor we need to find a way to start Django without blocking the thread. (see Unresolved Questions)
- Schema verification check: during CI, we pull the latest active schema and make sure it works
- Activation, Duplo runs the health check and halts activation of other containers if the
/healthendpoint returns anything different than HTTP 200. We do a basic HTTP verification
- Unit and Integration level work the same as described on this document.
- Acceptance, E2E and Component Integration Tests need to be solved shortly after the first pilot is released.
What names and terminology work best for these concepts and why?
Test environments and levels will be documented on our Testing Guide along with simple examples and pointers to actual examples. We'll complement this with presentations and recorded videos of the Testing Workflow which will cement some of this concepts. For further details we'll defer to official docs.
How is this idea best presented? As a continuation of existing patterns, or as a wholly new one?
Both. For some engineers, drawing correlations with Ember might be relevant, so we have a mapping without restricting ourselves to Ember concepts.
Would the acceptance of this proposal mean the UIE documentation must be re-organized or altered? Does it change how we teach to new users at any level?
Yes. Mostly our Testing Guide, but potentially parts of our existing training videos might not be as relevant.
How should this feature be introduced and taught to current developers?
During on-boarding for React projects. We'll work with current teams to make sure they know them and can use this knowledge to close the current testing gaps, which is the main goal of all this work.
Why should we not do this?
We would only ignore our current Testing gaps if we wanted all products built on top of the new UI Platform to be thrown away in 12 months. The cost of doing it is high, but the cost of not doing it, long term will be even higher and we can't afford not writing tests. However, we can focus on key areas, close those gaps first and continue working on others.
consider integration of this feature with other existing and planned features,
We'll have to replace some of our existing tests that rely on enzyme. This can be a gradual process.
There are tradeoffs to choosing any path, please attempt to identify them here.
More on alternatives.
- Instead of Cypress we could use Puppeteer. That's a simpler solution and we already have a PR open for it. Cypress builds on top of it and provides additional functionality. If we needed some of the lower level functionality of Puppeteer like performance profiling, at that point we could look into adding this support to Cypress or just writing newer tests that rely on Puppeteer directly.
- Instead of Cypress we could use Functionize or Behave. While both are valid solutions for E2E, they don't solve the Acceptance Test Level. We could use it for E2E, but it seems unnecessary having a different tool when Engineers will be writing Acceptance Tests and the E2Es they would be writing could simply be an extension of their Acceptance Test.
- Instead of react-testing-library we could continue to use enzyme. Enzyme had problems when using it for Component Tests with Cosmos. Enzyme also has 3 different APIs which can lead to a lot of confusion, out of the 3, one is useless (shallow) and the other two have minor trade-offs but picking one adds cognitive overhead. Additionally, react-testing-library's principle of "...resembling the way your software is used..." (more) is a good way to test and provides reasonable escape valves for exceptions.
- Instead of using Cosmos we could wrap providers manually. We had that option in the past, but the complexity led to poor test coverage. We wrote some abstractions to make this easier, we could continue down that path, but we would likely end up with a shittier version of Cosmos.
- Instead of Fixtures we could continue using our GraphQL Mocks. While GraphQL Mocks served well to get started in Talent, they didn't scale even when the application was still relatively small and simpler. One of the problems was state manager and supporting multiple use cases. Simplifying that would help make mocks maintainable, but would still make it harder to use them to cover all edge cases, which again would increase the complexity of the Mock Server. Instead of having a single point where we maintain Mocks, having Fixtures gives us more flexibility. We also avoid testing beyond a screen or flow, which reduces the need to maintain state on a Mock server.
- Instead of jest we could use mocha or another test runner. Jest was a really early decision and it seems to be working well and there seems to be no need to replace it. We also have other tools like jest snapshots or support for jest workspaces that other libraries likely won't have. A possible argument would lbe to unify with Cypress, which uses mocha or to have the ability to run unit and other tests in a browser context.
- We could move to Jenkins instead of waiting for Gondor. Leaving this as an alternative since it's a viable option depending on our timelines and progress.
- The fixtures section needs feedback adn further experimentation
- Would Functionize or Behave be a better fit?
- The e2e CI setup needs further discussion with Glynn and the Infra team
- Setup from yp3's CI
- How to do e2e inside of z-frontend:
- We could try using inside of the new image is "yp3 CI" docker image (8GBs) inside of Travis.
- We could use service that starts that, so we can run it inside our ECS and Travis talk to that host We're planning to do this for Functionize and Behave. Even when we have it, we need to make sure Travis can talk to it. This takes about 3 to 4 minutes (faster than spoofs - 1:45 to start the container, 1-2 mins for Django to be operational)
- Use beta or a "sticky" spoof. We need to make sure Travis can talk to that environment
- Move away from Travis and use Gondor. We want to eventually do it anyway, this may just accelerate it.
- In all cases we need the ability to un seeders remotely
- e2e GraphQL tests. How to do this? Should we use Glue's JS as a proxy? Should we use the new "yp3 CI" image?
- How to run app's e2e or yp3/graphql e2e tests during zgraphql's CI
- Instead of Cypress use selenium webdriver, and possibly with BrowserStack or SauceLabs. This would give us cross-browser support now. But with the pains of selenium.