name: Why are our test slow!? What can be done? class: middle, center
class: middle, center background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/00857a6752824e1080a62caf6594c071217501ae/4_2_problem.png) background-size: contain
??? We had a problem... well... we had at least 3 problems making our unit tests slow.
With over 1500 tests, it was taken as read that running tests simply take a long time.
Our build servers took so long to run the tests, tests delegated to the ADO so we only had to wait for tests to run at the PR phase of CI.
You could tell developers were opting to not run tests locally because it was too painful. PRs would fail due to test failures regular enough.
As time moved on we slid down the slippery slope until we found ourselves in a position where tests ran too slow for ADO to merge PRs.
Productivity was impacted critically at the begining of April when merging PRs was running up against the 60min threshold that ADO would allow any given task.
- We write slow tests.
- Our build servers were underpowered
- Jasmine/karma get slow as projects get big.
??? The first thing I did was look for outliers. Troublesome tests that took noticeably longer than average.
Fortunately, I found some!
I identified 12 suites, a little under 10% of the tests, took half the time.
ADO went from running in hour (and failing because this was too slow) to about 24 minutes when I skipped those tests.
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/9f440e6bc2d32dada4d1dc07fcb981d7cb4698e1/drItHurts.jpg) background-size: contain
???
This was encouraging as it got the build back up and running, but clearly not a long term solution.
What is different about these tests?
How do we get them running in a more performant way?
A library which enhances your unit testing experience with Angular TestBed, greatly increasing execution speed of your tests.
??? Travis and Craig identified this module that makes setting up TestBeds more efficient. Travis dug in and found this module to be a fairly easy win.
The way we tend to set up TestBeds, Angular will re-compile everything we used to configure our TestBed for every test in a suite, thus increasing the overall test execution time.
Author claims that: With a really fat configurations it may come close to be N times faster, where N is the number of tests in the suite.
class: middle, center
class: middle, center
???
While it is possible and valid that you need dependencies injected for your tests to run, you should take a hard look at what you're injecting into the tests that ng-bullet helps the most.
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/60ed43e17e8116679619a17806e3ab14c85f00ac/beforeNGBullet.png) background-size: contain
??? Here's an example of a suite that's a prime candidate for ng-bullet. Each test is hundreds of ms... INCLUDING SHOULD CREATE! Something fishy is happening in the beforeEach().
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/60ed43e17e8116679619a17806e3ab14c85f00ac/afterNGBullet.png) background-size: contain
??? This is clearly much better as we've shaved almost a half a second off of nearly every test in the suite. There's still plenty of room for improvement.
background-image: url(https://gist.github.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/7c65e3812a3c47c32671a15d8279aab0f7dfc7ba/slow.jpg) background-size: contain
???
"Bullet" may be a bit of an overstatement. With Travis' changes in place the 12 suites were now fast enough that ADO could deploy, but it's not time to celebrate yet.
class: middle, center background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/f821a15afcac473f92056e7752fa6ecb9ef12735/testbed.jpg) background-size: contain
??? TestBed provides an easy way to get all the dependency injection goodness that our app needs. That's arguably outside of what a unit test wants, let alone needs.
Do you need to render your component to test it?
Do you need to inject all of it's dependencies for this spec? These are questions you should be asking when you create a new spec file.
- Render one level deep
- Mock beyond that
??? TestBed makes it fairly easy to inject all the dependencies to wire up a fully functioning component. That doesn't mean we should.
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/00857a6752824e1080a62caf6594c071217501ae/needToSurvive.gif)
import { NO_ERRORS_SCHEMA } from '@angular/core';
...
TestBed.configureTestingModule({
...
schemas: [NO_ERRORS_SCHEMA]
});
??? NO_ERRORS_SCHEMA was not obvious to me when I first encountered it in our code base, does everyone know what that's doing here?
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/00857a6752824e1080a62caf6594c071217501ae/no_errors_schema.png) background-size: contain
??? ಠ_ಠ
In a nutshell, this stops angular from complaining about missing dependencies so you don't have to inject things that would otherwise be needed. You are free to mock any dependencies that you actually need to get tests to pass. This feels kind of icky to me, but seems like a popular approach. In my time with React and Vue, they had first class solutions to shallow renders that provided additional affordances. I'm looking into a couple modules that can make this nicer.
Skips errors from missing declarations.
Note, the Pluralsight video says only do this when necessary as it disables helpful warnings as well.
class: middle center
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/7c65e3812a3c47c32671a15d8279aab0f7dfc7ba/modernProblems.jpg)
??? So, about the time I first saw that successful 24 minute test run on ADO, I started testing the performance of Jest. The same tests running via Jest took about a 1/3 the time to run.
class: middle, center
"Jest is a delightful JavaScript Testing Framework with a focus on simplicity." --Jest's Mother
??? I'm familiar with it from working on React and Vue projects.
Jest is an alternative to Karma and Jasmine that is mostly compatible. When I say mostly compatible, around 1200 of our 1500 tests "just worked" when I swapped out Karma for Jest.
3x speed improvment on inital comparison
Incremental tests can make feedback closer to real time.
jasmine.createSpy('name') --> jest.fn()
and.returnValue() --> mockReturnValue()
spyOn(...).and.callFake(() => {})
--> jest.spyOn(...).mockImplementation(() => {})
Asymmetric matchers: jasmine.any, jasmine.objectContaining, etc. --> expect.any, expect.objectContaining
??? The main differences you'll notice are around mocking. To ease the transition, I added a 3rd party shim that brings jest a createSpyObj helper that works just like Jasmine's helper by the same name.
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/00857a6752824e1080a62caf6594c071217501ae/crystalResults.jpg) background-size: contain
??? Jest is a faster way of running tests, but slow tests will still be slower.
background-image: url(https://gist.githubusercontent.com/RyanParsley/4c8f73073e5db68b73c69686b5053e86/raw/00857a6752824e1080a62caf6594c071217501ae/incremental.png) background-size: contain
??? Things are getting a little better every day.
