Skip to content

Instantly share code, notes, and snippets.

@bvaughn
Last active January 9, 2023 15:46
Show Gist options
  • Save bvaughn/d3c8b8842faf2ac2439bb11773a19cec to your computer and use it in GitHub Desktop.
Save bvaughn/d3c8b8842faf2ac2439bb11773a19cec to your computer and use it in GitHub Desktop.
Experimental React DOM test selector API

RFC Test Selectors

Owner: Brian Vaughn


As of facebook/react/pull/22760, the experimental Test Selector API is now available in the experimental release channel.

To test the API, first install the experimental release:

npm install react@experimental react-dom@experimental scheduler@experimental

Then alias the React DOM import. For example, in Webpack this might look like:

module.exports = {
  // ...
  resolve: {
    // ...
    alias: {
      // ...
      'react-dom$': 'react-dom/testing',
    },
    // ...
  },
  // ...
};

Open questions

  • Does this API satisfy the needs of Jest e2e for React DOM? What about React Native?
  • Will this API work for secondary renderers (e.g. ART, Three)?

Historical background

Jest e2e infra enables developers to write tests that interact with and assert things about current state of the DOM. It supports a variety of "selectors" (similar to CSS selectors) for specifying an element on the page, which you can then assert things about (e.g. is it visible?) or perform actions on (e.g. click it).

Selectors can reference a variety of things including tag name (e.g. div), CSS class names (e.g. .foo), id (e.g. #foo), arbitrary DOM attributes (e.g. [role="button"]), or React component name (e.g. AdsLoginButton). They can also refine based on text content (e.g. :text("exact match") or :contains("fuzzy match")).

Although the existing APIs provide a lot of power, they have a few downsides:

  • They rely on React internals to implement the React component name matching.
  • They currently only support DOM.
  • They do not do enough to discourage brittle test selectors from being written (e.g. "ProfilerHeader NavLink label a").

Motivation

A first-party React selector API could encourage infra like Jest e2e to no longer be renderer-specific, (although this might require imposing more constraints on what the selector syntax supported). It could also make open source libraries like [react-testing-library](https://github.com/testing-library/react-testing-library) more powerful.

Specifically, the API proposed below has the following goals:

  • Remove dependency on React internals that may change in future releases.
  • Support all React renderers.
    • The examples in this doc are all DOM, but this API should support other renderers (e.g. React Native).
    • For example, this API would unblock Jest e2e supporting full React Native e2e tests.
  • Enable useful tests to be written with minimal knowledge of component implementation details.
    • Components opt-in by adding a data-testname attribute to host elements they wish to expose for testing. This enables significant refactoring flexibility (e.g. adding/moving/removing rendered elements) without breaking tests.
  • Reduce the amount of test-specific annotations required in product code.
    • The only host elements requiring data-testname attributes are ones that tests interact with directly.
    • In many cases (e.g. clicking or verifying visibility) data-testname attributes aren’t even required.
  • Support HOCs and other behavioral components without requiring pass-thru test id props.
    • e.g. a behavioral component like LoginButton can be combined with a generic data-testname attribute like “button” to select a specific host element, even if that element was rendered by a generic/shared React component like Button.
  • Enable “sparse” selectors to be written without requiring the full path.
    • e.g., identifying the login button in the page header is often possible without specifying the full path to that element. This allows intermediate components to be added/moved/removed without breaking tests.
  • Avoid the constraint of globally unique ids.
    • Although data-testname attributes could be unique, this design does not require uniqueness.
  • Enable additional behavior (or constraints) to be implemented on top of React selectors.
    • e.g. pseudo-selectors like :contains and :text can be built on top of this API ¹
    • e.g. syntax sugar ("Navigation Link#link:text("Contact")") could be supported by this API ¹

¹ Examples below show how this could be implemented.

Constraints

  • Supporting all renderers will require a more constrained API than if we targeted only one renderer (e.g. DOM).

Implementation

The APIs below would likely be implemented as (optional) methods on the HostConfig which get eliminated in non-test builds. The DOM HostConfig could use general browser APIs like getClientBoundingRect and IntersectionObserver to implement most of the behavior described.

If browser APIs prove not to be precise enough, we could detect that we’re in a JEST E2E environment and call a global object passing a DOM node expecting a response, e.g.

if (typeof __JEST_E2E_OBSERVE_RECT__ === "function") {
  __JEST_E2E_OBSERVE_RECT__(node, ...);
}

This would not be considered public API but rather a contract between React and Jest E2E.

I propose we add the following methods to a special testing build of renderer . The names proposed in this RFC are not final, although the expected behavior is.

Creating selectors

The core API methods below work with an array of “selectors”. A selector can be any of the following:

  • Component type (React$AbstractComponent<empty, mixed>)
  • Role (string, e.g. “header” would match an <h1> element)
  • Text contents (string)
  • Test ID (data-testname attribute)
  • Pseudo selector (array of selectors used to identify an element without actually selecting its subtree)

The methods below are used to create selectors which can then be passed to other methods (like findAllNodes()) to specify which element(s) you want to operate on.Component

createComponentSelector()

Matches host elements rendered by a specific React component type.

createComponentSelector(ExampleComponent);

createRoleSelector()

Matches host elements with a specific accessibility role. Matches can be explicit (e.g. role="button") or implicit (e.g. <button>).

createRoleSelector("button");

createTextSelector()

Matches host elements containing the specified text string.

createTextSelector("Exaple text");

createTestNameSelector()

Matches host elements with a specific test name data attribute (e.g. data-testname="LoginFormButton").

createTestNameSelector('LoginFormButton')

createHasPseudoClassSelector()

Inspired by CSS pseudo-classes, this selector matches host elements with subtrees that match the sub-selector array.

For example, given the following app:

<div>
  <article>
    <h1>Should match</h1>
    <p>
      <button>Like</button> <!-- We want to select this one -->
    </p>
  </article>
  <article>
    <h1>Should not match</h1>
    <p>
      <button>Like</button>
    </p>
  </article>
</div>

Then we could use createHasPseudoClassSelector to select the "like" button inside of the first list item:

[
  createRoleSelector('article'),
  createHasPseudoClassSelector([
    createRoleSelector('heading'),
    createTextSelector('Should match'),
  ]),
  createRoleSelector('button'),
]

Primary Core API

findAllNodes()

Finds all host elements (e.g. HTMLElement) within a host subtree that match the specified selector criteria.

function findAllNodes(
  hostRoot: Instance,
  selectors: Array<Selector>,
): Array<Instance>

How does it work?

  • This method starts searching at the specified root, which must be either:
    • A host parent node above the React container (e.g. document.body) in which case, React will traverse the native tree until it finds the first React container, or...
    • A React container, or...
    • A React-rendered host instance with a data-testname attribute (e.g. a match from an earlier findAllNodes query).
  • Starting from the root (or React container) React traverses the internal Fiber tree ¹ looking for all paths matching the specified selector.
    • While traversing the tree, each node will be compared to the current selector index.
      • If the current selector doesn’t match, the tree traversal continues.
      • If the current selector matches, the selector index is advanced.
    • Once all selector criteria have been satisfied, the remaining host elements will be returned.
  • (If the selector array is empty, the root host instance’s fiber will be used.)

¹ Traversing the Fiber tree (instead of the host tree) provides several benefits:

  • Easier to implement and share cross-renderer logic.
  • Portals (e.g. tooltips) can be matched even if they are in a different host tree.

What does it return?

  • An array of host elements that match the specified criteria. (This array will be empty if no matches are found.)

getFindAllNodesFailureDescription() ¹

Prints the names of components within a matched subtree that directly renders host elements.

function getFindAllNodesFailureDescription(
  hostRoot: Instance,
  selectors: Array<Selector>,
): string | null

¹ I believe this API would help with adoption, but it is a non-essential part of the RFC and could be dropped. I had initially planned for findAllNodes to throw an error with this description when no matches were found, but there are cases where no matches is expected (e.g. expect('...').not.toBePresent()). An e2e framework could try/catch these cases to suppress the error from end users, but this would also complicate chained selector implementation (see example below).

How does it work?

  • This methods processes selectors in the same way as findAllNodes.
  • If no matching host elements can be found, it returns a string describing why.

What does it return?

  • If the method was unable to match all of the specified selectors, it returns a string specifying which selectors were matched and which selectors were not matched.
  • If the selector finds at least one matching host element, this method returns null.

Secondary API

The findAllNodes API is powerful because it gives you full access to the DOM node. However, it also requires low-level interaction with that DOM node. Certain higher level interactions are broadly useful.

For example: how much space does a component take up in its parent container? The reason this can be thought of as part of the public API is because you can measure it using a wrapper element’s bounding box. (In other words, it can be measured without compromising the component’s technical implementation details.) Furthermore if a component’s size changes, it’s likely part of an intentional visual change.

findBoundingRects()

For all React components within a host subtree that match the specified selector criteria, return a set of bounding boxes that covers the bounds of the nearest (shallowed) Host Instances within those trees.

function findBoundingRects(
  hostRoot: Instance,
  selectors: Array<Selector>,
): Array<Rect>

How does it work?

  • This methods processes selectors in the same way as findAllNodes.
  • Within each matching path:
    • Find the first Host Instances in the matched component. (Might be several for Fragments.)
    • For each Host Instance, call a getBoundingClientRect Host Config method. Add the result to a list of matched boxes.
    • Optionally merge adjacent boxes and exclude boxes that are fully covered by other boxes. (This ensure less reliance on implementation details like how many nodes make up the tree.)
  • Return the list of matches bounds.

What does it return?

  • An array of bounding boxes relative to the viewport. E.g. { x: number, y: number, width: number, height: number } (This array will be empty if no matches are found.)

What can this API be used for?

  • Clicking or hovering over the center of a Component.
  • Take a snapshot of the Component for screen shot tests.
  • Assert on the maximum size a component is expected to take up.
  • Scrolling the Component into view.
    • This might only be enough for scrolling the outer most scroll. It’s not enough for scrolling inner components. However, inner component’s scrolling may be implemented with other techniques than native scrolling so exposing a method for this would leak implementation details. For this use case data-testname on the scrollable container might be appropriate.

observeVisibleRects()

For all React components within a host subtree that match the specified selector criteria, observe if it’s bounding rect is visible in the viewport and is not occluded.

function observeVisibleRects(
  hostRoot: Instance,
  selectors: Array<Selector>,
  callback: (intersectingRects: Array<{ ratio: number, rect: Rect }>) => void,
  options: IntersectionObserverOptions,
): { disconnect(): void }

How does it work?

  • Create a new IntersectionObserver or equivalent HostConfig specific API, passing a callback and options argument to it.
    • Note: Host Config implementation needs to convert each entry into an object containing only { ratio, rect }, otherwise this callback works the same as the DOM version.
  • This methods processes selectors in the same way as findAllNodes.
  • Within each matching path:
    • Find the first Host Instances in the matched component. (Might be several for Fragments.)
    • For each Host Instance, call the observe(intersectionObserver, instance) Host Config method to add it to the list to be observed.
  • Attach an internal (for test builds only) callback to the React reconciler that is called every time React commits a new update.
    • When this callback is invoked, disconnect the Observer and start over from the top to attach a new one. This makes this observation live updating. This prevents accidentally relying on implementation details such as if a DOM node is reused or remounted.
      • Optionally observe/disconnect only the added/removed Host Instances.

What does it return?

  • An object containing a disconnect() method. When this method is called, we disconnect the IntersectionObserver and disconnect the callback in the React reconciler.

What can this API be used for?

  • Waiting for a Component to become visible before doing further operations.
  • Asserting that a Component is already visible.

Why is this API necessary?

  • Ideally findBoundingRects would be enough for this but these APIs don’t give access to where in the DOM this component is, nor really all the other things that might obscuring the bounding rect. This lets us solve this use case without exposing more implementation details and we can build in details such as if it’s obscured by something virtual (e.g. if an ART component is covered by another ART component within a canvas).

focusWithin()

For all React components within a host subtree that match the specified selector criteria, set focus within the first focusable Host Instance (as if you started before this component in the tree and moved focus forwards one step).

function focusWithin(
  hostRoot: Instance,
  selectors: Array<Selector>,
): boolean

The internal structure of a node is an implementation detail. However, you can start from the outside of a component and move forward (e.g. hit tab) to focus within a component. This has behavior that is defined, and so it can be thought of as part of the public API fo the component.

With the Focus Selectors API used for a11y we might be able to even expose more specific capabilities for the outside of a component move focus into a child.

How does it work?

  • This methods processes selectors in the same way as findAllNodes.
  • Within each matching path:
    • Find the first Host Instances in the matched component. (Might be several for Fragments.)
    • For each Host Instance:
      • If this Host Instance is considered Focusable by calling the Host Config isFocusable(instance), call the Host Config focus(instance).
      • Return true once we’ve found a focusable host instance.
  • If we didn’t find a focusable host instance, return false.

What does it return?

  • A boolean indicating if something focusable was found and focus was set on it.

What can this API be used for?

  • Setting focus on the canonical part of a Component such as text input.

Usage examples

findAllNodes()

Here is a small app with a few components. We’ll use this app for the examples below.

// App.react.js
export default function App() {
  return (
    <main data-testname="main" `role``=``"main"`>
      <Header />
    </main>
  );
}

// Header.react.js
export default function Header({ title }) {
  return (
    <Fragment>
      <PageTitle title="Example" />
      <Navigation />
    </Fragment>
  );
}

// PageTitle.react.js
export default function PageTitle({ title }) {
  return <title>{title}</title>;
}

// Navigation.react.js
export default function Navigation() {
  return (
    <nav role="navigation" aria-label="Main">
      <SearchInput />
      <ul data-testname="list">
        <li><Link label="Home" /></li>
        <li><Link label="About" /></li>
        <li><Link label="Contact" /></li>
      </ul>
    </nav>
  );
}

// SearchInput.react.js
export default function SearchInput() {
  return <input data-testname="search" />;
}

// Link.react.js
export default function Link({ label, ...rest }) {
  return (
    <a data-testname="link" {...rest}>
      {label}
    </a>
  );
}

The app above would render HTML like this:

<main>                                               <!-- App -->
  <title>Example</title>                             <!-- PageTitle -->
  <nav `role``=``"navigation"`>                            <!-- Navigation -->
    <input data-testname="search" />                <!-- SearchInput -->
    <ul>                                             <!-- Navigation -->
      <li>                                           <!-- Navigation -->
        <a data-testname="link">Home</a>            <!-- Link -->
      </li>
      <li>                                           <!-- Navigation -->
        <a data-testname="link">About</a>           <!-- Link -->
      </li>
      <li>                                           <!-- Navigation -->
        <a data-testname="link">Contact</a>         <!-- Link -->
      </li>
    </ul>
  </nav>
</main>

Verifying that an element was rendered

Here is how a test might verify that the Navigation menu was rendered:

await expect('Navigation#link').toBeVisible();

The selector above could be parsed and converted to use the findAllNodes API:

import {
  createComponentSelector,
  createTestNameSelector,
  findAllNodes,
} from "react-dom/testing";

const elements = findAllNodes(
  document.body,
  [
    createComponentSelector(require("Navigation.react")),
    createTestNameSelector("link"),
  ],
);

if (elements.length === 0) {
  // Throw error
}

Additional processing could then be done with the returned element to assert visibility. (See toBeVisible.)

Verifying the number of rendered elements

Here is how a test might verify that three links are rendered by Navigation:

await expect('Navigation Link#link').toBePresentCount(3);

The selector above could be parsed and converted to use the findAllNodes API:

const elements = findAllNodes(
  document.body,
  [
    createComponentSelector(require("Navigation.react")),
    createComponentSelector(require("Link.react")),
    createTestNameSelector("link"),
  ],
  "list"
);

Additional processing could then be done on the matched elements (e.g. toBePresentCount)

Note that given the application described above, the following selectors would also work:

  • App#link
  • App Navigation#link
  • Navigation#link
  • App Navigation Link#link

Implementing advanced pseudo-selectors (e.g. :contains, :text)

Here is how a test might use additional pseudo-selectors to click the “Contact” link rendered by Navigation:

await wwwPage.click('Navigation Link#link:text("Contact")');

The selector above could be parsed and converted to use the findAllNodes API:

const elements = findAllNodes(
  document.body,
  [
    createComponentSelector(require("Navigation.react")),
    createHasPsuedoClassSelector([
      createComponentSelector(require("Link.react")),
      createTestNameSelector("link"),
      createTextSelector("Contact"),
    ]),
  ],
  "list"
);

Portals

Portals cause divergence between the React tree and the host tree. Consider the following demo application:

function Parent() {
  return (
    <div>
      <Child />
    </div>
  );
}

function Child() {
  return (
    <div>
      <Grandchild />
    </div>
  );
}

function Grandchild() {
  return createPortal(
    <div data-testname="portal" />,
    document.getElementById("portal")
  );
}

This app would be seen by React as the following tree:

Parent
  <div>
    Child
      <div>
        Grandchild
          <div data-testname="portal" />

However it might be rendered into the DOM as the following:

<body>
  <div id="root">
    <div> <!-- owned by parent -->
      <div> <!-- owned by child -->
  <div id="portal">
    <div data-testname="portal"/> <!-- owned by grandchild -->

Because findAllNodes traverses the fiber tree, it will be able to find the portal even if given the “root” container. It should be possible then for all of the following queries to find matching host elements:

  • Parent#portal
  • Parent Child#portal
  • Parent Child Grandchild#portal
  • Child#portal
  • Child Grandchild#portal
  • Grandchild#portal

Render props

Render props are another interesting edge case for an API like this. Consider the following demo application:

function Parent() {
  const render = () => (
    <div key="parent" data-testname="parent" />
  );
  return <Child render={render} />;
}

function Child({ render }) {
  return (
    <div key="child" data-testname="child">
      {render()}
    </div>
  );
}

This app would be seen by React as the following tree:

Parent
  Child
    <div data-testname="child"/>
      <div data-testname="parent"/>

And it would be rendered into the DOM as the following:

<div id="root">
  <div data-testname="child">
    <div data-testname="parent">

Because findAllNodes traverses the fiber tree, the following selectors will correctly identify test host nodes:

  • Parent#parent
  • Parent Child#child
  • Child#child

getFindAllNodesFailureDescription

Debugging failed selectors

If a test fails because an expected match was not found, it is useful to provide a developer with actionable feedback. Here is a selector that would not find a match given the example app above:

await wwwPage.click('Header PageTitle Link#link');

In this case, the e2e framework knows that a match was expected (because the click action was used) so it should provide the user with an actionable error message. getFindAllNodesFailureDescription can help with this.

getFindAllNodesFailureDescription(document.body, [
  createComponentSelector(require("Header.react")),
  createComponentSelector(require("PageTitle.react")),
  createComponentSelector(require("Link.react")),
  createTestNameSelector("link"),
]);

In this case the returned description might be something like the following (depending on the displayName values for the specified React components):

``findAllNodes` was able to match part of the selector:
  `Header` > `PageTitle
`
No matching component was found for:
  `Link``

findBoundingRects()

Test may want to take screenshot snapshots of a component:

await expect('Navigation.react').toMatchScreenshot();

We could use the findBoundingRects API for this:

const rects = findBoundingRects(document.body, [
  createComponentSelector(require("Navigation.react")),
]);

User Puppeteer as an example then, we might do:

const rect = merge(rects);

await page.screenshot({
  clip: {
    x: rect.left,
    y: rect.top,
    width: rect.width,
    height: rect.height
  }
});

observeVisibleRects()

Verifying that an element was rendered

The findAllNodes example above could be rewritten to use the observeVisibleRects API instead. This change would likely encompass a larger host subtree but would also eliminate the need for a data-testname. For example:

await expect('Navigation').toBeVisible();

The selector above could be parsed and converted to use the observeVisibleRects API:

let matched = false;

const callback = recs => {
  recs.forEach(({ ratio: number, rect: Rect }) => {
    // Some business logic to approve or reject based on the ratio, e.g.
    if (ratio > 0.5) {
      matched = true;
    }
  });
};

const { disconnect } = observeVisibleRects(
  document.body,
  [createComponentSelector(require("Navigation.react"))],
  callback
);

// Optionally wait some amount of time until the element becomes visible.

disconnect();

if (!matched) {
  // Throw error
}

focusWithin()

Clicking on a link

Given the above example code, we could click on the first link within the Navigation menu using the focusWithint API:

focusWithin(document.body, [
  createComponentSelector(require("Navigation.react")),
]);

// document.activeElement is now the 1st focusable host instance in Navigation.

User Puppeteer as an example then, we might do:

`const`` element ``=`` await page``.``evaluateHandle``(()`` ``=>`` document``.``activeElement``);
``await element.click();`

Entering text into a search input

Given the above example code, we could use the focusWithint API to search:

focusWithin(document.body, [
  createComponentSelector(require("SearchInput.react")),
]);

// document.activeElement is now the 1st focusable host instance in SearchInput.

User Puppeteer as an example then, we might do:

await page.keyboard.type("Type this into the focused element...");
@aarongarciah
Copy link

aarongarciah commented Nov 18, 2021

I wonder if data-testid would be preferred over data-testname because highly adopted libraries like Testing Library and Cypress support it out of the box:

(unless of course data-testname is something React is wanting the community to adopt)

@eps1lon
Copy link

eps1lon commented Nov 18, 2021

Could you elaborate what the target audience of this RFC is? I do not recognize the current state of the (DOM) ecosystem in the historical background you describe. Especially your example for a brittle test selector is not something I have seen outside of Enzyme based testing.

I'm viewing this proposal as it relates to React Testing Library.

It could also make open source libraries like react-testing-library more powerful.

In the context of robust vs brittle test selectors this statement could benefit from a concrete example. For example, Enzyme is way more powerful than React Testing Library. But this is intentional since in the Testing Library mentality more powerful queries lead to hard to maintain tests. The only "power" I can see is that React Testing Library can become renderer agnostic. Though Testing Library is not just about querying but also arranging and acting in tests.

If you only care about the querying part of tests then this proposal would make React Testing Library obsolete in existing renderer agnostic tests. Though I haven't seen any tests that are renderer agnostic.

Regarding Portals Testing Library encourages to always start from the document.body and then scope to the area you want. We consider them implementation details and don't think there's a need for specialized queries.

Test ids provide an alternative to this but they have scaled very poorly in practice.

This is the part of the proposal that I find really interesting. As I understand your criticism, the practice of data-testid requires that components actually pass it to a host element they deem appropriate. With your proposal the target component implementation doesn't need to know about this prop at all. It's just the owner (the component that renders the target component) that needs to know about data-testname?

Though is this really resulting in more robust selectors? Imagine the following UI:

function Input() {
   return <label><input /></label>
}

function App() {
  return <Input data-testname="input" />
}

By querying for data-testname Input doesn't need to know about data-testname. But this can also be a negative when Input decides to change around their host elements. Whereas with data-testid the implementation could always decide what should be returned by data-testid (or the "relevant host element") with data-testname this control is lost. It's not clear to me that either approach is more robust than the other.

@bvaughn
Copy link
Author

bvaughn commented Nov 18, 2021

I wonder if data-testid would be preferred over data-testname because highly adopted libraries like Testing Library and Cypress support it out of the box:

(unless of course data-testname is something React is wanting the community to adopt)

@aarongarciah

I don't think we are taking this strong of a position. If I recall, "name" was decided on because it didn't imply any requirement of uniqueness (nor does the technical design). The term "id" does imply that. I think this was really it.

If it ended up being a sticking point, we could probably support either/both.

@bvaughn
Copy link
Author

bvaughn commented Nov 18, 2021

Could you elaborate what the target audience of this RFC is? I do not recognize the current state of the (DOM) ecosystem in the historical background you describe. Especially your example for a brittle test selector is not something I have seen outside of Enzyme based testing.

@eps1lon

One of the big motivators for this design was Facebook itself. We have tons of integration tests and have seen a lot of brittle patterns at scale. Over time, this lead to Jest e2e– but for it to work correctly, it needed to occasionally dip into React internals in ways that aren't resilient to future planned changes.

Speaking of which, we've also noticed (both within Facebook and externally) that Enzyme usage is a frequent pain point when it comes time to upgrade (because of its usage of internals).

But this is intentional since in the Testing Library mentality more powerful queries lead to hard to maintain tests.

Yeah, FWIW the design of this API was an attempt balance "power" with a subset of queries that would generally not lead to maintainability problems. (That's why the API is so small.)


Regarding your examples and question about data-testname, I'm confused. This attribute would be attached to host elements (e.g. <div>) not to function/class components.

@eps1lon
Copy link

eps1lon commented Nov 18, 2021

Regarding your examples and question about data-testname, I'm confused. This attribute would be attached to host elements (e.g. <div>) not to function/class components.

Then how is it different from existing data-testid approaches that you said scale poorly?

@bvaughn
Copy link
Author

bvaughn commented Nov 18, 2021

Then how is it different from existing data-testid approaches that you said scale poorly?

@eps1lon

Ah, okay. I see what you're asking. Great question!

Firstly, test names and test IDs are the same thing with only a minor semantic difference. It's still a useful concept to have as an escape hatch, just not as the primary mechanism.

Within Facebook, tests used to written primarily using test ids– special attributes added to DOM elements that we stripped from production (e.g. [test-id="LoginButton"]).

When drafting this API proposal, I spoke at length to the Jest e2e team about why these ids didn't work out so well at scale and I think it basically boiled down to a bad developer experience:

  1. People often misunderstand test ids and add them in arbitrary/unnecessary places.
  2. Certain types of tests would require a lot of test ids to be added, which can be slow to do (particularly with heavy abstractions).
  3. People dislike having to manually add the test ids.
  4. Test ids were found to be very brittle/unreliable for testing purposes (e.g. they'd often get moved or deleted by diffs and lead to serious production outages).

There is an additional scenario in which test ids fail (or at least are not scalable for sufficiently large projects)- the case of a component that does not directly render any DOM output. For example:

function LoginButton(props) {
  return <GenericButton {...someProps} />
}

In the case of the component above, LoginButton doesn't actually render a DOM output to attach a test id to, so it couldn't be referenced in a test. Or at least, not in a way that scales well.

  1. You could write your tests to target GenericButton, but since it's a generic component- tests would be brittle. Unrelated changes in the page layout could cause your selector to match the wrong thing.
  2. You could pass a test id prop to GenericButton and have it attach to the DOM element it renders, but what if there are several levels of indirection before an actual DOM element is rendered? What if more than one meta component wants to pass down a test id?
  3. You could add a wrapper <div> with a test id. (In fact, some internal code used this pattern- but unnecessary wrapper elements harm performance.)

Introducing the concept of selectors (internally) to Jest e2e tests saw a dramatic improvement in developer satisfaction around testing, and a big boost in test reliability which translates to a lot of saved money.

@eps1lon
Copy link

eps1lon commented Nov 18, 2021

Firstly, test names and test IDs are the same thing with only a minor semantic difference.

Definitely agree with the ID vs name difference. We don't put any constraint on data-testid. The name is even configurable.

Within Facebook, tests used to written primarily using test ids– special attributes added to DOM elements that we stripped from production (e.g. [test-id="LoginButton"]).

I remember getting builds that did not strip these 😉

(e.g. they'd often get moved or deleted by diffs and lead to SEVs/UBNs)

SEV means an incident (from SEVerity)? I don't know what a UBN is in this context.

In the case of the component above, LoginButton doesn't actually render a DOM output to attach a test id to, so it couldn't be referenced in a test.

Quick terminology check: Are these queries returning host instances e.g. HTMLButtonElement or a React element ({ type: 'button', props: {} })? If they should always return a host instance I don't see a difference between document.querySelector([data-testname="LoginButton"]) and the proposed API.

You could pass a test id prop to GenericButton and have it attach to the DOM element it renders, but what if there are several levels of indirection before an actual DOM element is rendered? What if more than one meta component wants to pass down a test id?

I can't really speak to the testing requirements in facebook. I'd love to understand more why this is a problem. In React Testing Library the mantra stays the same: Indirections are an implementation detail and therefore not subject to selectors. It would probably help to show a concrete test that would only be possible with the proposed API and is more robust than existing approaches.

Introducing the concept of selectors (internally) to Jest e2e tests saw a dramatic improvement in developer satisfaction around testing, and a big boost in test reliability which translates to a lot of saved money.

Which comes back to the original question about the target audience of this proposal. The question remains if the same boost could've been achieved by using React Testing Library.

@bvaughn
Copy link
Author

bvaughn commented Nov 18, 2021

SEV means an incident (from SEVerity)? I don't know what a UBN is in this context.

Basically it means a serious outage that requires immediate attention ("UBN" = "unbreak now")

Are these queries returning host instances

The create-selector APIs return a "selector" (an object with a few specific properties). You use these selectors by passing them to methods like findAllNodes or findBoundingRects.

If they should always return a host instance I don't see a difference between document.querySelector([data-testname="LoginButton"]) and the proposed API.

They're different because they're more powerful than document.querySelector, which can only operate on the DOM tree and has no awareness of React components or the React tree (as it pertains to things like portals).


I'm very intentionally avoiding a Test Selector API vs RTL discussion. There's room for both. Some of the goals of the Test Selector API is to encourage good testing patterns (ones we've seen scale well and aren't brittle) and to be renderer agnostic. If RTL has the same goals, then great. There's room for both.

@sebmarkbage
Copy link

sebmarkbage commented Dec 1, 2021

The way I see this proposal is that we've observed that:

  • Enzyme style tests are way over powered because they can access any implementation detail in the tree, which makes the tests fragile to refactoring. Including changes to React itself. Changing a small part of an implementation detail can break the test which discourages refactoring existing components and adding new ones. There's little in the API that guides you into testing the "public interface" and little to tell you what that is.

  • Testing more E2E event flows seems to be more useful for UI testing and more resilient to refactors/improvements since you're testing a specific user flow. Especially when it comes to imperative code like event dispatching. Rather than trying to invoke a higher level concept part way down the tree. People use it more once they have it.

RTL solves these issues by making introspection much more constrained. Effectively, encouraging only selectors that the user can observe like roles. Therefore you're writing it against the actually UI and not implementation details. However, you can also explicitly opt-in to exposing something interactable with data-testid/name.

However, the observation is that RTL can be too limiting and we want a little bit more of the abstract introspection back.

  • Roles are great once you've drilled down far enough, but not enough to scope where you're looking.
  • data-testid only applies to "host" components. If you have very reusable abstractions, naming it with a hard coded name becomes too generic so now you need to pass it from above. The problem comes from managing the best practices for threading those through levels of abstractions. You effectively need to come up with a convention for naming your higher level components in a globally unique name which is just the name of the component anyway.
  • When you expose too much through testid, you can now start asserting on them as if they were implementation details. In other words, you end up exposing everything about the tree again. So you have to re-add some convention for limiting how you use them.
  • Portals are a problem because they often end up in the root of document.body or similar so your selectors have nowhere to go. Like how do I get the first item of a tooltip, on the flyout on the first News Feed story on Facebook. You can maybe deal with it by concatenating some index and strings to give it a unique ID.
  • Managing all these test ids also has runtime performance concerns in the cases where they.

I suspect these are less of a problem if you use few reusable abstractions, have relatively small pages or just render a smaller component during tests. For something like Facebook, you might have a full little mini-app in just a small subtree of the page that is also repeated many times. There are also a lot of abstractions that get reused and interleaved.

So, what additional things could we expose to give us back some of the power of Enzyme but not too much?

One observation is just that bounding rects is something you can extract at any level. All components have some size and that's how they fit into the parent. How that size is computed is an implementation detail but it'll always be some size. As a thought experiment you can just place a component in a parent DOM node and measure the parent. You've not done anything to break the abstraction. It's also a powerful feature.

You can for example have simple cases like click in the middle of the "Button" component without exposing anything about the DOM structure.

That can also build things like whether it's visible or not. That's not an implementation detail of the component itself.

Similarly, for accessibility purposes, you can always focus somewhere within a component. As a thought experiment you can imagine placing the focus on a parent/sibling and then shift focus to the next focusable item. That's something the user can observe similar to roles. This is also powerful for simple cases like focus the "Button" component without exposing anything about the DOM structure.

How do you get to a "component" without exposing implementation details of the tree? This is a bit of a compromise since arguably the whole app is opaque.

However, there is already a concept in JavaScript of what is public/private given a scope of some code. The modules and package APIs.

Anything that is private to a module and isn't exported is clearly an implementation detail. If I change a child component that is part of a module scope and isn't exported, I have no expectation that will break some caller. So with this API design there's no way you can write a selector against a private component.

Similarly, if your package uses package "exports" field to only expose a subset of modules as the public API, there's no way for other packages to write a selector against components that are private to that package.

By using component instances as the selectors (as opposed to global string names), we use these clearly defined rules for ensuring that abstraction boundaries aren't violated.

If you have access to a component, then it's exposed to you then it's not an implementation detail. You can consume it directly and if you can consume it you must be able to test it.

Also, if a file moves or becomes private, you can easily statically track that everywhere since it's just the module system.

We were also careful not to add certain selectors (like direct sibling selectors) that can observe whether intermediate abstractions are added/removed.

@eps1lon
Copy link

eps1lon commented Dec 1, 2021

All components have some size and that's how they fit into the parent. How that size is computed is an implementation detail but it'll always be some size.

This breaks down very fast once you deal with various position styles. React Devtools (as well as regular browser devtools) already have this problem: facebook/react#22703

If you have access to a component, then it's exposed to you then it's not an implementation detail.

I think we just want to test different things (in an end-to-end test). I ultimately don't care about the React component. I care about what is given to the browser. If you care about the React component then the module level makes sense, I would definitely agree there. But if I care about the browser then the host instances make more sense. In other words, the proposed API wants to test what the developer uses. I want to test what the browser uses.

Given my understand of what end-to-end testing means, the test can't stop at the framework boundary. It has to deal with the thing the user interacts with. And the user is the one using the app not the one writing the app.

For libraries the proposed API definitely fills some holes that Enzyme left (or that are very brittle due to Enzyme's nature). On the app level it's unclear to me whether the proposed API makes tests strictly more robust with regards to user interaction.

@sebmarkbage
Copy link

sebmarkbage commented Dec 1, 2021

This breaks down very fast once you deal with various position styles.

The concept isn't limited to just the raw size but also the bounding box and its stacking order. Some of those details aren't even exposed by the browsers, which is why higher level concepts has to be built on top of it. That's why observeVisibleRects exists as a separate API for example because findBoundingRects doesn't expose enough data. Other methods could be added that builds on the the concept being exposed.

I ultimately don't care about the React component. I care about what is given to the browser.

I get what you're saying but I'm not sure these semantics details are actually that helpful when discussing testing. It's like "unit" or "end-to-end". It's easy to get lost in principles. I think we ultimately care about the same thing and there are just different practicalities to achieving it.

Given my understand of what end-to-end testing means, the test can't stop at the framework boundary.

I think end-to-end just means that you're mostly not stubbing out anything so you're testing the same thing and the I/O on some boundary (the browser in both cases) is based on either data or user input or both being simulated. In both models, this remains true. Although in either model it's possible to have simplified pages (e.g. a permalink page) that doesn't include everything the most common interaction would have (e.g. a feed).

It has to deal with the thing the user interacts with. And the user is the one using the app not the one writing the app.

The limitation in either solution is that you don't have a real user and hard coding specific coordinates or keystrokes (e.g. replaying actual user interactions) is too brittle. So you have to express it conceptually what the user wants to do.

The user doesn't interact with the fifth segment role or the first button role with some text or "the first button with the text Deletion in the first segment that has a title of exactly Post". They interact with concepts like Posts, Stories, Comments. Those concepts are also what the developer is expressing in the code.

If you send a user to manually test a page and perform some task, they don't look for exact test strings or aria roles. They start inferring the concepts based on those but they look for the concepts which makes it more resilient to change.

That's why you need things like testid in the first place I think.

@bvaughn
Copy link
Author

bvaughn commented Dec 17, 2021

FYI I added an API docs section for the create-selector methods. Realized it was missing previously.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment