Ideas for a report format for happo

report-format.md

Ideas for a report format for happo

After running happo, a snapshot report file is either created or updated. In it, all tests you have specified are listed, with urls to the screenshots. Here's an example format:

{
   // snapshot-report.json
   '<component-name>': {
     '<variant>': {
       '<target>': 'https://url/to/screenshot',
       '<targetB>': 'https://url/to/screenshot2',
     },
     '<variantB>': {
       '<target>': 'https://url/to/some-screenshot',
       '<targetB>': 'https://url/to/some-screenshot2',
     },
   }
   '<component-nameB>': {
     ...
   }
}

In this example, <component-name> is the name of the thing you are testing. I'm thinking this could be a React component name (e.g. <Button>), or a file name (e.g. src/components/Button.jsx).

Each test has a set of variants. In a React world, these would be things like disabled, secondary, without border, etc. Basically a name for a specific set of props.

Each variant is tested in multiple targets. A target can be e.g. Firefox, iOS, Android, Chrome, etc. Each example will have a snapshot url for each target. You'll probably have to configure targets through config. By allowing custom names and properties, we could allow examples to render in different viewport sizes. E.g.

// .happo.js
const { RemoteBrowserTarget } = require('happo');
const ChromeDesktopTarget = require('./ChromeDesktopTarget');

module.exports = {
  targets: {
    'firefox-desktop': new RemoteBrowserTarget('firefox', { viewport: '1024x768' }),
    'firefox-mobile': new RemoteBrowserTarget('firefox', { viewport: '320x768' }),
    'chrome-desktop': new ChromeTarget(),
  }
}

In this example, we're using three targets. The top two are provided by happo, the bottom one is custom. The custom one will have to implement the same interface that the remote service does. At this point, it's basically:

1. Provide an endpoint where the happo client/runner can POST examples to render. (What's a good URL pathname here? How about /snap!?)
2. Send back screenshot image URLs for all rendered examples

Here's more of a real-world example of what the report file could look like:

{
   // snapshot-report.json
   'Button': {
     'standard': {
       'firefox-desktop': 'https://happo.io/@/5140a14ac3a7',
       'firefox-mobile': 'https://happo.io/@/455140a14ac3',
       'chrome-desktop': 'https://some-internal-host/@/4b2ede43e23',
     },
     'secondary': {
       'firefox-desktop': 'https://happo.io/@/35140a14ac3a',
       'firefox-mobile': 'https://happo.io/@/2235140a14ac',
       'chrome-desktop': 'https://some-internal-host/@/d4b2ede43e2',
     },
   },
   'Dialog': {
     'small': {
       'firefox-desktop': 'https://happo.io/@/aa35140a14ac',
       'firefox-mobile': 'https://happo.io/@/cd3aa35140a1',
       'chrome-desktop': 'https://some-internal-host/@/cbd4b2ede43',
     },
     'fullScreen': {
       'firefox-desktop': 'https://happo.io/@/f4aa35140a14',
       'firefox-mobile': 'https://happo.io/@/12cd3aa35140',
       'chrome-desktop': 'https://some-internal-host/@/21cbd4b2ede',
     },
   }
}

Two-pass or one-pass

If you want to, this file can be added to source control. Then people would have to make sure to re-generate the file every time they make changes. If you don't want this in source control, you can run happo twice. First on an old commit (master?) to generate the baseline report, then on the new commit to do the actual diff. And if you want to avoid having to regenerate the report from the old commit (master?) all the time, we can have a mechanism to use a cached file from somewhere. The process would then be something like

1. Get the hash of the previous commit
2. Check a cache if it has a report for that commit hash
3. If not, run happo on previous commit and cache the report
4. Run happo on new commit

Generating report pages

On completing a run, we'll end up with either a "green" run (nothing changed), or we have diffs/new examples. To make it easier to review things in one go, we can have a service that accepts some POST payload, then generates a "review" page for you. The logic would be something like:

1. At the end of the happo run, collect all the examples that are diffing
2. POST these to a remote service, e.g. https://happo.io/generate-report
3. Get a URL back, e.g. https://happo.io/x/34d3xed21f

The URL you get back can then be used in a comment on the PR/commit.