Unified solution to <picture> element

unified-picture.md

On Feb. 6th, I published a proposal for a simplified <picture> design, in order to make the element accessible to end-users. After mailing list dicussion, the community was asked for a unified solution that doesn't sacrifice any functionality or performance present in original design.

Without breaking compatibility or usability of the current <picture> design, I believe the that we can also support a simplified, declarative model that focuses on use case 3.1, and is easier for end-users and content management systems to suppport. I introduced the unified proposal on Feb. 27th 2013 via the public-html-comments mailing list, and also notified respimg.

In lieu of duplicating all content from my original proposal, I've only specified the differences between the current <picture> spec and the unified system. Please read the the original proposal before assuming a flaw or loophole exists.

Example

  @media (min-width: 18em){
      picture{
          max-width:320px;
      }
  }
  @media (min-width: 45em){
      picture{
          max-width:800px;
      }
  }

  <picture alt="Description of image subject."> 
    <source src="large.webp" w="1200" h="800" type="image/webp" />
    <source src="large.jpg" w="1200" h="800" />
    <source src="med.jpg" w="600" h="400" />
    <source src="small.jpg" w="300" h="200" />
    <img src="small.jpg" alt="Description of image subject."> 
  </picture>

##Suggested changes to the Use Cases and Requirements for Standardizing Responsive Images (Issue #30)

11 The solution SHOULD offer an method to leverage breakpoints defined in CSS.

12. The solution SHOULD support a simplified syntax to support primary use case 3.1 (preferably a list of images and their dimensions), in order to reach users of content management systems and those without detailed knowledge of CSS media queries.

This allows complexity to be moved from HTML to CSS, and removes the need for high-volume repetition of breakpoint logic.

Authors who wish to use responsive web design will be able to use a CSS framework or snippet and matching CSS classes on <picture> to achieve responsive images - a path much less intimidating than CSS media queries, and much easier for CMSes and authoring tools to support (how would a GUI for media queries + srcset be designed)?

##Suggested changes to the The Picture Element draft

Add attributes w and h to <source>

Define 2 additional attributes on the <source> element: w and h. These attributes should describe the dimensions of the image. The values are not required to be precise, but incorrect values may cause incorrect image selection; the browser does NOT re-evalutate selection based on the actual dimensions of the received image. If these attributes are used in conjunction with srcset, they describe the lowest-dpix image, and the browser may assume that the dimensions for all other sources can be calculated through multiplication.

Matt Marquis brought up the point that using existing attribute names width and height to provide estimates for the selection algorithm, while not using them to enforce a display width or height, would not be consistent with existing usage of those attributes.

Define alternate selection behavior when w or h are present (Issue #22)

When the <picture> element contains <source> element(s) with a w or h attribute, wait until CSS has been downloaded to begin inital media selection. In this scenario, the performance characteristics of <picture> would precisely match those of CSS background-image: instead of <img>. This statement is based on analysis of WebKit source code and profiling; comments by browser developers are welcome.

Using the width, height,mid-width, max-width, min-height, and max-height values applied to the <picture> element, select the best image based on the w and h size declarations, excluding all <source> elements with unsupported image types and non-matching media queries.

Note: Modern browsers support high-quality image downsampling (at least on the refresh pass); therefore we can safely say that it is better to download an image that violates max-* constraints than one which violates min-* constrains.

Declarative algorithm selection used when w and h are present

1. Exclude all sources with unsupported media types and non-matching media queries.

2. If no min-* or max-* constraints exist on the element, and no dimensions have been set in CSS, use the first matching ‘source’ element, and select the appropriate image from srcset if used. If constraints exist, continue to #3.

3. Expand all ‘srcset’ entries using multiplication, retaining original order. We should now have a simple list of URL:Dimensions pairs.

4. Multiply all constraints (min-width, max-width, etc) by the device pixel ratio, so all values involved are in physical pixel units. The browser may choose to use a device pixel ratio of 1 here instead of the actual device pixel ratio, if lower bandwith is required.

5. If a fixed size has been applied to the <picture> element with CSS, use the greater of (width and min-width) for the min-width constraint, and do the same for height and min-height). Now we only have min/max values for with and height.

6. Select the first image which meets all present constraints (between 1 and 4 constraints exist at this point).

7. If no image matches, remove the max-* constraints from the query (these will still be applied to the resulting image for client-side scaling, see Note).

8. If no image matches the min-* constraints, try only with min-width, then with only min-height.

9. Fallback to <img> element.

Dealing with inconsistent usage of w and h.

While CMSes and authoring will likely be quite consistent about populating w and h, as it can be done with javascript, we must specify behavior with an image only includes one of the attributes; and when not all images declare dimensions.

If 1 or more <source> elements within <picture> use the w or h attributes, any <source> element lacking both w and h will never be considered.

If a <source> element only uses w OR h, but not both, it will only be considered if (a) no constraints exist for the missing value, (b) all other source elements also lack the same value, or (c) all prior source elements fail to meet the min-width and min-height constraints together and individually.

Limitations of purely dimension-based selection

1. It does not offer a way to make pie charts readable on monochrome displays; the media attribute must be used.
2. It does not offer a way to distinguish between art-direction and high-resolution. I.e, you may wish to display a different image, of the same dimensions, on a retina device, in order to make objects in the image more visible.

Based on commentary from current users of responsive imaging, these scenarios are infreqent; most responsive images are simply scaled versions of a single, original image, as this is a much more manageable scenario. Custom art direction tends to have poor ROI for content authors, considering pinch-to-zoom has become standard across most mobile devices.