Quick notes on TVs and standards

gistfile1.md

I need to write this up in more detail soon, but in chatting with Patrick, it is clear that 140 chars isn't cutting it.

The way I've been thinking about the web on TVs and consoles is something like this:

• What are the design patterns and interaction models that would make for a successful experience?
• Can we build those experiences now? If not, what would need to change to make it possible?

In my mind, this starts with the basics of designing an interface that works well on a TV and can be interacted with using a remote control. Once we have the basics out of the way, we can proceed to multiscreen interactions.

A basic television layout

There is a pattern for a basic TV interface layout. It is a grid of some sort. The Netflix interface is a good example.

On many TVs, you navigate around the grid using a d-pad. When you hit the end of a row and then select right on the row, additional items come onto the screen. Same with the bottom of the screen.

This is very similar to the way interfaces in the Smart TV's own menus work. And it makes intuitive sense to people.

How do you build something like that?

Building an interface like that using web technology that would work across devices is currently very difficult. Here are some of the challenges I saw when I was testing TVs this year. These are in no particular order:

• Other than UA, you don't know if it is a TV or not.
• You don't know if the remote is using a pointer or a d-pad.
• If it is a pointer and your web page is built to respond to someone reaching the end of the row in order to push in new items, the page will never work for someone with a pointer.
• Oh, and a lot of TVs allow users to switch between pointers and d-pad in the browser which means you may or may not have an intuitive interface.
• To replicate that Netflix interface, you'd like to get rid of the Chrome. 2013 Samsung TVs support the full screen API, but it doesn't get rid of the chrome. Others you can turn on full screen in a menu that is buried. On the Sharp TV, you switch the setting to enable fullscreen and the next time you open the browser, the chrome is back.
• Samsung's browser supports the d-pad, but you cannot capture its events. It will only go from link to link. In fact, they call that Link Browsing mode and it behaves differently than any other TV browser that supports d-pad that I tested.
• The Panasonic TV browser has a persistent left panel with a picture in picture and some bookmarks that you cannot get rid of.
• The LG 60LA7400 allowed you to switch between pointer and d-pad and acted much like a Wii remote in your hand. However, the LG55LN5700 had a more traditional remote that only had a d-pad. Ironically, this TV only supported the pointer in the browser and did not support switching to a d-pad mode at all.

And these are just a handful of the things I saw when testing on TVs.

But these TVs have great standards support!

Really, they do.

• HTML5 Test Scores
• Modernizr results
• CSS3 Test Scores

The LG TVs I tested had better HTML5 and CSS3 test scores than my iPhone 4S running iOS 6 does.

So what's to be done?

Patrick wrote:

so you meant "we need tv browsers to support standards" rather than "we need new standards for tv browsers"? if so, agree!

I think it might be both. I'm not sure yet.

1. At minimum, it seems that TV browsers could benefit from a recipe of standards that are important on TVs and as well as feedback on where the pain points are. "Hey, fullscreen API is a big deal. Stop screwing it up with your implementations." This is similar to how position:fixed to viewport became more important as mobile took off.
2. It seems like there are defacto standards that it might be nice to make sure everyone agrees with and explicitly say we expect. For example, d-pad controls trigger the keycodes for arrow keys.
3. There are clearly gaps such as the way to identity that a 10ft UI is approach for a given screen that beg for either a new standard or common implementations such as lying about device width that would allow developers to be successful.
4. There are other gaps like being able to know what type of input device is being used and being able to define how the d-pad works that it is unclear to me whether they are new standards, existing standards, better browsers, or what.

And that's just to support the basic interface scenario. That's not beginning to look at the really interesting stuff that can be done with multiple screens or new forms of input:

• multiscreen interactions
• Cross Device Design Patterns

All of which is why I said:

I don’t believe you can build something using existing standards that would provide a good exp across TVs & consoles.

Excellent summary of the problem. Seems like your points 1 and 2 just need some discussion/kicking around to get things to work the way they should. The harder issue seems to be how to identify what state/capabilities a particular device has to the page author. This clearly violates the 'one web' philosophy, I get that, but as your Google TV discovery non-link discovery shows, it's clearly not working... Something needs to give.

What standards exist for a UA to declare attributes about itself? Surely this has been discussed before? I'm wading into I've-got-no-clue waters here but I'm just brainstorming. It would be a non-binding declaration but that would be suggestions to the web page. Things that could go into this would be:

• input type (possible list with the primary type first)
• orientation
• usage pattern (TV, Tablet, phone, Laptop, (with unfortunately lots of 'etc'))

CAVEAT: This approach likely has all sorts of horrible on it. I'm just trying to find SOMETHING that can indicate a broad usage pattern that the web page author can respond to.

@scottjensen you might check out the Client Hints proposal. It may one day scale to carry info like you've listed, though initial implementations in Chrome dev branches only support screen size so far.

Great stuff, this is indeed better than 140 chars ;)

A hopefully not too long response to your points, from my perspective.

Building an interface like that using web technology that would work across devices is currently very difficult.

Agreed. Many of the points actually seem to echo the other current discussion I'm trying to stay involved in, which is the "how can I detect if the user is using a mouse or a touchscreen", where my holistic answer is "design for both, it's not either/or anymore". And sometimes yes, having to still have a completely separate TV (or leanback, big screen, etc) experience may currently be the only option.

Other than UA, you don't know if it is a TV or not

Yup, that's been my major bug-bear too. As I said, I still believe that a smooth solution would be for TVs to respond to viewport meta / @Viewport CSS directives, AND to effectively lie about their device-width/device-height and pixel density. From a user experience, the perceived size of a big TV at 10' is on par with that of a smartphone at 2', so regardless of the real-world pixel density, TV should indicate that it has a high density (2 or 2.5) to account for the perceived density at normal viewing distance. Immediately, that would make mobile-optimised layouts work on TV, including not just the layout, but affordances like larger buttons, larger text, etc.

You don't know if the remote is using a pointer or a d-pad.

And currently on desktop/laptop you don't know if a user is using a keyboard, or a mouse, or a touchscreen, and certainly can't say if they're using either one exclusively (see Chromebook Pixel, Win8 laptops with touchscreen - and there's no concept here of "primary" input either, as they're equipotent).

If it is a pointer and your web page is built to respond to someone reaching the end of the row in order to push in new items, the page will never work for someone with a pointer.

How do you currently account for things like carousels for users with mouse and/or keyboard and/or touch? Same principles apply here. This is not a new problem (the keyboard accessibility issue has been a constant one forever), just one that designers seem to just find out about in a roundabout way ("what, there's no hover on touch? what now?" when there's not been hover - but rather focus - for keyboard users forever too).

In this specific case, make affordances like tiny arrow areas to move back/forward in the carousel? Make them only work with mouse/pointer, if you must. If the user is firing key events, hide them...but bring them back in view when a pointer is moving? Options are there.

Oh, and a lot of TVs allow users to switch between pointers and d-pad in the browser which means you may or may not have an intuitive interface.

See above. Either design an interface that has elements for both modes, or interactively (depending on last used input mode) show/hide them gracefully.

Samsung's browser supports the d-pad, but you cannot capture its events. It will only go from link to link.

Not so dissimilar from what keyboard users have been doing on desktop forever. Or how blind users with VoiceOver on iOS, or TalkBack on Android, experience most sites. Linearly tabbing back/forward through focusable elements in source order. In theory, we should already be accounting for that in our sites.

To replicate that Netflix interface, you'd like to get rid of the Chrome [...]
The Panasonic TV browser has a persistent left panel with a picture in picture and some bookmarks that you cannot get rid of [...]
[...] the LG55LN5700 had a more traditional remote that only had a d-pad. Ironically, this TV only supported the pointer in the browser and did not support switching to a d-pad mode at all.[...]

Now this is getting to a more fundamental area. You, as a designer, have decided that your app would be best experienced without any distracting chrome (this is along the same lines as that advice from the comedy Google TV UI app to not use anchor elements because they'll show a "distracting" tooltip/hint about the link's target), and that it needs a certain input modality. But truth is: your site won't live in a vacuum. You may decide that it would look best without any chrome or work best with d-pad controls, but this is not necessarily what users on that TV will know from any other site they visit. You can argue that they're getting a crappy experience on all other sites and that YOUR site will make it better, and lament that TVs don't give you that control...but this really reminds me of the old days when sites launched chromeless giant new windows (ideally self-centering on your desktop and then expanding to take over the screen.width and screen.height ... remember those?) to control the whole user experience. It's a shade arrogant to say "I know that platform conventions are different, but I want my site to look like this on your TV".

I tweeted this (as a cheeky way to upload the image somewhere), but: beware of the apparent high scores in things like HTML5test for those TVs. It's not uncommon to find that some features are nominally there, but practically useless (as in the example below...a 2012 Sony BluRay that yay, has local/sessionStorage...but only ~50kb/500kb respectively).

If I were a cynic (and not somebody previously on the inside actually hearing those conversations), I'd posit that some OEMs do it to fudge the scores and look better than the competition...

(above image from my Web on TV - the joy and (mostly) pain of TV development / jQuery Europe / Vienna 23.02.2013 presentation)

I agree with designing things on a continuum so they support the different modes. I think that on occasion it would be easier to switch the interface if you could detect what input is being used instead of trying to build interfaces that support all of them at the same time. But I could be persuaded the other way.

The one thing I fundamentally disagree on is that the platform conventions for everything BUT the browser is that the apps take over the screen. It is the same on mobile. This isn't about knowing better, it is about having parity instead of getting stuck in the browser ghetto.

One reason I feel so strongly about this is because I used Vimeo, NPR and other apps on my Google TV for months before I accidentally discovered they were web pages. The web can provide fantastic experiences on TVs. I've seen and experienced it. That's what I'm fighting for.

So yeah, I want to be able to build things as powerful as native apps on these platforms and that means having more control over the full screen. No apologies.

Regarding breaking the browser ghetto: this is exactly the same situation as, say, trying to get parity of web apps on iOS. Without breaking my previous NDAs, I can just hint that some OEMs are actually still not comfortable with just having a kick-ass browser that can provide app-like experience vs their own native apps (because OEMs still, for the most part, live in a B2B world where they prefer to do licensing deals, or run their own content distribution networks that providers need to sign up for). Perhaps a reason why Apple TV, from what I remember, still doesn't actually have Safari (unless we buy the late Steve's comment that people don't want web on TV, which I half agree with). So the battle will be tough, no doubt.

Sorry to come back to this, but another aspect that impacts this discussion (and relates to my "your app doesn't live in a vacuum" point) is that I don't think many of the TVs (apart from Google TV perhaps) have the concept of saving a URL to the homescreen, with its (fav)icon side-by-side with native apps. Certainly the devices I tested from Sony (Bravia TV and BluRay player) don't allow this at all (you start the browser, then can access favourites from there, but no way to launch straight from home). Even with halfway attempts like the Opera TV Store app (effectively a chromeless browser) you first have to launch that app, before getting to your hub of web apps to launch. The situation seems similar on Samsung's "lovely" SmartHub.

So yes, on most platforms, even if your web site/app wants to take over the screen and give an app-like experience, it will first have to be accessed not from the homescreen itself, but from the browser.