Instantly share code, notes, and snippets.

What would you like to do?
The Problem Of UI (User Interfaces)


Note: a lot of programmers talk about UI without mentionning the user even once, as if it was entirely a programming problem. I wonder what we’re leaving off the table when we do that.

Evaluation Axes

  • input handling, and latency,
  • composition,
  • dataflow,
  • layout,
  • painting,
  • styling,
  • extension,
  • resource demands: power draw, cpu cycles

Caveats/blind spots of homegrown solutions

Accessibility is a big issue. How to make your UI Accessible? Usually platform vendors provide APIs to enumerate/navigate/queryh UI elements, extracting some metadata for text-readers and the like.

Multi-viewport UIs: support for multiple windows, multiple displays/monitors with mixed DPI.


Collaboration: support for more than one editing client for multi-user collaboration on the same data.

Responsive layouts and scalable/zoomables UI help users adapt an UI to viewing distances or screen sizes.

Internationalized input methods. (“IME” on windows)

Adaptability to large teams (adding new controls, new assets, new kinds) without contention.

Multi-touch and touch interfaces create many questions: they allow multiple-items to be interacted with at a time (contrary to the pair mouse-keyboard) ; should this be constrained somehow (to only similar items? to only similar items supporting similar interactions? what if more than one person is touching the screen?) ; they also either expect the items to be bigger or the hotzones to be more lenient.

Hi-DPI requires a great deal more pixels to fill. Also what should be the internal units?

Plugin supports: this exposes lots of problematic scenarios when plugins are allowed to hook to the underlying event loop and desktop APIs. For instance imagine dealing with non-DPI aware plugins within a DPI aware process on Windows.

GPU-based rendering:

  • how does it affect latency?
  • does it fit the type of graphics shown in UI?

Multiple-interactions in one: Very often UI reach a difficulty when we want to select multiple elements and change them all in parallel. Increment/Change more than one element at a time.




Consider each quad representing the bounding box of a UI control. The minkowski sum of that quad and a round region of about radius ‘tr’, representing the tolerance radius, depends on the input device (mouse, touch etc…) and represents the collision box between that input device’s center point and the control. In case n>1 controls match, the ambiguity must be resolved. @idea{separating axes test}

Window Management

In an UI where moving windows or elements is offered to users, it’s nice to satisfy needs for tightly packing those elements. Two options:

  • snap (magnetic)
  • bump+friction (solids) { allows for putting elements next to each other without them touching each other }

Layout algorithm (Flutter)

layout transformation :: min/max width, min/max height, fixed size elements, flexible size elements -> sizes per element

constraints go in, traverse the tree, bring out size of elements traversed. tree has row nodes and column nodes. go through fixed size elements first, taking the dimension that match the node type (row => heights, column => widths) apply size in chosen dimension to flexible elements


Sometimes people get confused between these two opposites:

  • the product is the user interface,
  • the user interface surfaces or allows user interacting with a deeper model or simulation

What remains true is that in the two points of view, data has to be shared across multiple systems, and data ultimately has to be seen and interacted with by the user, and belongs to the user interface.

Some surface level aspects of user interfaces (what people call design):

  • structure, sequence
  • 2d/3d layout
  • chrome, animations

Logic of interactions are difficult to express.

Example1: Firefox Web Render

@url: @title: The Whole Web At Maximum Speed

GPU based rendering.

Transformations used:

  • Page transformed into “stacking context” tree (Compositing tree?)
  • Early culling of display items to remove those that are not shown in the viewport.
  • Compositing tree turned into render task tree, aftering having optimized it (reduce number of intermediary textures
  • Batching draw calls (maximal)
  • Assist pixel shaders by allowing Early Z-culling
    • Opaque pass where opaque objects are draw front to back
    • Transculent shapes are drawn back to front

Pathfinder project: rendering font glyphs on GPU as well @url:


Reading notes for @url:


Confusing when it's more than one level deep. User going back to their UI: "How did I put my layout together?"

Window Management

  • Ability to move windows
  • Sizing vertically, horizontally, diagonally

However it gets tricky to lay things out. Solution: Snapping extremities to boundaries of windows, with visible guide lines. Snapping extremities to the parent container also makes sense.

Auto-anchoring feature. Whenever two windows are snapped together, moving the separator between resizes each side. Windows are "connected." Pressing down for a while then moving allows you to let go of that restriction.

Resizing containers

Internal windows are moved and resized using a physics (spring-mass) system.

UIAutomation + Accessibility module

  • describing an UI as a tree
  • element types:
    • content: textual, visual
    • container: . regions tagged with some purpose . to assist with navigation
  • focus:
    • there is a concept of focus connected with non-spatial control devices such as the keyboard

Any node may have 0..n children elements

Why a tree? Alternatives?

Linear sequences: Focus management conflict. Groups tagged with ids? (I.e. the tree exists somewhere; ids may be non-sensical)


APIs for defining trees:

  • serial form: s-exps / xml / json / iff
  • host language literal equivalent
  • element based:
    • construction of fragments
    • children list manipulation
    • re-rooting
  • iterator based (zipper?)
  • infix/postfix

Some Problems:

  • compactness
  • can lead to creation of graphs




  • WAI-ARIA proposes some tags to inform about roles of UI elements
  • Microsoft's UIAutomation also has control types

General Information about Accessibility on Multiple Platforms on Linux ATK appears to be a thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment