NSView Drawing Issue on macOS Big Sur

NSView Drawing Issue on macOS Big Sur.md

This is an excerpt from our internal documentation describing an issue with drawing in NSViews on macOS Big Sur.

1️⃣ Introduction

In macOS Big Sur (probably starting with β9), Apple changed the default contents format for backing layers of NSViews. Instead of an explicit CALayerContentsFormat.RGBA8Uint value, an „Automatic“ value is now used. Even though it also resolves into „RGBA8“ in our testing, it has some serious implications as it breaks assumptions our code relies on.

I first stumbled upon this issue in this tweet by Frank. It links to a thread on Apple Forums by Mark that contains valuable information as well as ideas for workarounds. The changed behavior was also confirmed by Marcin in this tweet.

2️⃣ Impact on Diagrams

The issue affects Diagrams in the following ways:

1. As a canvas-based app that renders contents using the old-fashioned drawing approach via NSView.draw(_:), we heavily rely on drawing optimization. Only the invalidated rectangles in CanvasView are considered for inspection, generation of render operations, and rendering. Since the usage of automatic backing stores prevents us from getting information about rendered rectangles (NSView.getRectsBeingDrawn(_:count:)), we’re out of luck for any optimizations. With the new behavior, we only receive the rectangle for the whole tile (that equals to the dirty rectangle).
2. A much bigger issue is the difference in handling the clipping areas. It seems like that drawing outside of the invalidated rectangles is forbidden, which leads to missing parts of shadows in cases where the render operation defining the shadow (shape) doesn’t intersect the invalidated rectangles. We run into a similar issue with clipped shadows that reached into neighbored tiles in the past, but the current issue looks slightly different as it’s not tied to tiles.
3. We assumed a bitmap CGContext with the type kCGContextTypeBitmap would be provided to NSView.draw(_:), which is no longer the case. You now get a CGContext with the type kCGContextTypeCoreAnimationAutomatic. The recently introduced custom tile pattern rendering in CanvasKit leveraged the possibility of getting the height of the context. A non-bitmap context throws an exception when calling CGContext.height.

The issue is only present when linked against the new SDK. The latest version of Diagrams in the MAS (1.0.4) is not affected.

3️⃣ Investigation

In order to understand the issue, I went quite deep with reverse engineering and found out a few interesting bits.

NSViewUsesAutomaticLayerBackingStores

It seems like NSWindows hold a flag telling all CALayers which contentsFormat to use when they’re created. This is the case for layers backing NSViews as well. On Catalina, it seems that „RGBA8“ was the default value, but on Big Sur it changed to „Automatic“. To be more precise, this value is configured by consulting the NSViewUsesAutomaticLayerBackingStores user defaults flag that has been present already on Catalina, but its default value was false, whereas it’s true on Big Sur.

This is the relevant part of the call stack that accesses the user defaults flag:

UserDefaults.object(forKey:) // arg="NSViewUsesAutomaticLayerBackingStores"
_NSGetBoolAppConfig()
_NSSetWindowDepth()
NSWindow.setDepthLimit(_:)
NSWindow._commonAwake()

There is also a new property NSCGSWindow.usesAutomaticContentsFormat that wasn’t present on Catalina. It seems to store the value, but from my testing, it isn’t consulted after being set.

None of these details are relevant, though. The crucial thing is that overriding the value of the flag indeed reverts the behavior for the whole application and everything gets back to normal. The downside is that it affects layers in all views within all application windows, which is probably not what we want. Furthermore, telling from the experience with a similar approach we used for circumventing new scrolling behavior in NSScrollView some time ago, it’s very likely that this user defaults flag will be removed in the future.

Standalone Backing Layer

I prefer to keep the change as local as possible. The workaround presented in the forum thread does just that, as it changes the contentsFormat of an individual layer in NSView.viewWillDraw(). According to my tests, this works fine with layer-backed views that have only a flat layer, e.g. in Marcin’s simplified example.

_NSViewBackingLayer (contents=_NSBackingLayerContents)

Tiled Backing Layer

Although this approach solves the issue with our preview view (getting the height of the CGContext), it is not applicable to the canvas. The CanvasView uses the NSScrollView architecture and typically has a large size. AppKit splits the backing layer into multiple tile layers that come and go.

_NSViewBackingLayer (contents=_NSBackingLayerContents)
└ _NSTiledLayer (contents=_NSTiledLayerContents)
   ├ NSTileLayer (contents=CABackingStore)
   ├ NSTileLayer (contents=CABackingStore)
   ├ NSTileLayer (contents=CABackingStore)
   └ ...

Unfortunately, setting the contentsFormat on the parent layer doesn’t affect sublayers. I tried to set it recursively and even reached to the superlayers (clip view, scroll view), but never got the right result. I haven’t managed to reliably change the format and NSView.getRectsBeingDrawn(_:count:) always reported the size of the whole tile. It felt like there is too much going on underneath that is not meant to be touched. So no local solution found yet.

Roll Your Own System™

This response from Mark is also very worrying. According to him and the message from Apple he got, this is now the preferred way, and developers who need drawing optimizations should „roll their own system“. I mean, we now have Apple Silicon, so there’s no need for optimizations anymore, right?

4️⃣ Current Approach

As the workaround for setting the contentsFormat doesn’t cover all of our cases, I decided to go with overriding the value for the user defaults flag NSViewUsesAutomaticLayerBackingStores and set it to false. This is an immediate but temporary fix, which will be revisited later.

To ensure that CanvasKitMac is always used in the correct way, I added a precondition that checks that the passed-in CGContext is a bitmap context. The nice side effect is that we can safely remove the recently introduced workaround for rendering tile patterns on Big Sur.

If we find a way to apply the local workaround to CanvasView, we should then remove the global one and add this logic to CanvasKitMac.ComponentView.viewWillDraw() that covers both the preview view as well as Xcode previews.

@alphagruis Once again, thanks for the valuable insights! We observed the issue with a beta of macOS Big Sur and employed the workaround mentioned above. Since then, we haven’t touched it. I’ll try to remove the workaround and check whether it’s still an issue on macOS Monterey once it comes out and I upgrade my development machine. I’ll post my findings in this thread.