Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
NSView Drawing Issue on macOS Big Sur

This is an excerpt from our internal documentation describing an issue with drawing in NSViews on macOS Big Sur.

1️⃣ Introduction

In macOS Big Sur (probably starting with β9), Apple changed the default contents format for backing layers of NSViews. Instead of an explicit CALayerContentsFormat.RGBA8Uint value, an „Automatic“ value is now used. Even though it also resolves into „RGBA8“ in our testing, it has some serious implications as it breaks assumptions our code relies on.

I first stumbled upon this issue in this tweet by Frank. It links to a thread on Apple Forums by Mark that contains valuable information as well as ideas for workarounds. The changed behavior was also confirmed by Marcin in this tweet.

2️⃣ Impact on Diagrams

The issue affects Diagrams in the following ways:

  1. As a canvas-based app that renders contents using the old-fashioned drawing approach via NSView.draw(_:), we heavily rely on drawing optimization. Only the invalidated rectangles in CanvasView are considered for inspection, generation of render operations, and rendering. Since the usage of automatic backing stores prevents us from getting information about rendered rectangles (NSView.getRectsBeingDrawn(_:count:)), we’re out of luck for any optimizations. With the new behavior, we only receive the rectangle for the whole tile (that equals to the dirty rectangle).
  2. A much bigger issue is the difference in handling the clipping areas. It seems like that drawing outside of the invalidated rectangles is forbidden, which leads to missing parts of shadows in cases where the render operation defining the shadow (shape) doesn’t intersect the invalidated rectangles. We run into a similar issue with clipped shadows that reached into neighbored tiles in the past, but the current issue looks slightly different as it’s not tied to tiles.
  3. We assumed a bitmap CGContext with the type kCGContextTypeBitmap would be provided to NSView.draw(_:), which is no longer the case. You now get a CGContext with the type kCGContextTypeCoreAnimationAutomatic. The recently introduced custom tile pattern rendering in CanvasKit leveraged the possibility of getting the height of the context. A non-bitmap context throws an exception when calling CGContext.height.

The issue is only present when linked against the new SDK. The latest version of Diagrams in the MAS (1.0.4) is not affected.

3️⃣ Investigation

In order to understand the issue, I went quite deep with reverse engineering and found out a few interesting bits.

NSViewUsesAutomaticLayerBackingStores

It seems like NSWindows hold a flag telling all CALayers which contentsFormat to use when they’re created. This is the case for layers backing NSViews as well. On Catalina, it seems that „RGBA8“ was the default value, but on Big Sur it changed to „Automatic“. To be more precise, this value is configured by consulting the NSViewUsesAutomaticLayerBackingStores user defaults flag that has been present already on Catalina, but its default value was false, whereas it’s true on Big Sur.

This is the relevant part of the call stack that accesses the user defaults flag:

UserDefaults.object(forKey:) // arg="NSViewUsesAutomaticLayerBackingStores"
_NSGetBoolAppConfig()
_NSSetWindowDepth()
NSWindow.setDepthLimit(_:)
NSWindow._commonAwake()

There is also a new property NSCGSWindow.usesAutomaticContentsFormat that wasn’t present on Catalina. It seems to store the value, but from my testing, it isn’t consulted after being set.

None of these details are relevant, though. The crucial thing is that overriding the value of the flag indeed reverts the behavior for the whole application and everything gets back to normal. The downside is that it affects layers in all views within all application windows, which is probably not what we want. Furthermore, telling from the experience with a similar approach we used for circumventing new scrolling behavior in NSScrollView some time ago, it’s very likely that this user defaults flag will be removed in the future.

Standalone Backing Layer

I prefer to keep the change as local as possible. The workaround presented in the forum thread does just that, as it changes the contentsFormat of an individual layer in NSView.viewWillDraw(). According to my tests, this works fine with layer-backed views that have only a flat layer, e.g. in Marcin’s simplified example.

_NSViewBackingLayer (contents=_NSBackingLayerContents)

Tiled Backing Layer

Although this approach solves the issue with our preview view (getting the height of the CGContext), it is not applicable to the canvas. The CanvasView uses the NSScrollView architecture and typically has a large size. AppKit splits the backing layer into multiple tile layers that come and go.

_NSViewBackingLayer (contents=_NSBackingLayerContents)
└ _NSTiledLayer (contents=_NSTiledLayerContents)
   ├ NSTileLayer (contents=CABackingStore)
   ├ NSTileLayer (contents=CABackingStore)
   ├ NSTileLayer (contents=CABackingStore)
   └ ...

Unfortunately, setting the contentsFormat on the parent layer doesn’t affect sublayers. I tried to set it recursively and even reached to the superlayers (clip view, scroll view), but never got the right result. I haven’t managed to reliably change the format and NSView.getRectsBeingDrawn(_:count:) always reported the size of the whole tile. It felt like there is too much going on underneath that is not meant to be touched. So no local solution found yet.

Roll Your Own System™

This response from Mark is also very worrying. According to him and the message from Apple he got, this is now the preferred way, and developers who need drawing optimizations should „roll their own system“. I mean, we now have Apple Silicon, so there’s no need for optimizations anymore, right?

4️⃣ Current Approach

As the workaround for setting the contentsFormat doesn’t cover all of our cases, I decided to go with overriding the value for the user defaults flag NSViewUsesAutomaticLayerBackingStores and set it to false. This is an immediate but temporary fix, which will be revisited later.

To ensure that CanvasKitMac is always used in the correct way, I added a precondition that checks that the passed-in CGContext is a bitmap context. The nice side effect is that we can safely remove the recently introduced workaround for rendering tile patterns on Big Sur.

If we find a way to apply the local workaround to CanvasView, we should then remove the global one and add this logic to CanvasKitMac.ComponentView.viewWillDraw() that covers both the preview view as well as Xcode previews.

@lukaskubanek
Copy link
Author

lukaskubanek commented Sep 22, 2021

🆕 Follow-up from WWDC 2021

[The AppKit team is] aware of this issue and they confirmed that it’s not intended for NSView.getRectsBeingDraw(_:count:) to return the whole bounds or tiles in the case of NSScrollView. It’s a bug they didn’t manage to fix in macOS Big Sur, but they’ll do their best to fix it in a later release of macOS Monterey.

Furthermore, they confirmed that our current solution of setting a user defaults flag is appropriate. First, it’s safe as it doesn’t have negative implications on other parts of the app as it’s an app-wide override (other than a higher memory footprint which is no big deal). Should they change its internal behavior, the flag would become a noop which also shouldn’t do any harm. Second, it’ll be the way to go for macOS Big Sur anyway, as the fix most likely won’t be deployed back to Big Sur.

This means that there’s no need for rolling out our own system for tracking the rectangles. Ufff… 😅

With that being said, we haven’t yet run tests on macOS Big Sur 11.6 nor macOS Monterey beta. If you happen to know how it behaves on these system, please leave a comment.

@alphagruis
Copy link

alphagruis commented Sep 25, 2021

Hi Lukas,

We encountered the same big bug on macOS Big Sur, that can be permanently fixed by patching (swizzling) the method -[CALayer init]. The fix also works for NSScrollView (all NSTileLayer objects created, handled and destroyed by _NSTiledLayer, will be automatically initialized with the correct storage format).

alphagruis's Advanced Computation Group

//
// CALayer (alphagruisPatch) category.
//

@interface CALayer (alphagruisPatch)
@end

@implementation CALayer (alphagruisPatch)

-(instancetype) initPatch
{
    /* Invoke the original method (only apparently recursive). */
    CALayer * layer = [self initPatch];

    if (layer != nil)
    {
        /* Set the storage format to its own hint. */
        layer.contentsFormat = layer.contentsFormat;
    }

    return layer;
}

//
// Invoked whenever a class or category is added to the Objective-C runtime.
//

+(void) load
{
    static dispatch_once_t once_token;

    dispatch_once (& once_token, ^{

        if (@available (macOS 11.0, *))
        {
            const Class cls = CALayer.class;

            SEL selector = @selector (init);
            Method method = ::class_getInstanceMethod (cls, selector);

        #if qDebug
            NSAssert (method != NULL, @"the instance method -[%@ %@] is NULL", cls, ::NSStringFromSelector (selector));
        #endif

            SEL selectorPatch = @selector (initPatch);
            Method methodPatch = ::class_getInstanceMethod (cls, selectorPatch);

        #if qDebug
            NSAssert (methodPatch != NULL, @"the instance method -[%@ %@] is NULL", cls, ::NSStringFromSelector (selectorPatch));
        #endif

            ::method_exchangeImplementations (method, methodPatch);
        }
    });
}

@end

@lukaskubanek
Copy link
Author

lukaskubanek commented Oct 3, 2021

@alphagruis Thank you for sharing your workaround. Have you, by any chance, tested it on macOS Monterey?

@alphagruis
Copy link

alphagruis commented Oct 4, 2021

Hi Lukas,

good news:

  • on macOS Monterey 12.0 (build: 21A5534d), the bug appears fixed by Apple!

As far as we know:

  • the bug appeared on macOS Big Sur 11.0.1 (but not on macOS Big Sur 11.0.0). Then it was fixed by Apple on macOS Big Sur 11.2.x. It then magically reappeared on macOS Big Sur 11.3.x and still persistent on macOS Big Sur 11.6.1.

The -[CALayer init] patch, if enabled, appears to be harmless on macOS Monterey. Therefore, we have decided for now to keep it enabled (at the time of macOS Monterey release, everything can still change)...

alphagruis's Advanced Computation Group

@lukaskubanek
Copy link
Author

lukaskubanek commented Oct 18, 2021

@alphagruis Once again, thanks for the valuable insights! We observed the issue with a beta of macOS Big Sur and employed the workaround mentioned above. Since then, we haven’t touched it. I’ll try to remove the workaround and check whether it’s still an issue on macOS Monterey once it comes out and I upgrade my development machine. I’ll post my findings in this thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment