Skip to content

Instantly share code, notes, and snippets.

@abathur
Last active June 2, 2020 19:52
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save abathur/5cd33339a742adf0b8b6b828df03f7b1 to your computer and use it in GitHub Desktop.
Save abathur/5cd33339a742adf0b8b6b828df03f7b1 to your computer and use it in GitHub Desktop.
Test procedure for exploring impact of Catalina's networked executable assessments on Nix build times.

Goal

In Catalina, it appears that there's a networked assessment of new executables (including scripts). Nix is constantly generating new executables during a build, so I'm trying to get a bead on how stable/variable this effect is. (AFAIK, it's only meaningful to test this on single-user installs. We have yet to identify any way to successfully exempt builds under multi-user/daemon installs.)

Setup

Open 2 terminal tabs/windows and set up the first to monitor the relevant logs by running:

log stream --debug --info --predicate 'process == "syspolicyd" AND subsystem == "com.apple.securityd" AND category == "gk"'

sanity check

First, we need to make sure assessments/gatekeeper aren't globally disabled:

spctl --status

If this says assessments enabled, you should be good to go. If it says assessments disabled, you'll need to enable them globally before continuing. (Not really sure when this is true--let me know if you know why your assessments are disabled?) If your assessments are being automatically disabled on startup, you may also need to re-run this command after reboots?

sudo spctl --master-enable

Now that you know assessments are globally enabled, run the following sanity check in the other tab/window:

echo $'#!/bin/sh\necho Hello' > /tmp/test.sh && chmod a+x /tmp/test.sh
time /tmp/test.sh && time /tmp/test.sh

The first run "should" be slower than 2nd and should generate log lines unless you've already added an exemption. If you don't see this behavior, check whether you've already added an exemption with:

sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" "select client,allowed from access where service = 'kTCCServiceDeveloperTool'"

It's OK if you already have an exemption--you'll follow the same basic steps, but you'll be measuring in reverse.

Baseline with security assessments

  • run a few time nix-build and GC cycles of something your system doesn't depend on to get a sense of the time they take; these should generate log lines
  • leave at least a minute between runs

Anything helps, here, so I'm happy to hear back on a single package. If you've got spare time and curiosity to run this comparison for multiple packages, that's even more helpful. It'll be faster/easier to collect the first baseline for each package at the same time because creating and disabling the exemption is a little disruptive.

exempt terminal from security assessments

Once you've got a bead on ~normal, add a Security & Privacy "Developer Tools" exemption for your terminal (Terminal.app, iTerm2.app, etc.). (If you already had an exemption, you'll need to remove it and reboot.)

You can open preferences straight to the option with open -n "x-apple.systempreferences:com.apple.preference.security?Privacy_DevTools".

This process should prompt you to quit your terminal after you add the exemption, so you may want to close/stop running processes before doing it. You can also just reboot if you prefer leaving the app open and letting it auto-resume.

Baseline w/o security assessments

  • Make sure to re-set one tab/window to monitor the log stream
  • Repeat the same steps you used for the previous baseline
  • Verify that it's not generating log lines
  • Estimate the % slowdown with assessments relative to without

Report

There are a few pieces of useful information:

  • Roughly where you are in the world. Region/country/city are all fine; as precise as you're comfortable with but no more. :)
  • Any info about your Mac hw (model, year, options) you think is relevant to its performance.
  • MacOS version
  • Nix/nixpkgs version/channel
  • What you built
  • Assessed/exempted build times and % difference

If you've tested a single package, you can send this information in a comment here, mention/query me on IRC, or post it on the associated forum thread in whatever format is convenient. If you ran it for multiple packages, it'd be great to get the list of packages/times/differences in a more legible format like a markdown table or CSV.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment