automation-report.md

Real-browser testing tricks on macOS

A field guide to the techniques used in this session to reproduce and diagnose a Safari-only CSS bug (a spurious horizontal scrollbar inside a <textarea> driven by its placeholder) against real browsers installed on the machine, not just headless Playwright builds.

The core insight that drove most of this: the bug only reproduced in real Safari, and Playwright's bundled WebKit is not the same as Safari. So the workflow had to drive the actual /Applications/Safari.app and capture its real window pixels. That requires a different toolkit than normal Playwright scripting.

---

1. Stand up the app without its real auth

The site gates the feature behind GitHub OAuth. Rather than do a real login, run Datasette directly and forge a signed actor cookie using Datasette's own signer (so the signature is valid for that --secret).

# Launch with a known secret and an open agent permission
GITHUB_CLIENT_ID=x GITHUB_CLIENT_SECRET=x DATASETTE_SECRETS_GEMINI_API_KEY=x \
  uv run datasette serve --host 127.0.0.1 --port 1237 \
    --config datasette.yml --template-dir templates \
    --static static:static --plugins-dir plugins \
    --internal internal.db --secret 1 \
    global-power-plants.db content.db

Sign a cookie for {"id": "root"} using the same machinery the server uses:

# uv run python sign.py
from datasette.app import Datasette
ds = Datasette(secret="1")                       # must match --secret
print(ds.sign({"a": {"id": "root"}}, "actor"))   # namespace is "actor"
# -> eyJhIjp7ImlkIjoicm9vdCJ9fQ.<sig>

Two ways to use the resulting cookie:

• Root token URL (Datasette built-in, but single-use per token): http://127.0.0.1:1237/-/auth-token?token=...
• Set ds_actor directly — works in Playwright and survives reloads:

await context.add_cookies([{
    "name": "ds_actor",
    "value": "eyJhIjp7ImlkIjoicm9vdCJ9fQ.<sig>",
    "url": "http://127.0.0.1:1237",
}])

For a throwaway permissions setup, copy the config and loosen the gate so you can test anonymously instead of forging cookies at all:

permissions:
  datasette-agent: true          # was: id: "*"
  datasette-agent-explore: true

---

2. Drive Playwright against the installed browser binaries

Playwright ships its own browser builds under ~/Library/Caches/ms-playwright/. You can point launch() at a specific one, or launch the standard engines. Useful for quick scripted measurement, but remember these are not the user's Safari.

import asyncio
from playwright.async_api import async_playwright

# A specific Chromium build on disk
EXE = ("/Users/<you>/Library/Caches/ms-playwright/chromium-1223/"
       "chrome-mac-arm64/Google Chrome for Testing.app/Contents/MacOS/"
       "Google Chrome for Testing")

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(executable_path=EXE, headless=True)
        ctx = await browser.new_context(viewport={"width": 1200, "height": 900})
        page = await ctx.new_page()
        await page.goto("http://127.0.0.1:1237/")
        await browser.close()

asyncio.run(main())

Run with deps pulled on the fly:

uv run --with playwright python script.py

Forcing classic (non-overlay) scrollbars in Playwright

macOS hides scrollbars by default (overlay style), which masks scrollbar bugs. Each engine exposes a different knob:

# Chromium: disable overlay scrollbars via a feature flag
await p.chromium.launch(args=["--disable-features=OverlayScrollbar"])

# Firefox: legacy scrollbars via a user pref
await p.firefox.launch(firefox_user_prefs={"ui.useOverlayScrollbars": 0})

Gotcha: Playwright rejects single-dash Chromium args like -AppleShowScrollBars with BrowserType.launch: Arguments can not specify page to be opened. Only --double-dash Chromium flags are accepted.

---

3. Toggle macOS scrollbar visibility — reversibly, per app

The system-wide setting lives in AppleShowScrollBars. You can scope it to a single app's bundle id so you never touch the user's global preference. Always restore it in the same command so an error can't leave it changed.

# App-scoped (preferred — Chrome for Testing only)
defaults write com.google.chrome.for.testing AppleShowScrollBars Always
uv run --with playwright python script.py
defaults delete com.google.chrome.for.testing AppleShowScrollBars   # restore

# Other bundle ids seen this session:
#   org.mozilla.nightly      (Playwright Firefox)
#   org.webkit.Playwright    (Playwright WebKit)

# Global (use sparingly; always restore):
defaults write -g AppleShowScrollBars Always
# ...run tests...
defaults delete -g AppleShowScrollBars
defaults read  -g AppleShowScrollBars 2>/dev/null || echo "restored"

Find an app's bundle id from its Info.plist:

/usr/libexec/PlistBuddy -c "Print :CFBundleIdentifier" \
  ~/Library/Caches/ms-playwright/webkit-2287/Playwright.app/Contents/Info.plist

In this session, neither the macOS pref nor the Playwright knobs reproduced the bug — which was itself the clue that the bug was specific to real Safari's rendering, not WebKit-the-engine. That sent us to section 4.

---

4. Drive the REAL browser and screenshot its actual window

This is the key trick. To test the user's genuine Safari / Firefox / Chrome:

1. Open a URL in a named app with open -a.
2. Find that browser's on-screen window id via Quartz.
3. Capture just that window with screencapture -l <id>.

open -a Safari  "http://127.0.0.1:1237/"
open -a Firefox "http://127.0.0.1:1237/"
open -a "Google Chrome" "http://127.0.0.1:1237/"

List windows (number, title, bounds) for a given app — no accessibility permission needed, unlike AppleScript/System Events:

# uv run --with pyobjc-framework-Quartz python windows.py
import Quartz

wins = Quartz.CGWindowListCopyWindowInfo(
    Quartz.kCGWindowListOptionOnScreenOnly, Quartz.kCGNullWindowID)

for w in wins:
    if (w.get("kCGWindowOwnerName") or "") == "Safari":
        print(w.get("kCGWindowNumber"), "|",
              w.get("kCGWindowName"), "|", w.get("kCGWindowBounds"))

Capture that exact window to a PNG (no shadow, no cursor):

screencapture -x -o -l 166891 /tmp/safari-modal.png
#   -x  no capture sound
#   -o  omit window shadow
#   -l  capture the window with this CGWindowID

Then read the PNG back to inspect it.

Why not AppleScript? osascript -e 'tell application "System Events" ...' fails with osascript is not allowed assistive access. (-1719) unless the controlling app is granted Accessibility permission. The Quartz window-list approach sidesteps that entirely — it only needs Screen Recording permission, which the terminal/host already had here.

Title matching caveat: a single app has many windows; match on the page <title> to find the right one (e.g. "Datasette Agent" in name), and beware that open -a may reuse an existing window or open a new tab, so re-query the window list after each open.

---

5. Auto-open transient UI so a screenshot can catch it

The modal under test only appears after a / keypress. You can't easily send a real keystroke to an arbitrary window, but you can make the page open itself by injecting a tiny script into the template — dispatch a synthetic event matching exactly what the app listens for.

The app's listener (for reference):

document.addEventListener("keydown", (e) => {
  if (e.key === "/" && !this.isInputFocused() && !dialog.open) { /* open */ }
});

Inject a matching synthetic event (place it inside a real template block so Jinja/CSP don't drop it):

<script>
window.addEventListener("load", function () {
  setTimeout(function () {
    document.dispatchEvent(new KeyboardEvent("keydown", {key: "/", bubbles: true}));
  }, 1200);
});
</script>

Serve it via --template-dir pointing at a throwaway copy of the template so you never edit the real project files:

mkdir -p /tmp/ds-test/templates
cp templates/index.html /tmp/ds-test/templates/index.html
# ...edit the copy, then: datasette serve --template-dir /tmp/ds-test/templates ...

---

6. Exfiltrate live measurements from the real browser

Screenshots show symptoms; to get numbers (scrollWidth, computed styles, devicePixelRatio) out of a real browser you can't script, run a one-file HTTP collector and have the injected page POST its findings back. CORS-open so any origin can reach it.

# /tmp/collector.py  —  python3 /tmp/collector.py &
from http.server import HTTPServer, BaseHTTPRequestHandler

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", 0))
        open("/tmp/diag.json", "w").write(self.rfile.read(n).decode())
        self.send_response(200)
        self.send_header("Access-Control-Allow-Origin", "*")
        self.end_headers()
    def do_OPTIONS(self):
        self.send_response(200)
        self.send_header("Access-Control-Allow-Origin", "*")
        self.send_header("Access-Control-Allow-Headers", "*")
        self.end_headers()
    def log_message(self, *a):  # quiet
        pass

HTTPServer(("127.0.0.1", 9999), H).serve_forever()

Injected page-side probe — note it reaches into the shadow DOM of the Web Component, which site CSS can't touch but JS can:

const host = document.querySelector("navigation-search");
const ta   = host.shadowRoot.querySelector("textarea");
const cs   = getComputedStyle(ta);
fetch("http://127.0.0.1:9999/diag", {
  method: "POST",
  body: JSON.stringify({
    dpr: window.devicePixelRatio,
    scrollWidth: ta.scrollWidth, clientWidth: ta.clientWidth,
    whiteSpace: cs.whiteSpace, width: cs.width,
  }),
});

rm -f /tmp/diag.json
open -a Safari "http://127.0.0.1:1237/"
sleep 8
cat /tmp/diag.json    # the real Safari's measurements

In-page A/B mutation to isolate a cause

Once you can post results back, script a sequence of mutations and re-measure after each — this is how the placeholder was pinned as the culprit (scrollWidth 648 with placeholder, 566 without):

const m = () => ({sw: ta.scrollWidth, cw: ta.clientWidth});
const out = [["baseline", m()]];

const ph = ta.placeholder;
ta.placeholder = "";          out.push(["no placeholder", m()]);
ta.placeholder = ph;          out.push(["restored", m()]);
ta.style.resize = "none";     out.push(["resize none", m()]);
ta.style.resize = "";
ta.style.overflowX = "hidden";out.push(["overflow-x hidden", m()]);
ta.value = "hello";           out.push(["with value", m()]);

fetch("http://127.0.0.1:9999/diag", {method: "POST",
  body: JSON.stringify(out, null, 1)});

---

7. Static HTML repro harness (no app needed)

To bisect CSS without the whole stack, copy the exact rules into a plain HTML file with several variants side by side, open it in the real browser, and screenshot. A small script reports the environment (UA, scrollbar thickness, DPR) on-page:

<div id="info"></div>
<textarea class="plugin" rows="3" placeholder="Ask a question about your data..."></textarea>
<script>
  const d = document.createElement("div");
  d.style.cssText = "overflow:scroll;width:100px;height:100px;position:absolute;visibility:hidden;";
  document.body.appendChild(d);
  const thickness = d.offsetWidth - d.clientWidth;   // 0 = overlay scrollbars
  d.remove();
  info.textContent = `scrollbar: ${thickness}px | ${navigator.userAgent} | dpr ${devicePixelRatio}`;
</script>

open -a Safari /tmp/textarea-scrollbar-test.html

For faithful shadow-DOM reproduction, build the component in JS with attachShadow({mode:"open"}) and replicate dialog.showModal() + the slideIn keyframes — subtle layout bugs can depend on those.

---

Cleanup checklist

Everything was kept reversible and outside the project tree. Things to undo:

# Stop background servers / collector
pkill -f "datasette serve --host 127.0.0.1 --port 123"
pkill -f "collector.py"

# Restore any scrollbar prefs you set (app-scoped or global)
defaults delete com.google.chrome.for.testing AppleShowScrollBars 2>/dev/null
defaults delete -g AppleShowScrollBars 2>/dev/null
defaults read  -g AppleShowScrollBars 2>/dev/null || echo "global pref unset"

# Throwaway artifacts all under /tmp:
#   /tmp/ds-test/  /tmp/*.py  /tmp/*.html  /tmp/diag.json  /tmp/*.png

---

Quick reference

Need	Tool
App without real OAuth	forge ds_actor via Datasette(secret=...).sign(...)
Scripted measurement	Playwright + --with playwright
Force classic scrollbars	--disable-features=OverlayScrollbar (Chromium), ui.useOverlayScrollbars=0 (FF), defaults write ... AppleShowScrollBars Always
Test the user's real Safari	open -a Safari + Quartz window id + screencapture -l
Find a window id	CGWindowListCopyWindowInfo (no a11y permission)
Capture one window	screencapture -x -o -l <id> out.png
Open transient UI for capture	inject synthetic KeyboardEvent via --template-dir copy
Get numbers out of a real browser	local CORS-open HTTP collector + page fetch POST
Reach into a Web Component	host.shadowRoot.querySelector(...) (JS only; CSS can't)
CSS bisect	static side-by-side HTML harness