An agent skill that can rebase an openpilot fork onto the latest upstream/master, handling submodule repos and differences

SKILL.md

name	openpilot-rebase-upstream
description	Rebase an openpilot fork (sunnypilot, dragonpilot, FrogPilot, or a personal fork) onto the latest upstream, replaying modified submodules such as opendbc or panda in step. Use when the user asks to rebase onto upstream, catch up with upstream, or update an openpilot fork to the latest upstream master.

Rebase an openpilot fork onto upstream

Replays the fork's commits onto the newest upstream/master, keeping submodule commits and their corresponding super-repo submodule bumps in sync.

Assumes an openpilot-derived repo: git submodules, a uv-managed venv, and a scons build. Adjust names to the fork at hand — nothing here is specific to one fork or one submodule.

The core problem

A super-repo commit that bumps a submodule records a specific old submodule SHA. After rebasing, that SHA is meaningless — the submodule commit has been replayed onto a new base and has a new one. So every submodule-bumping cherry-pick conflicts, and the fix is to repoint the gitlink at the rebased submodule commit.

Resolve each conflicted gitlink via an old-SHA → new-SHA map. git rebase replays a stack in order and preserves it, so zipping the pre-rebase commit list against the post-rebase one gives an exact, mechanical mapping (Phase 2). Never infer the target from commit subjects: subjects need not be unique, need not match across repos, and a fork may not use matching subjects at all.

The invariant to preserve

Every super-repo commit ends up with exactly the submodule bumps it started with, pointing at the rebased version of the same submodule commits. Concretely:

• A commit that bumped a gitlink before still bumps it after, resolved through the old→new map.
• A commit that bumped no gitlink gains none.
• No submodule change migrates between commits, gets absorbed by a later bump, or collects into a new bump at the end.

Rebasing each submodule as a stack preserves this automatically, because order is preserved and the map is exact. The fork's existing structure is the specification — reproduce it, don't improve it.

Forks vary in how they pin submodules

Two shapes are both common and both fine:

• Interleaved — many super commits each bump a submodule, often sharing a subject with the submodule commit. Here each commit is individually atomically extractable: git show plus its pointer is the complete change, liftable as an upstream PR.
• Trailing pointer — submodule work lives on a submodule branch and a single commit near the tip (subjects often look like <submodule> - branch: <name>) pins the whole stack at once. Individual commits are not atomically extractable, and that is the author's choice, not a defect.

Atomic extractability is a property some forks have, not a goal of the rebase. Never restructure a trailing-pointer fork into an interleaved one, or vice versa. If the structure looks wrong to you, say so after the rebase — don't change it during.

Rules

• Never git reset --hard without creating backup refs first (Phase 1).
• Rebase each submodule onto the commit pinned by upstream/master (git rev-parse upstream/master:<path>), not the submodule's own upstream/master HEAD. They are usually equal and occasionally are not; the wrong one silently desyncs the build.
• Verify after every commit, not just at the end. A mid-stack break is far cheaper to find now than nine commits later.
• Use git -C <abs-path>, never bare cd. Shell cwd persists between commands, so a stray cd silently relocates later ones — a check written with cd can fail on both sides, compare empty to empty, and report a confident false pass.
• Work through the commit list in original order.
• Keep resumable state in a scratch file: base SHAs, ordered commit lists, the old→new submodule SHA map, the record of which super commits bump which gitlink to what, and what's done so far.

Phase 0 — Survey

git fetch --all --recurse-submodules

For the super-repo and each submodule: MB=$(git merge-base HEAD upstream/master), then git log --oneline --reverse $MB..HEAD for the commits to replay. Find which super-repo commits touch submodules by checking each one's --stat against the submodule paths in .gitmodules.

Record the bump table — the specification the rebase must reproduce. For every super-repo commit to replay, for every submodule path, note the gitlink before and after:

for c in $(git rev-list --reverse $MB..HEAD); do
  for p in <submodule-paths>; do
    old=$(git rev-parse -q --verify $c^:"$p" 2>/dev/null)
    new=$(git rev-parse -q --verify $c:"$p" 2>/dev/null)
    [ "$old" != "$new" ] && echo "$(git rev-parse --short $c) $p $old -> $new"
  done
done

Every line is a bump you must reproduce; every commit absent from this table must end up with no gitlink change. Write it to the scratch file — Phase 3 resolves against it and Phase 4 verifies against it.

Then confirm each -> <new> SHA is reachable in that submodule's $MB..HEAD range (git -C <path> merge-base --is-ancestor <new> HEAD). If a pointer targets a commit outside the stack being replayed — an unfetched SHA, a commit on some other branch — Phase 2 produces no mapping for it. Stop and ask.

Do not require submodule commits to correspond one-to-one with super-repo commits, and do not treat subject mismatches as a problem. A submodule stack of 14 commits pinned by a single trailing super commit is a valid shape; so is a 1:1 interleaving. Reproduce whichever you find.

Check whether upstream moved files. Diff the top-level trees (git ls-tree --name-only $MB vs upstream/master), and test whether each path your commits touch still exists (git cat-file -e upstream/master:<path>). Gitlink paths always report missing here — that's an artifact, not a move; confirm real submodule path changes against upstream/master:.gitmodules.

If upstream restructured, git's rename detection will usually replay patches onto the new paths by itself — don't hand-rewrite patches. Raise the limits first or detection quietly gives up on a large move, producing spurious add/delete conflicts:

git config merge.renameLimit 999999
git config diff.renameLimit 999999

Tell the user a restructure is in play before starting, and spot-check the first cherry-pick's --stat to confirm changes landed at the new paths.

Show the user the plan and commit counts before touching anything. If any working tree is dirty, stop and ask.

Phase 1 — Backup

In the super-repo and every modified submodule:

git branch backup/pre-rebase-$(git rev-parse --short HEAD) HEAD

Report the ref names. Recovery is git reset --hard <backup-ref> in that repo. These are local-only and are the only thing making a later force push reversible — keep them until the user confirms they're happy.

Phase 2 — Rebase each modified submodule

In each submodule with commits to replay:

TARGET=$(git -C <super-repo> rev-parse upstream/master:<submodule-path>)
git rebase --onto $TARGET $MB HEAD

Resolve conflicts, git add, git rebase --continue. If a conflict is non-obvious — upstream refactored the same code — stop and ask rather than guessing at vehicle-control logic.

Then record the old→new SHA map by zipping the two lists positionally — rebase preserves order, so row i of one is row i of the other:

paste -d' ' \
  <(git log --reverse --format=%H $MB..<backup-ref>) \
  <(git log --reverse --format=%H $TARGET..HEAD)

Both columns must have the same number of rows. If they don't, a commit was dropped (empty after rebase, or --skipped) — stop and work out which before continuing; a silently shortened map misaligns every row after the gap.

Sanity-check a few rows with %s on each side. Subjects lining up is confirmation the map is right, never the mechanism that built it. Phase 3 consumes this map.

Anchor the rebased commits to a branch before leaving Phase 2:

git -C <submodule-path> branch -f rebased/<name> HEAD

Submodules sit on a detached HEAD after rebasing, so the new commits are reachable only through the reflog. Phase 3 opens with git submodule update, which checks out the upstream-pinned SHA and moves that HEAD off them — and the Phase 1 backup refs point at the pre-rebase commits, so they don't cover this. Without a branch, the entire rebase is one git gc away from being unrecoverable.

Phase 3 — Replay super-repo commits

git reset --hard upstream/master
git submodule update --init --recursive

Clean stale build artifacts

reset --hard leaves untracked build output behind. After a restructure, stale .o/.a/.pyc/.so at the old paths can shadow the new ones and give a meaningless green build.

Dry-run and read the list first (git clean -fdxn). For a routine rebase remove just the known-stale directories and keep the warm build cache. For a messy one — large restructure, unexplained failures — a full git clean -fdx plus a fresh install is the reliable reset. It deletes .venv/; that's fine, the next step rebuilds it. Scan the dry-run output for anything else untracked worth keeping.

The worst offenders after a restructure are whole stale source trees, not just object files. When upstream relocates top-level packages (e.g. moving them under a new parent directory), the old directories survive reset --hard — they are untracked now — and shadow the real modules on PYTHONPATH. Imports then resolve against pre-rebase source: green build, passing import checks, both meaningless. Compare the top-level tree listings from Phase 0 and treat every path that disappeared upstream as suspect.

git clean -fdx will not remove a stale submodule checkout. It skips nested git repositories, reporting Skipping repository <path>/. If the restructure moved a submodule's path, the checkout at the old path is left behind and shadows the new one. Re-run the dry-run after cleaning and clear any leftovers with a second -f:

git clean -ffdxn   # dry-run: shows the nested repos -fdx skipped
git clean -ffdx

Before removing one, confirm it really is stale: the path is untracked at upstream/master (git ls-tree upstream/master <path> prints nothing) and the new location is populated. -ff deletes git repositories outright, so an unpushed submodule commit living only there is gone.

Loop the dry-run until it prints nothing — that empty output is the check that the old tree is fully gone.

Re-sync the Python environment

A venv built against the old upstream is stale: the lockfile moves, and after a restructure the editable install points at paths that no longer exist. Symptoms are ModuleNotFoundError during scons's SConscript phase.

uv sync --all-extras   # extras matter: tooling deps live outside the base set

If system-level dependencies are missing too, tools/op.sh setup handles them, but it may need sudo or be interactive — ask the user to run it themselves in an interactive terminal rather than attempting it.

Per-commit loop

For each commit in original order:

1. git cherry-pick <sha>

2. On a submodule gitlink conflict (CONFLICT (submodule)), look this commit up in the Phase 0 bump table to get the old target SHA, run it through the Phase 2 old→new map, check that commit out in the submodule, then git add <submodule-path>:

   git -C <submodule-path> checkout <new-sha>
   git add <submodule-path>

   Never git add a gitlink still pointing at a pre-rebase SHA — that bakes a broken pointer into history. Never pick the target by subject, and never just take the submodule's current HEAD: if the commit originally pinned a mid-stack commit, HEAD is the wrong answer and silently pulls in later changes.

   If a commit that bumped nothing shows a gitlink change, or one that bumped a gitlink doesn't conflict, stop. Both mean the replay has drifted from the bump table. Check git diff --cached --submodule=short before continuing.

3. Resolve ordinary file conflicts normally.

   During a restructure replay, expect frequent CONFLICT (file/directory): directory in the way of <path> ... moving it to <path>~<sha> instead, alongside a successful commit and exit status 0. This is usually benign: the pre-restructure tree had a compatibility symlink where upstream now has a real directory. Git resolves it itself and rename detection still lands the patch at the new path. Don't hand-fix it — but do confirm, since the message looks alarming and the exit code hides it:

   git show HEAD --stat        # changes landed at the NEW paths
   git status --short          # no stray `<path>~<sha>` left untracked

   Treat it as a finding only if the stat shows old paths, or a ~<sha> entry got committed.

   The conflicts that matter during a restructure are almost all import lines, and the resolution is usually both sides, not either: your commit's semantic change (a new symbol, a reordered import) re-expressed against upstream's new module path. Resolve by evidence, never by picking a side — check the symbol actually exists where you're about to import it from (python -c "from <mod> import <name>", git grep), because taking your own side wholesale reintroduces a path upstream deleted and still builds green.

4. git cherry-pick --continue

5. Build: source .venv/bin/activate && scons -j$(nproc). The venv must be activated, not just invoked — calling .venv/bin/scons directly leaves .venv/bin off PATH, so scons can't find cython and you get Error 127 ("command not found") that looks like a missing system package but isn't. Without source, set PATH and VIRTUAL_ENV explicitly.

6. Import-check the Python this commit touched. The build gives Python no coverage — scons compiles C/C++, Cython (.pyx) and capnp, and has no py_compile, lint or typecheck target, so a green build says nothing about any .py file. After a restructure this is the likeliest failure: a line your commit adds importing a module upstream moved applies cleanly, never conflicts, builds green, then dies at runtime.

   for f in $(git diff --name-only HEAD^ HEAD -- '*.py'); do
     [ -f "$f" ] && python -c "import $(echo "${f%.py}" | tr '/' '.')" || echo "FAILED $f"
   done

   Static checkers do not catch this. py_compile, ruff and ty all pass a file importing a name that no longer exists, even project-aware, because the package directory still exists and only the symbol moved. Importing is the only reliable check. Quick pre-filter: resolve each import line the commit adds (git diff HEAD^ HEAD -- '*.py' | grep -E '^\+\s*(from|import) ').

   Importing also raises SyntaxError, so it subsumes py_compile — no need to run both.

   Importing runs the module's top-level code. Screen it BEFORE importing, never after. In this codebase that means opening raylib/Qt windows, grabbing a joystick or camera, probing USB, or blocking on a device. "Fall back to py_compile when the import fails" does not help — the window already opened and the device was already claimed. Importing a directory's worth of UI and tools/sim modules will carpet the user's desktop in GUI windows.

   Skip, and python -m py_compile instead, any file that:

   grep -lE '^(if __name__|.*\b(raylib|pyray|cv2|metadrive|panda3d)\b)' <files>   # entry points, GUI, sim, hardware

   plus anything under a ui/, tools/sim/, webcam/, debug/ or third_party/ path. When unsure whether a module is safe, py_compile it — a missed import check costs one Phase 5 traceback; a wrong import costs the user their desktop session.

   Scope the file list to the commit, always HEAD^ HEAD. Never widen it to $MB..HEAD for an end-of-rebase sweep: that range spans every file upstream touched in its own commits, so you import hundreds of unrelated modules — which is exactly how the GUI flood happens. There is no version of the tip-wide sweep worth running; the per-commit checks already cover every file you changed, and Phase 5 covers the assembled tree.

7. Grep the commit for leftover conflict markers. A botched resolution can strand <<<<<<< in a file. C/C++ fails the build and Python fails the import, but JSON, YAML, capnp and asset files fail silently and only surface at runtime.

   git diff HEAD^ HEAD --name-only | xargs -r grep -lE '^(<{7}|>{7})( |$)'

8. If any check fails, fix and amend into the current commit so each stays independently valid. If the failure is upstream's rather than the commit's, say so and ask before amending.

   First establish whether the commit was ever green. A mid-stack failure is only yours if the same commit built before the rebase. Check the pre-rebase tree (git grep <symbol> <backup-ref>, or the submodule SHA that commit originally pinned) before touching anything. Two common causes of a failure you did not introduce and must not "fix":

   • Cross-submodule dependency inversion. On a trailing-pointer fork, a commit that bumps submodule A can reference symbols defined in submodule B, whose bump lands in a later commit. Every commit in between fails to build, typically as a linker undefined reference rather than a compile error. To confirm: find the symbol's definition (git -C <B> grep <symbol> <B's rebased branch>), then check it is absent from the submodule SHA that commit originally pinned. If it was absent before too, the breakage is pre-existing.
   • A genuinely incomplete commit the author never built in isolation.

   In both cases the correct action is to reproduce the breakage, not repair it. Note it, carry on, and confirm the tip is green. Silently fixing it makes the rebase no longer a faithful replay and hides a real defect from the author. Report it at the end instead.

   This is why the per-commit build is a change-detector, not a gate: on a trailing-pointer fork some intermediate commits legitimately do not build. Compare against the pre-rebase behaviour, and treat only a newly broken commit as your bug.

Never batch these across commits — the per-commit verification is the point.

Phase 4 — Verify

Confirm the commit count and subjects match the survey, and that git submodule status shows no +/- prefixes.

Bump fidelity: rebuild the bump table (same loop as Phase 0) against the rebased branch and compare it to the one recorded then. They must correspond row for row: same super-repo commits bumping the same submodule paths, in the same order, with each old -> new pair being the Phase 2 map applied to the original pair.

Check both directions — a commit that gained a bump it never had is as wrong as one that lost one. A shorter table means bumps were dropped or absorbed; a longer one means the replay invented them.

Do not check that pinned submodule subjects equal super-repo subjects. That property holds only on interleaved forks, and asserting it on a trailing-pointer fork reports a failure on a correct rebase.

Fidelity — every commit still matches its original

The claim to establish: each rebased commit adds and removes exactly the same lines as the commit it came from, in the same order, in the same file. Run it over the super-repo and each rebased submodule, pairing positionally — old $MB..<backup-ref> against new upstream/master..HEAD, both --reverse, row i to row i, exactly as in Phase 2. Confirm the two lists are the same length first; if they aren't, every pair after the gap is comparing unrelated commits and the whole check is meaningless.

Compare, per file, only the +/- lines (drop the +++/--- headers), in order, after normalizing paths. Build that from git show <sha> --format="", keying each line by the file from the preceding diff --git line. Derive the path normalizations from the Phase 0 restructure findings — a new top-level prefix, an upstream move, a rename your own commit made. Sort by normalized path so a file landing in a different sort position isn't mistaken for changed content.

Three tempting shortcuts that all give the wrong answer:

Approach	Failure
git diff <backup> HEAD	Swamped by upstream's own commits. Says nothing.
Sort all +/- lines together	Passes even if lines moved between files or reordered within one.
Diff whole patches in order	False alarms: hunk headers shift whenever upstream adds lines above yours, and a moved file re-sorts within the patch.

Files with no +/- lines — binaries and pure renames — are invisible to a line comparison. git show --numstat marks them - - or 0 0. Verify those by blob (git rev-parse <sha>:<path> must match on each side), expanding git's rename notation (dir/{old => new}) to the post-commit path. Skipping this silently exempts every asset your commits touch.

Negative-test the check before believing it. A comparison reporting everything identical is worthless until you've watched it fail — re-run it with one path normalization removed, or against a range missing a commit, and confirm it reports a difference.

Read the residual differences, not just the verdict: a hunk-header or path-only difference means incomplete normalization; a difference in an actual +/- line means content was dropped or altered.

Expect zero content differences even on commits you hand-resolved — upstream restructures collide with the context around your changes, not the changes themselves, so resolving correctly leaves your diff untouched. A difference on a resolved commit means you reverted upstream work or dropped your own.

Phase 5 — Run the app

The last check: actually launch it. A rebase can leave every commit building and still produce an app that dies on import. Run the UI entry point (selfdrive/ui/ui.py, or wherever the fork keeps it) in the background, wait ~10s, and confirm the process is still alive — not merely that it started.

Then separate real failures from environment noise. Dev boxes lacking hardware or system services (D-Bus, NetworkManager) produce tracebacks that are caught internally and harmless. Before dismissing one, confirm the failing function is upstream code your commits didn't touch — never wave away an error in a file you modified. A traceback that kills the process is real, and is usually a module path upstream moved.

Known bench noise — WSL2 (and headless dev boxes generally). Repeating ValueError: Object path ('T') must start with / from system/ui/lib/wifi_manager.py (_get_adapter, via jeepney/DBusAddress), often alongside Error getting adapter type 2. There is no NetworkManager on WSL2 (systemctl is-active NetworkManager → inactive), so the D-Bus device path comes back malformed. It loops for as long as the UI runs, is caught internally, and does not kill the process — the UI staying alive past the timer is still a PASS.

Confirm rather than assume, with the two checks that make it noise instead of a finding:

git diff --name-only upstream/master...HEAD | grep -i wifi_manager   # must be empty: not yours
git diff --stat upstream/master HEAD -- openpilot/system/ui/lib/wifi_manager.py   # must be empty: identical to upstream

Report it as a bench artifact, and don't try to fix it — it says nothing about the rebase.

Fixing a bad commit in place

Fix the offending commit; never bolt a fixup on the end, which leaves the broken state pinned in every commit between the two.

git log -1 -S '<the bad line>' -- <file>     # which commit introduced it
git checkout <bad-sha>                       # detached
git submodule update --recursive             # keep gitlinks consistent
# ...fix...
git add <only the files you fixed>           # not -a: the lockfile is often dirty
git commit --amend --no-edit
git rebase --onto $(git rev-parse HEAD) <bad-sha> <branch>

git rebase refuses to start with unstaged changes — stash them first and pop after. Then re-run the bump-fidelity, line-fidelity and boot checks: the replay gave every commit after the amended one a new SHA.

Pushing

Do not push unless the user explicitly asks. A rebase rewrites a published branch, so this is a force push and always their call — and authorization for one push doesn't carry to a later rewrite.

When authorized, push each rebased submodule before the super-repo. The super-repo's gitlinks point at submodule commits that exist only locally until the submodule is pushed; land the super-repo first and anyone cloning gets a gitlink resolving to nothing. Between the pushes, confirm each pinned commit reached the submodule's remote:

git -C <submodule> fetch origin
git -C <submodule> merge-base --is-ancestor <pinned-sha> origin/<branch>

Use --force-with-lease, never --force, so the push aborts if the remote moved. Before pushing, show the user what's being discarded (git rev-list --count origin/<branch> ^<branch>) — on a clean rebase that equals the number of replayed commits, and those are the pre-rebase originals held by the backup refs. A larger number means the remote has commits you never replayed: stop and ask.