Three parameter-efficient fine-tuning methods surfaced via Outrider, drafted through the smellslikeml/peft fork, and shepherded into huggingface/peft — the fine-tuning ecosystem's hub. This gist tracks what's landed, what's in review, what's pending on external coordination, and how the shape of these contributions compares against the year's other merged tuner PRs.
Per Appendix G Table 15 of Scaling DoRA (arXiv:2603.22276), five of six DoRA-supporting fine-tuning frameworks route their implementation through PEFT:
| Framework | DoRA implementation |
|---|---|
| HuggingFace PEFT | direct — the canonical implementation |
| Unsloth | falls back to PEFT for norm computation |
| SWIFT | no custom DoRA code — delegates to PEFT |
| LLaMA-Factory | no custom DoRA code — delegates to PEFT |
| Axolotl | no custom DoRA code — delegates to PEFT |
| torchtune | independent implementation (own copy of the same algorithm) |
A PEFT-side improvement lifts five of six frameworks with no additional integration work. Contributions that would otherwise need to be repeated across five downstream projects can land once at the ecosystem hub. That leverage is why the target is worth the review overhead.
Paper: Zhang & Pilanci, Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models — arXiv:2402.02347
Upstream PR: huggingface/peft#3382 — merged 2026-08-03 (+401 / −3, 6 files, 32.7 days review latency)
Fork branch: smellslikeml/peft feat/riemannian-preconditioned-lora
Predecessor drafts: smellslikeml/peft#8 (still open — factored weight-norm proposal) → landed to smellslikeml/peft:main via smellslikeml/peft#14 before upstream filing
Replaces the Euclidean gradient with the natural gradient computed under the Riemannian metric induced by the low-rank matrix manifold. Aligns optimizer steps with the intrinsic tangent space; quotients the (A, B) ↦ (A·C, C⁻¹·B) reparameterization gauge that Euclidean SGD/Adam otherwise depends on.
Analytical composite of PR #3382 (Outrider-drafted; how it sits in the year's PR cohort):
Six panels: (a) size — 401 LOC additions (p82 of the year's merged PRs); (b) composition — 41% test + 48% src + 11% benchmark, well above the cohort median's 13% test / 1% benchmark; (c) merge latency — 32.7 days (p89), lands despite being from a first-time contributor; (d) scope shape — focused + additive (didn't refactor unrelated code); (e) author recurrence — the contributor's first merged PR to a repo where the top maintainer has 174 merged; (f) three feature-utility signals — registry attachment, backwards-compat commitment (no default flip, no feature flag, no kwarg on existing APIs), and bench-harness registration under method_comparison/MetaMathQA/.
Paper: Ilin, Zmushko & Richtárik, Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning — arXiv:2607.09287
Coordination Issue: huggingface/peft#3450 — direction confirmed by @BenjaminBossan
Upstream PR: huggingface/peft#3518 — open, filed 2026-08-05 (+1309 / −3, 24 files)
Paper-author sign-off: vectozavr/SuperTuning#3 — Ivan Ilin (paper first author, @vectozavr) is a co-author on the branch commits
Fork branch: smellslikeml/peft feat/supertuning-supra-magnitude
Superseded internal draft: smellslikeml/peft#10 — closed, rebuilt on current main with the API reduction the maintainer requested (removed set_precomputed_indices in favour of the deterministic magnitude scoring for the first landing)
Freezes the base weight and trains only a sparse support of scalar entries selected by weight magnitude — a distinct point in the trainable-parameter Pareto vs LoRA. Setting r additionally allocates a LoRA-style low-rank adapter composed additively (the paper's Supra hybrid). Runs against method_comparison/MetaMathQA/ produce numbers directly comparable to the harness's existing LoRA / SHiRA baselines.
Paper: Zelenin & Zhuravlyova, Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels — arXiv:2603.22276
Internal PR: smellslikeml/peft#18 — open, awaits license clarification before upstream filing
Fork branch: feat/dora-factored-kernel
Kernel package: remyxai/dora-factored-kernel on HF Hub
Blocking coordination: sockeye44/dorafactors#1 — upstream reference has no LICENSE file, blocking downstream redistribution
Sequencing: two-step path recommended by @BenjaminBossan — kernels on HF Hub first, PEFT integration second
Two contributions bundled: (a) a factored-norm identity that replaces one [d_out × d_in] materialization with three small matmuls — ~5% per-step training-time speedup at r=2048 on Qwen-7B, 316× per-module intermediate reduction at the paper's r=64 regime; (b) fused Triton kernel implementing DoRA's compose + norm as a single-pass GPU op.
Concrete comparison to the year's tuner-adding PRs (source: gh pr view <n> --repo huggingface/peft):
| PR | Method | Merged | +Additions | −Deletions | Files | Notes |
|---|---|---|---|---|---|---|
| #2584 | SHiRA | 2025-07-14 | +1623 | −9 | 27 | closest analog for our sparse work |
| #2851 | GraLoRA | 2025-11-18 | +1238 | 0 | 20 | — |
| #3037 | PSoFT | 2026-02-27 | +1556 | −1 | 23 | — |
| #3084 | PEANuT | 2026-03-16 | +1096 | 0 | 23 | — |
| #3195 | BEFT | 2026-04-30 | +773 | −5 | 22 | — |
| #3382 | Riemannian LoRA | 2026-08-03 | +401 | −3 | 6 | Outrider-drafted, merged |
| #3518 | Super-Tuning & Supra | in review | +1309 | −3 | 24 | Outrider-drafted, open |
Observations:
- #3382 landed at the compact end of the range — 401 LOC vs a cohort median of ~1150 LOC. Small, focused, all-additive. The composite analytic above shows why the size worked: exceptional test coverage (41%) and benchmark integration (11%) offset the small src surface.
- #3518 sits in the median — 1309 LOC across 24 files, test file + docs page + integration hooks in 6 generic test files alongside comparable tuners (Shira). Same shape as PSoFT, PEANuT, GraLoRA.
Neither PR is an outlier — both are shaped like contributions that PEFT reviewers routinely accept.
Outrider is a cross-repo exploration agent. It surfaces papers whose methods are candidate targets for a given host repo, drafts a fork branch that implements the method following the host repo's conventions, and hands the draft off for human refinement before a public PR is opened.
All three contributions above were surfaced by Outrider, drafted through the smellslikeml/peft fork, and then shepherded by a human through coordination-issue-first review with @BenjaminBossan. The rhythm is:
- Method discovery — arXiv ↔ repo mapping identifies candidates whose mechanisms haven't been ported to the host repo
- Draft on the fork — Outrider produces an initial branch on
smellslikeml/peftwith tests, docs page, integration hooks - Human refinement + coordination — open a coordination issue upstream (per PEFT's
CLAUDE.md), incorporate maintainer feedback, iterate on the fork - Upstream PR — file against
huggingface/peft:mainreferencing the coordination issue and the paper-author sign-off
The composite PR #3382 analytics above show the utility of this rhythm: even a first-time contributor's PR can land through PEFT's rigorous review process when it arrives shaped like the maintainer expects — focused, additive, test-heavy, with a durable feature-utility signal (in this case, an entry under method_comparison/MetaMathQA/experiments/).
- Paper authors — Ivan Ilin (@vectozavr) is a co-author on the Super-Tuning branch commits per
vectozavr/SuperTuning#3; Zhang & Pilanci (Riemannian LoRA) and Zelenin & Zhuravlyova (Scaling DoRA) are cited in their respective PR bodies and diagram footers. - Outrider drafts — remyx-ai[bot] generated the initial method ports on the fork branches.
- Human review + coordination — @smellslikeml (Terry Rodriguez) shepherded the coordination-issue-first workflow with @BenjaminBossan across all three methods.
All contributions include AI-assistance disclosure per huggingface/peft's CLAUDE.md — commit trailers identify the co-author breakdown, PR bodies link the coordination issues and approval comments.
- Outrider: https://github.com/remyxai/outrider
- PEFT contribution guide: https://huggingface.co/docs/peft/main/en/developer_guides/contributing
- PEFT
CLAUDE.md(AI-assistance policy): https://github.com/huggingface/peft/blob/main/CLAUDE.md - Paper infographics — the three diagrams above were generated with
llmsresearch/paperbanana, an automated academic-illustration tool.