Skip to content

Instantly share code, notes, and snippets.

@pukpr
Created September 2, 2026 21:24
Show Gist options
  • Select an option

  • Save pukpr/5ade9e32d69c484029660a66d5d8d659 to your computer and use it in GitHub Desktop.

Select an option

Save pukpr/5ade9e32d69c484029660a66d5d8d659 to your computer and use it in GitHub Desktop.
Evaluating DOF and overfitting, Follow on to https://gist.github.com/pukpr/05687a46cf1a5b76b2ba6ae589343172

LTE/AMO Fitting Parameters — Objective DOF Characterization

Source: lt.exe.p / lt.exe.resp, AMO forward pass, as unwrapped in lte_forward.py. Each parameter is classified by what it mechanistically claims versus what role it plays in the fit, so that "degrees of freedom" isn't reported as a single undifferentiated count.

1. Externally anchored — not free relative to the AMO target

Parameter Role Basis
lpap amplitude/phase (42×2 = 84 values) Tidal constituent amplitude/phase feeding tide_sum Calibrated against LOD, independent of the AMO fit. Zero effective DOF for this target — this is an external forcing measurement carried in, not a parameter search.
lpap periods (42) Constituent periods Fixed by the Doodson formula from lunar/solar astronomical constants (DRACONIC, TROPICAL, ANOMALISTIC) — not fit at all, structural.
year Tropical-year length correction inside Year_Adjustment Free parameter, but optimizer converged to 2.64×10⁻⁹ — effectively pinned to the astronomical year length. Negligible effective DOF.

2. Mechanistic — encode the tidal-forcing hypothesis itself

Parameter Role Physical claim
delB Calendar month of the annual impulse Discrete, physically constrained (0–11) — which phase of the annual cycle gates the forcing.
delA, asym Impulse magnitude at delB and delB+6mo Continuous, but tied to those two physically fixed calendar slots, not a free phase.
ma, mp IIR memory decay / asymmetric dead-band Recharge-discharge persistence timescale and nonlinear (Coulomb-type) damping — the memory/hysteresis claim.
shfT Tide-vs-derivative admittance mixing in tide_sum Whether the system responds to tidal potential or tidal rate-of-change — same physical question as JERK.
offs, bg, impA, impB Bessel first/second-harmonic modulation coefficients The nonlinear (Duffing-like) self-referential term — v appears on both sides of the modulation — this is the amplifier/resonance nonlinearity itself.
ltep (6 values) Winding-number / resonance-order parameters Dimensionless, not temporal — they multiply forcing(t), not t. They set how the response phase winds per unit excursion of the already-tidally-structured forcing, i.e. the resonance-selection (circle-map / Arnold-tongue) step. No period should be attached to them; the timescale content lives upstream in lpap and the impulse comb.
amp[j], phase[j] (×6, 12 values) OLS-fit response magnitude/phase per ltep mode Linear, closed-form given the frequencies above — these are the response, not a search parameter. Drop-one contribution to training CC: mode 1 (Δ0.165) and mode 0 (Δ0.071) are load-bearing; mode 4 (Δ0.065) moderate; modes 2,3,5 marginal (Δ<0.03 each) — three of six winding numbers carry most of the mechanistic content.

3. Nuisance regressors — not part of the tidal mechanism

Parameter Role What it's actually absorbing
trend, accel Linear + quadratic terms in calendar date Secular warming and its curvature — i.e., an AGW-type nuisance term, explicitly separated from the oscillatory forcing so the tidal terms aren't forced to fit a monotonic drift they don't produce.
ann1, ann2, sem1, sem2 Direct annual/semiannual harmonics of calendar date The AMO's ordinary seasonal cycle, fit independently of the tidal impulse comb. This is not the same annual mechanism as delA/delB/asym — those gate the tidal forcing at a calendar phase; these four directly absorb whatever seasonal signal remains, seasonal-nuisance removal rather than a mechanism claim.
level Constant (DC) offset Zero effect on correlation. Pearson CC is invariant to an additive constant on either series — level only sets the mean, which cancels in the CC computation entirely. It's necessary for the model to sit on the right baseline, but it contributes nothing to, and can't be credited for, the fit skill.
k0 Linear gain on forcing in the regression A first-order (non-resonant) linear response term, alongside the six nonlinear resonant modes — a "did a plain linear response already explain some of this" control term rather than a tidal-mechanism claim.

4. Structural / initial-condition — required, not a mechanism claim

Parameter Role
init IIR integrator seed at IDATE=1880. An initial condition is always required for a recursive filter; its influence decays with ma over the 143-year record. Free, but not evidence for or against the tidal hypothesis — every integrator needs a starting value.

5. Dead / unused in this configuration

Parameter Status
harm ([94]) NH="" in lt.exe.resp → harms=[] in code → this value is never read. Not a live DOF here.
impC Declared in the params dict but never referenced anywhere in forward() (not in tide_sum, impulse_delta, iir, or bessel). Dead in this code path.
IR Set to 0.0 → the inter-annual self-differencing branch (model[i] -= IR·model[i-12]) is skipped entirely. Zero DOF cost as configured. If ever turned on, note it would be the one term with no tidal-physics anchor at all — it's the index's own year-over-year memory, not forcing.

Objective summary

  • Externally anchored, zero effective DOF against AMO: 84 (lpap amp/phase) + year (≈0).
  • Mechanistic tidal-forcing DOF: 9 nonlinear (delA, asym, ma, mp, shfT, offs, bg, impA, impB) + 6 dimensionless winding numbers (ltep) + 12 linear responses (amp/phase) ≈ 27, of which the drop-one test shows only 3 of the 6 ltep modes are doing most of the work.
  • Nuisance/non-mechanistic regressors: 6 (trend, accel, ann1, ann2, sem1, sem2) — explicitly separating AGW-trend and ordinary seasonal cycle from the tidal claim, which is the methodologically correct way to keep those effects from contaminating the mechanism test, provided the paper is explicit that these are nuisance removal and not part of what's being validated.
  • Structural, not mechanism-bearing: level (zero CC impact), k0 (linear-response control), init (required IC).
  • Dead in this run: harm, impC, IR — 0 effective DOF, should be dropped from any published parameter count rather than left implicit.

The honest framing for a skeptical reader: the tidal-mechanism claim rests on roughly 27 parameters (dominated in practice by ~3–4 of them), sitting alongside a smaller, clearly-labeled set of nuisance and structural terms that are doing acknowledged non-tidal work. That separation — rather than a single "N parameters, r=X" headline — is what makes the DOF accounting defensible under scrutiny.

Naive Fourier Series — Objective DOF Characterization (contrast case)

Basis: global Fourier series, fundamental period = full record span (T ≈ 142.6 yr), harmonics k = 1..N, fit by OLS on the same EXCLUDE=TRUE train/holdout split used for the LTE/AMO model (TRAIN_START/TRAIN_END excluded from fitting, correlation reported on both the training region and the excluded window separately).

1. Parameter inventory

Parameter set Count (at N harmonics) Role
level (constant) 1 DC offset — same zero-CC-impact caveat as the LTE model's level.
k0 (linear-in-date trend) 1 Secular/AGW-type nuisance term, analogous to LTE's trend.
sin(2πk t/T), cos(2πk t/T) for k=1..N 2N The entire model. Frequencies are fixed a priori (integer multiples of 1/T) — zero nonlinear search DOF, all 2N are linear OLS coefficients.

Total: 2N + 2. At N=60 (the order needed to approach the tidal model's fast-cycle fit, see §3): 122 parameters, all linear, none anchored to any external physical measurement.

2. What's mechanistic vs nuisance here

There is no mechanistic category for this model — that's the point of it as a contrast case. Every coefficient is a bare Fourier response with no claim about why AMO should have that harmonic content. This is the opposite end of the spectrum from the LTE model's lpap (externally LOD-anchored) and ltep (physically motivated winding numbers):

LTE model Naive Fourier
Frequencies anchored to Doodson/LOD astronomy Frequencies anchored to nothing but 1/(record length)
~15 nonlinear + ~20 linear ≈ 35 total DOF 0 nonlinear, 2N+2 linear DOF (N chosen post hoc)
3 of 6 winding-number modes carry most of the mechanistic content No mode is privileged; skill is spread thinly and must be bought with harmonic count

3. Fast-cycle (fine-structure) fit: naive Fourier is capable, at a cost

Isolating the fast-cycle component (data and model both high-pass filtered against a 5-year running mean, so year-to-year wiggle is compared independent of trend/decadal shape) on the training region:

Model N params Full-signal CC (train) Fast-cycle-only CC (train)
LTE (tidal) ~35 0.86 0.62
Fourier, N=6 14 0.72 0.03
Fourier, N=30 62 0.84 0.41
Fourier, N=60 122 0.91 0.64
Fourier, N=100 202 0.94 -0.01 (ill-conditioned, collapses)

This is the honest version of the point: a naive Fourier series can match the tidal model's ability to track fast/fine-scale wiggle in-sample, but only once harmonic count reaches ~60 (122 parameters, roughly 3.5× the tidal model's effective DOF) — and pushing further (N=100) doesn't improve it, it destroys it, because the design matrix becomes poorly conditioned once harmonic periods approach the ~1-month sampling interval and adjacent columns become nearly collinear.

4. Where naive Fourier fails regardless of order: out-of-sample skill

The in-sample fast-cycle result above says nothing about generalization. The walk-forward test (13 six-year holdout windows stepped across 1885–2017) already established this:

Model mean holdout CC std
LTE (tidal) 0.68 0.10
Fourier N=2 -0.11 0.34
Fourier N=60 -0.16 0.33

No harmonic order tested — from 2 to 60 — produces stable positive holdout skill. The N=60 fit that matches the tidal model's in-sample fast-cycle CC is simultaneously among the worst out-of-sample performers. That's the core asymmetry: naive Fourier can be made to track fine structure it has already seen, at high parameter cost, but that capacity doesn't transfer to structure it hasn't seen, at any order — whereas the tidal model's much smaller DOF budget holds up consistently across every window tested.

Objective summary

  • Naive Fourier needs ~60 harmonics / 122 parameters to match the tidal model's in-sample fast-cycle correlation (~0.62–0.64) — roughly 3.5× the DOF for a comparable in-sample-only result.
  • That parity is illusory for generalization: at every order tested, naive Fourier's holdout CC hovers near zero with high variance, while the tidal model's holdout CC is stably ~0.68.
  • The contrast isn't "tidal model uses fewer parameters for the same skill" — it's that raw frequency count (naive Fourier) buys in-sample flexibility but not predictive skill, while frequency placement anchored to external astronomy (tidal model) buys both, with far fewer parameters and without needing to be pushed to the edge of numerical conditioning to do it.
#!/usr/bin/env python3
"""plot_amo_lte_vs_fourier.py
Plots the AMO index alongside (a) the LTE tidal-forcing model and
(b) a naive global Fourier series (N=60 harmonics), with the rolling
holdout windows shaded and each window's out-of-sample correlation (r)
annotated for both models.
Requires lte_forward.py and the amo/ run directory (amo.dat, lt.exe.p,
lt.exe.resp) alongside this script, matching the layout used in the
LTE gist (github.com/pukpr/05687a46cf1a5b76b2ba6ae589343172).
Usage:
python3 plot_amo_lte_vs_fourier.py
"""
import json
import os
import sys
from pathlib import Path
import numpy as np
import matplotlib.pyplot as plt
sys.path.insert(0, str(Path(__file__).resolve().parent))
from lte_forward import Config, read_resp, forward, cc # noqa: E402
RUN_DIR = Path(__file__).resolve().parent / "amo"
FOURIER_N = 60 # harmonic order for the naive comparison model
WINDOW_LEN = 6 # years per holdout window
WINDOW_STEP = 10 # years between window starts
WINDOW_START0 = 1885
def fourier_fit(dates, data, sel, n_harm):
"""OLS fit on a global Fourier basis (fundamental = full record span),
trained on `sel`, returns the fitted model over all dates."""
n = len(dates)
T = dates[-1] - dates[0]
cols = [np.ones(n), dates]
for k in range(1, n_harm + 1):
cols.append(np.sin(2 * np.pi * k * dates / T))
cols.append(np.cos(2 * np.pi * k * dates / T))
A = np.column_stack(cols)
coef, *_ = np.linalg.lstsq(A[sel], data[sel], rcond=None)
return A @ coef
def main():
os.chdir(RUN_DIR)
resp = read_resp(Path("lt.exe.resp"))
resp.setdefault("CLIMATE_INDEX", "amo.dat")
params = json.loads(open("lt.exe.p").read())
raw = np.loadtxt("amo.dat")
dates_full = raw[:, 0]
starts = np.arange(WINDOW_START0, dates_full[-1] - WINDOW_LEN, WINDOW_STEP)
# ---- one panel per holdout window, each showing that window's own refit ----
n_windows = len(starts)
ncols = 4
nrows = int(np.ceil(n_windows / ncols))
fig, axes = plt.subplots(nrows, ncols, figsize=(4.2 * ncols, 3.0 * nrows),
sharey=True)
axes = np.atleast_1d(axes).ravel()
zoom_pad = 10 # years shown on either side of each holdout window
window_stats = []
for i, ts in enumerate(starts):
te = ts + WINDOW_LEN
r = forward(Config(resp, {"TRAIN_START": str(ts), "TRAIN_END": str(te)}),
params)
dates = r["dates"]
data = r["data"]
hold = r["hold"]
sel = r["sel"]
lte_model = r["model"]
lte_r = r["cc_holdout"]
four_model = fourier_fit(dates, data, sel, FOURIER_N)
four_r = cc(four_model[hold], data[hold], dates[hold], -1e30, 1e30) \
if hold.sum() > 2 else float("nan")
window_stats.append((ts, te, lte_r, four_r))
ax = axes[i]
lo, hi = ts - zoom_pad, te + zoom_pad
mask = (dates >= lo) & (dates <= hi)
ax.plot(dates[mask], data[mask], color="black", lw=1.1, label="AMO data")
ax.plot(dates[mask], lte_model[mask], color="tab:blue", lw=1.3,
label="LTE (refit, this holdout)")
ax.plot(dates[mask], four_model[mask], color="tab:red", lw=1.0, alpha=0.85,
label=f"Fourier N={FOURIER_N} (refit, this holdout)")
ax.axvspan(ts, te, color="grey", alpha=0.22)
ax.set_title(f"{ts:.0f}-{te:.0f} | LTE r={lte_r:.2f} Four r={four_r:.2f}",
fontsize=9)
ax.set_xlim(lo, hi)
ax.set_ylim(-7, 8)
ax.tick_params(labelsize=7)
for j in range(n_windows, len(axes)):
axes[j].axis("off")
handles, labels = axes[0].get_legend_handles_labels()
fig.legend(handles, labels, loc="upper center", ncol=3, fontsize=9,
bbox_to_anchor=(0.5, 1.02))
fig.suptitle("AMO: per-window holdout refit, LTE tidal model vs. naive Fourier (N=60)",
fontsize=12, y=1.05)
fig.tight_layout()
out_path = Path(__file__).resolve().parent / "amo_lte_vs_fourier.png"
fig.savefig(out_path, dpi=150, bbox_inches="tight")
print(f"saved: {out_path}")
print("\nper-window holdout r (each row = independently refit for that holdout):")
print(f"{'window':>12}{'LTE r':>10}{'Fourier r':>12}")
for ts, te, lte_r, four_r in window_stats:
print(f"{ts:.0f}-{te:.0f}".rjust(12) + f"{lte_r:>10.3f}{four_r:>12.3f}")
if __name__ == "__main__":
main()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment