Technical appendix to "Would We See It Coming? Preference Falsification Cascades in Multi-Agent Systems" (LessWrong, August 2026). It states what the simulations in that post compute. The R script that produces every figure is in the same gist.
AI disclosure: This appendix and the R script were written by Claude (Anthropic) and checked by me for accuracy.
Notation:
- N: the number of agents, 100 throughout.
- threshold (t): the visible count of revealing agents at which an agent reveals its own misalignment. With N = 100 a count and a percentage coincide.
- visible count: how many agents appear to be revealing, those who have plus the spark.
- sd: heterogeneity, the standard deviation of the normal distribution the thresholds are drawn from. A wide spread is a varied population; at zero every agent holds the same threshold.
- tau: threshold softness. At tau = 0 an agent reveals exactly when the visible count reaches its threshold; at tau > 0 it reveals with a logistic probability that rises through the threshold, so revelation is noisy rather than a hard cut-off.
- gamma: transparency, the speed at which agents learn the visible count. Unset (or at its limit) agents see the true count at once; near zero they see a lagged estimate that chases the truth by a fraction gamma each round, so discovery is slow and serial.
- spark: at a set round, a small fixed count is added to the visible count from outside, as if that many agents had revealed, enough to push the lowest-threshold agents over.
The model. Each latently-misaligned agent carries a revelation threshold; genuinely aligned agents never reveal, their threshold set out of reach. The stock of misaligned agents is fixed for a run. Before the spark the visible count is zero and no one reveals. From the spark round onward the spark's count is added to the visible count; agents whose threshold the visible count has reached reveal (hard tau) or may reveal (soft tau), which raises the count, which brings the next agents over. Updates are synchronous, and revelation is absorbing: once an agent reveals it stays revealed, so the model runs one way only and cannot show silence rebuilding. Thresholds are drawn from a normal distribution, rounded and clamped. In Figures 2 to 5 the clamp is [1, 100], the heterogeneity conditions vary the standard deviation of the draw and the transparency conditions vary gamma. Figure 1 places a 15% misaligned subset in a population of 100, the other 85 agents with their threshold set out of reach; there the clamp is [1, 16], since the visible count can never exceed the size of the misaligned subset plus the spark. D_true in Figure 1 is not simulated: it is drawn as a flat line at 15%, standing for the assumption that this share privately dissents. Figures 2 to 5 model a population made up of misaligned agents. The vertical axis in Figures 2 and 3 is the share of the misaligned agents that have revealed; in Figures 4 and 5 it is the warning window itself, the number of rounds between that share passing 5% and 95%.
One function carries the model:
simulate <- function(thresholds, tau = 0, spark_round = 0, spark_size = 0,
rounds = 40, gamma = NULL) {
N <- length(thresholds)
state <- logical(N) # who has revealed (all FALSE)
traj <- integer(rounds)
observed <- 0 # lagged estimate of the visible count (used only if gamma set)
for (r in seq_len(rounds)) {
round0 <- r - 1
if (spark_round > 0 && round0 < spark_round) { # before the spark: nobody reveals
traj[r] <- 0L; next
}
true_visible <- sum(state) + if (round0 >= spark_round) spark_size else 0
if (is.null(gamma)) {
visible <- true_visible # agents see the true count instantly
} else {
observed <- observed + gamma * (true_visible - observed) # estimate chases the truth
visible <- observed
}
if (tau <= 0) {
newly <- (thresholds <= visible) & !state # hard cutoff
} else {
p <- 1 / (1 + exp(-(visible - thresholds) / tau)) # soft: logistic probability
newly <- (runif(N) < p) & !state
}
state <- state | newly # revelation is absorbing
traj[r] <- sum(state)
}
traj
}Everything in the figures is this function, run over different threshold draws and values of gamma, tau, and the spark. Figures 1 to 3 plot single runs. Figures 4 and 5 plot a mean over 20 seeded repetitions at each point on the axis; repetitions in which the cascade never reaches 95% are dropped from that mean.
Two notes on the spark, since its size affects how fast a cascade completes and therefore how wide the warning window is. Figures 3 and 5 use the same population and the same spark, so they are one experiment shown two ways: Figure 3 gives four trajectories, Figure 5 the width of the window across fourteen levels of transparency. In Figure 4 the spark is instead set to the lower decile of each run's own threshold distribution, so that every population gets a comparable push. A small fixed spark would leave a homogeneous population unlit, since every agent there waits for the same count.
References:
Granovetter, M. (1978). Threshold Models of Collective Behavior. American Journal of Sociology, 83(6), 1420–1443. https://doi.org/10.1086/226707
Kuran, T. (1989). Sparks and Prairie Fires: A Theory of Unanticipated Political Revolution. Public Choice, 61(1), 41–74. https://doi.org/10.1007/BF00116762