Previously

Last time, we found that two runs of the identical training command end at checkpoint cosine 0.41 — FlashAttention's backward accumulates with atomic adds, no seed reaches the scheduling race, and 1200 steps of SGD chaos amplify a ~1e-4 wobble into full decorrelation. We fixed it with a --deterministic flag, validated it the strong way (two runs, 0 / 1092 tensors differ), and closed with a confident sentence: with the flag on, any cosine below 1.0 is treatment signal, full stop.

This post is about the loophole in "full stop." We even wrote the loophole down ourselves — in the flag's own --help text — and then walked into it anyway.

The Setup

The same experiment line has since moved from training A/Bs to mechanism probes. Instead of comparing endpoint checkpoints, we measure pooled gradient directions at fixed noise levels σ: 40 images × 12 noise draws per σ-bin, debiased, pooled into a handful of direction vectors per bin. A full probe run costs a few GPU-hours, so the natural workflow is measure once, commit the store, reuse it forever — every later experiment compares its fresh vectors against the committed reference table.

On August 7 we committed two such native reference stores. They're protocol twins of each other and agree at cosine 0.999 — the estimand is solid, cross-run, replicated.

On August 9 we launched a new probe with an intervention: pin the σ fed to the network's conditioning pathway while the actual noising σ sweeps a grid. The pre-registration included a validation gate that is free by construction: at the one bin where the pinned σ equals the noising σ, the intervention is a no-op, so the new run must match the committed native reference there at |cos| ≥ 0.95. If a no-op doesn't read as a no-op, the instrument is broken and you read nothing else.

Everything ran with --deterministic. Every draw seeded. Protocol bit-matched to the reference runs.

Gate read, against a required 0.95 (and a committed native↔native baseline of 0.999):

cos = 0.33 / 0.43 / 0.43   (three direction estimands)

Blaming the Instrument (Wrongly)

A brand-new flag plus a failed validation gate has one obvious reading: the flag is broken. It claimed to be a one-argument swap at one call site; clearly it perturbed something else — the noising path, the RoPE handle, a seed stream.

Before rewriting the flag, we ran the cheap bisection: one native run (no flag at all) at the gate bin, in the current boot, on seed-independent draws. That splits the comparison into two pairs, and the two pairs disagree violently:

The flag is innocent — pinning the conditioning at its own bin is the no-op it was designed to be. The failure is the other pair: a native run in today's environment cannot reproduce the committed native stores. The reference table itself had died, and every run made after the break would have failed the gate, flag or no flag.

The Forensics

What changed between August 7 and August 10? We checked the entire dependency chain: adapter checkpoint, DiT weights, VAE, latent caches, text-embedding caches — all byte-identical and older than the reference stores. torch 2.12.0+cu132: unchanged. Driver 610.43.02: unchanged. Package history: clean.

The only events left standing: the machine rebooted twice on the evening of August 9, and /tmp — where torch.compile's autotune cache lived — came back empty. On the next run, the compiler re-benchmarked its kernel candidates from scratch and picked a fresh set of winners.

And the magnitude of the break — 0.32 to 0.47 — is exactly the chaos floor from the last post (0.413).

Why --deterministic Didn't Save Us

Because determinism was never a property of your program. It's a property of your program plus the kernel path it runs on, and the kernel path is chosen by an autotuner that benchmarks candidates with wall-clock timings and caches the winners in a directory that evaporates on reboot.

Walk the chain:

1. torch.compile autotunes: for each shape, it times several candidate kernels and caches the fastest.
2. Timing is noisy, so a re-autotune from cold can crown different winners.
3. Different kernels means different reduction orders, and floating-point addition is not associative — the two paths disagree at ~1e-4.
4. Each path is internally, perfectly deterministic. Run the command twice in the same boot: bit-identical. Run it across the cache wipe: a different deterministic answer.

--deterministic pins everything within a kernel path. It cannot pin which path you're on.

Note what's absent from this failure compared to the last post: chaos amplification. These aren't 1200-step training trajectories — they're single forward/backward probes, averaged over draws. The wobble doesn't need a thousand steps to grow, because the estimand itself is small: a debiased pooled direction is the residual left after subtracting shared structure, and a low-signal residual is exactly where a 1e-4 systematic shift dominates. Worse, averaging can't rescue you — 12 draws per bin, 40 images pooled, and the break still reads 0.32–0.47, because a kernel-path shift is not draw noise. It's a bias, identical across every draw taken in that environment. Averaging shrinks noise; it does nothing to a bias.

The bitter footnote: the flag's help text, written by us in July, warned that a cold compile cache "can still shift results." We wrote the warning, shipped it, rebooted, and then spent a diagnosis run discovering our own caveat, now measured at 0.32–0.47 on pooled debiased legs instead of hypothesized.

Three Nasty Properties

1. Reboots don't reliably break it. The machine also rebooted between August 7 and 8, and cross-run comparisons spanning those reboots worked at 0.99 — that time, the cache (or its winners) survived. So you can't audit by calendar; "no reboot since" is not a validity certificate, and "there was a reboot" is not a death certificate. The only thing that counts is a measured same-environment control.

2. Nothing inside a path warns you. Every same-boot reading is 0.96–0.999. The new store passes its internal validation suite. The break is invisible until a cross-boot comparison happens to have a known correct answer — our gate did, purely by pre-registered luck. Without it, we'd have read 0.33 as a spectacular intervention effect and written a paragraph about it. (Sound familiar? Last post, the chaos floor impersonated a treatment. This time, the boot break impersonated one.)

3. It impersonates instrument failure. The proximate symptom was "new flag fails its gate," and the prior on "my new code is broken" is correctly high. We came close to debugging a correct one-line flag against a dead reference table. The bisection that resolved it — one extra native run in the current boot — cost an hour and should have been the first move, not the recovery move.

The Fix: Seed Twins

Two designs survive this wall:

• (a) Re-measure the reference in the current environment. Same protocol, new boot, then compare within-environment.
• (b) Seed-twin arms — run the reference inside the same experiment, same seed, so both arms consume identical noise draws. The shared draw noise cancels exactly in every comparison. This is strictly stronger than the original committed-store design, which had independent draw noise on top of everything else.

We took (b): one native twin of the full grid, 3.8 hours, same boot verified. The gate re-read against it: pass — 0.95 / 1.09 / 0.97 (a debiased estimator, so slightly exceeding 1 is in-spec; the committed baseline read 1.02 the same way). The experiment proceeded, and the actual scientific result — the intervention's effect across the σ grid — was read cleanly against the twin.

Worth stressing what didn't break: every within-store estimand — relative gates, ratios, geometry across σ inside one run — is untouched. The wall only exists for absolute vector comparisons that cross an environment boundary.

Takeaways

A committed reference number has an environment silently attached to it. GPU model, driver, library versions — and the compiled-kernel cache, which lives in /tmp and dies with the boot. Treat any cross-environment vector comparison as invalid until a same-environment control proves otherwise.

More specifically:

• "Deterministic" means repeatable within a kernel path, and the kernel path is state, not code. If your autotune cache sits in tmpfs, your determinism guarantee has a reboot-shaped hole. Persisting the cache (TORCHINDUCTOR_CACHE_DIR pointed somewhere durable) narrows the hole but doesn't close it — version bumps and cache invalidations re-roll the dice too.
• Averaging doesn't help. A kernel-path shift is a bias shared by every sample measured in that environment, not noise. Our break survived 12 draws × 40 images of pooling at full strength.
• Design estimands to be same-run relative: seed twins, within-run ratios, paired arms in one submission. Absolute directions don't travel; differences and ratios do.
• Build no-op gates. Our pre-registered "this bin must be a no-op" check is the only reason the break was caught before an exciting-looking 0.33 became a result. A validation gate with a known correct answer is free insurance against failure modes you haven't imagined yet.
• When a gate fails right after you add code, still run the environment control first. One native run in the current boot separates "my code is wrong" from "my reference is dead" — an hour well spent before a debugging spiral.

Postscript: a Two-Level Taxonomy

The two posts are really one lesson at two scales. Within an environment, unseedable scheduling races decorrelate long trajectories — twins measure it, --deterministic removes it. Between environments, the kernel path itself shifts — no flag removes it, and only same-environment references or seed-twin designs are valid across it.

Both failure modes produce numbers with structure — consistent depth profiles, stable magnitudes, plausible stories. Both impersonate results. And both are caught by the same discipline: never read an absolute number without a control that shares everything except the thing you're measuring. Last post that control was a twin run. This post it was a twin environment. The GPU keeps inventing new confounders; the control keeps being the answer.