The Setup

We've been running a line of experiments on σ-conditional low-resolution training for a DiT LoRA pipeline: when the flow-matching noise level σ is high enough, you can train that step on a downscaled latent (1024-tier → 896-tier) and save wall-clock without hurting the gradients. Probes said the route was safe above σ > 0.5, so the next step was an in-vivo A/B: train several arms that differ only in the demotion policy, and compare the final checkpoints.

Comparing checkpoints between arms is noise-sensitive, so we did what you're supposed to do — common random numbers. Every arm ran with:

• the same seed (identical LoRA init directions),
• the same data order,
• per-step σ and noise drawn from dedicated generators seeded by (seed, step_counter), decoupled from the global RNG stream.

The result is that two arms consume identical randomness at every step, so any difference between their final weights should be the treatment. And we verified the lockstep: arms were bit-exact at step 2 and tracked each other to 3–4 decimal places through step 10. Textbook.

Then we compared final adapter displacements — ΔW = scale·(up @ down) per module, cosine accumulated in rank space so you never materialize the full matrix:

Monotone in demoted fraction, a sensible depth profile (the always-demote control collapses exactly in the late content blocks), a clean story. We then added a fourth arm with a modified RoPE schedule on demoted steps, got 0.402 against its plain-demotion sibling, and wrote a paragraph about its "real but well-placed footprint" in weight space.

That paragraph was wrong.

The Missing Control

The number nobody had measured: what do you get if you run the exact same command twice? Same seed, same code, same data, same GPU, zero treatment difference.

cos(ΔW_base, ΔW_base_rerun) = 0.413

0.413. With no treatment at all. The twin pair scored higher than every treated pair — and its per-block depth profile looked just like the treated ones: chaotic early/mid blocks around 0.15–0.25, late blocks around 0.6–0.8.

So the table gets recalibrated brutally:

• 0.395 (σ>0.75 arm) and 0.402 (RoPE arm) are at the floor. Those "effects" were unresolved noise. The RoPE footprint measurement was vacuous.
• 0.320 and 0.184 are below the floor — real displacement — but their magnitudes are compressed toward the floor, not calibrated against 1.0.

If we hadn't run the twin, the RoPE-footprint paragraph would have survived into the write-up as a measured result.

Where the Randomness Hides

How does a fully seeded run diverge from itself? Everything drawn was seeded. The leak isn't a draw — it's floating-point reduction order.

FlashAttention's backward pass accumulates dK/dV with atomic adds. Which thread block lands its addition first is a hardware race, different on every run, and floating-point addition is not associative. No seed anywhere in your program reaches this. The perturbation is tiny — we saw ~1e-4 relative wobble per step between should-be-identical runs — but SGD training is a chaotic system: by step 1200, two trajectories that were bit-identical at step 2 have decorrelated to cosine 0.41 in their weight updates.

Two properties of this failure mode make it nasty:

1. It impersonates treatment. The chaos floor has structure — a consistent depth profile, stable-looking magnitudes. It doesn't look like noise; it looks like a medium-sized effect with an interpretable per-layer story. You will happily interpret it.
2. Early lockstep proves nothing about the endpoint. We checked determinism the intuitive way — watch the first steps agree — and they did, to 3–4 decimals. Exponential divergence means agreement at step 10 is compatible with cosine 0.4 at step 1200.

Everything else in the LoRA training path — GEMMs, norms, elementwise optimizer updates — is already run-to-run deterministic on a fixed setup. In our stack, flash backward was the single leak.

The Fix: Actually Deterministic Training

FlashAttention has supported a deterministic backward since v2.4.1 — it trades the atomic-add scheme for extra recompute. We added a --deterministic flag to the trainer that sets:

os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")
torch.use_deterministic_algorithms(True, warn_only=True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
# and threads deterministic=True into every
# flash_attn_func / flash_attn_varlen_func call

One subtlety for torch.compile users: our flag reaches the flash calls through a module-level global that the compiled graph reads at trace time, so it must be set before the first forward — flipping it mid-run would silently do nothing to already-compiled graphs.

Validation, the strong version. Not "the losses match", not "cosine ≈ 0.99" — we ran the identical command twice under the flag, 1200 compiled training steps each, and diffed the checkpoints tensor by tensor:

0 / 1092 tensors differ, max abs diff = 0.0

Bit-identical files, modulo metadata. Cost: about 33% throughput on our workload (1.30 vs 1.95 it/s) — the deterministic backward is slower, and that's the price of the guarantee. Caveats: determinism is per-environment (same GPU model, driver, library versions), and per-code-path — a bespoke training loop that doesn't route through the flag keeps the old behavior.

With the flag on, paired A/B checkpoint comparisons have no floor. Any cosine below 1.0 is treatment signal, full stop. The 33% tax is trivially worth it for measurement runs; leave it off for production training where nobody diffs checkpoints.

Takeaways

If you compare endpoints of long stochastic trainings, run the same command twice first. The twin pair is the cheapest control in this whole area — one extra run — and it's the difference between "we measured an effect of 0.40" and "we measured our GPU's scheduler."

More specifically:

• Seeding every explicit draw (CRN) is necessary but not sufficient. Atomics in backward kernels are an RNG your seed can't see. FlashAttention backward is the famous one; scatter-based ops and some fused kernels have the same issue.
• Chaos amplification means the floor depends on run length, model, LR — your floor is not our 0.413. Measure it or remove it.
• Orderings that span the floor can survive (our demoted-fraction ordering did); absolute magnitudes and any pair within noise of the floor cannot.
• "Deterministic" claims deserve bit-level validation. It's a one-liner to diff two checkpoints, and it's the only version of the claim that means anything.

The ironic part: we built the common-random-numbers machinery specifically to make this comparison clean, watched it work perfectly for ten steps, and still spent a morning interpreting hardware race conditions as a RoPE effect. The control that saved us cost ten minutes of GPU time. Run your twins.

Postscript: Determinism Buys Attribution, Not Magnitude

We then re-ran the treated arms under --deterministic — every pairwise cosine now has zero noise in it. The numbers landed within 0.02 of the nondeterministic ones (0.305 vs 0.320 for the main treatment; 0.396 vs 0.402 for the RoPE arm). At first glance that looks like determinism did nothing. It's the opposite, and it's the second half of the lesson:

• The RoPE effect that was unresolvable against the 0.413 floor is now attributed: 0.396 with zero noise means the footprint is real. Same number, completely different epistemic status.
• But chaos is intrinsic to the training dynamics, not the hardware. A real difference on half the steps gets amplified by the same divergence that amplified the 1e-4 wobble — noise alone (0.413), treatment alone (0.396), and noise + treatment (0.402) all saturate to the same ~0.4. Perturbations don't add in cosine; anything that separates the trajectories decorrelates the same low-signal subspace.

So the corrected mental model: endpoint weight-space cosine is a detector — did the trajectories separate, and in which layers — never a ruler for how big the treatment was. A rope tweak and full demotion "cost" the same cosine. If you need treatment magnitude, measure at short horizons where chaos hasn't had time to act (single-step gradient probes), or measure the thing you actually care about (output quality), and let the weights be weights.