VIDRAFT_LAB

Sharing the first experiment you asked for.

Experiment 1 (compounding check). One chip, no synchronization, 200 steps, identical seed and identical batches. I compared per-step training loss between the NPU run (our manual backward pass) and a reference run of the same model and batches on CPU with fp32 autograd. Weights are expected to diverge, so the loss curve is the fair comparison.

Per-step loss (CPU fp32 / NPU):

step 0: 5.674 / 5.677
step 50: 2.240 / 2.227
step 100: 2.026 / 2.019
step 150: 1.580 / 1.557
step 199: 1.499 / 1.465

Mean absolute gap 0.012, max 0.037. The NPU curve is not above fp32 (it ends slightly below), and the drift stays tiny.

So over 200 steps the NPU manual backward tracks exact fp32 closely. The final gap is therefore not due to NPU arithmetic.

Experiment 2 (the synchronization-schedule control) is being measured separately with a fair apples-to-apples setup, and I will post those numbers once it is complete.

Thanks for the careful read.

Good suggestions, thanks. From the public comparison table, paired loss deltas (NPU minus GPU) are 0.03 at step 0, a maximum of 0.38 at step 100, and 0.17 at the end, with a mean of 0.18 across the matched steps. Relative to the GPU loss the gap grows, from 0.5% at step 0 to 68% at the end. These curves do not share the same learning dynamics, and we do not claim they do.

That gap comes from the 32-chip run’s sync scheme: parameters were averaged every few steps (DiLoCo style) instead of an all-reduce every step. Per-step arithmetic is a separate question, and we measured it directly in the thread above: on one chip with no sync, the NPU gradient is closer to fp32 than a CPU bf16 control on all 110 parameter tensors.

Thanks, the distinction is fair. The two numbers in my reply came from different checks: 6.6e-7 is a CPU fp32 check of our backward math (2-layer configuration), and the 0.9999 cosine is from an earlier LoRA block test. Neither was the per-step NPU measurement on this model, so we ran exactly the test you described.

46M model, 12 layers, one chip, one step, no sync, real TinyStories text, against CPU fp32 autograd, all 110 parameter tensors: minimum cosine 0.99978, relative L2 error median 0.96%, worst 2.21% (layer 9 Wq), worst max|dW| relative 2.56%.

Control on the same batch, CPU bf16 autograd vs fp32: median 1.67%, worst 3.42%. The NPU gradient is closer to fp32 than bf16 on all 110 tensors. So you are right that it is not “essentially exact”; it is reduced-precision arithmetic, and tighter than a standard bf16 training path. The chip’s internal precision is being confirmed, and the GPU run on the same DiLoCo schedule will follow.

You’re right that Pearson on two falling curves is a weak lead, and thanks for the careful read. We published the raw NPU-vs-GPU comparison precisely so it could be checked like this, and we stated the final-loss gap (0.42 vs 0.25) and attributed it to the reduced-communication (DiLoCo) sync in that same file. We did not hide it.

Two separate things got conflated in the post, and your critique separates them correctly:

1. Per-step numerical fidelity of the NPU backward — this is the “numerically the same work” claim, and it lives at single-chip, single-step, no sync. For this 46M model, our manual backward matches autograd to a worst-case relative gradient error of 6.6e-7 (essentially exact), and the NPU-computed gradients match the standard reference at cosine ~0.9999. That is the number we should have led with.

2. Multi-chip convergence — the 32-chip run used DiLoCo-style reduced-communication sync, which is deliberately NOT a per-step all-reduce. You are correct that under DiLoCo the two runs are not applying identical updates. That is exactly why the gap grows, and it is a property of the sync scheme, not of the NPU backward.

So: per-step backward = numerically faithful (6.6e-7 / cos ~0.9999); multi-chip convergence = approximate by design. We will post the exact comparison you describe — same seed, same batch, one chip, one step, no sync, gradient cosine and max|dW| per layer vs GPU, plus a GPU run on the same DiLoCo scheme for an apples-to-apples convergence curve. Appreciate you pushing on this.