| --- |
| license: mit |
| tags: |
| - distributed-training |
| - old-hardware |
| - int8 |
| - webgpu |
| - webrtc |
| --- |
| |
| # πΌ DaisyChain-Train β Old Hardware Training Pipeline |
|
|
| **Part of DaisyChain on π€ Hugging Face β https://huggingface.co/DaisyChainAI** |
| Model page (weights + card): https://huggingface.co/DaisyChainAI/DaisyChain-Train |
|
|
| --- |
|
|
| > **In plain terms:** DaisyChain-Train lets you use **old / spare machines** to |
| > train neural networks. The training runs through **emulated GPU logic** β |
| > verified INT8 units (GUDA-style) that stand in for a GPU's math β so machines |
| > *without* a modern GPU can still do the work. Chain several together and they |
| > train one shared model as a cluster. |
| > Before you rely on it, see what it **can't** do β [Limitations](docs/LIMITS.md). |
|
|
| **Use the hardware you already have to train.** Each machine runs the emulated |
| GPU logic (verified INT8 units β multiply / requantize / ReLU) to compute the |
| model, and DaisyChain pools the machines data-parallel: device selection, |
| capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard. |
| Two ways to run β **Docker** or **Python**. |
|
|
| --- |
|
|
| ## β οΈ Read this first |
| DaisyChain-Train is for **small models on spare hardware**. It **pools compute, |
| not memory** (the model must fit on one machine), scaling is **sublinear**, and |
| it is **not** a substitute for a real GPU on large models. Full envelope in |
| **[docs/LIMITS.md](docs/LIMITS.md)** β please read it before relying on it. |
|
|
| --- |
|
|
| ## Feature list |
|
|
| ### Python cluster trainer (`daisychain/`) |
| - **Data-parallel training across mixed machines** β each node trains its own |
| shard; gradients combine into the exact full-batch gradient, replicas stay |
| bit-identical. |
| - **Capacity-weighted sharding** β faster machines automatically take a |
| bigger share of the batch. |
| - **Emulated GPU compute (verified INT8 units)** β `VerifiedLinear` layers |
| run every forward multiply / requantize / ReLU through the bundled trained |
| units; cluster-wide unit-invocation counts printed by rank 0. |
| - **Bring your own model** β any `Task` (`build_model` / `sample` / `loss`) |
| via `DAISY_TASK`; template in `examples/my_task_template.py`. |
| - **Plain-float alternative task** β same cluster and pooling with ordinary |
| float math. |
| - **Live dashboard** (`daisychain-dashboard`) β readiness banner, P2P |
| connectivity scan, pooled cores/RAM, per-node capacity plan, live loss. |
| - **SpikeWhale control panel** (`spikewhale_panel`, `localhost:8899`) β |
| sliders for model size / training settings, any HF dataset you can access |
| (default streamed FineWeb-Edu), start/stop/re-adjust, live loss. |
| - **Docker demo cluster** β 3 nodes + dashboard in one command. |
| - **Windows helper** (`scripts\setup.bat`) and Tailscale mesh guide. |
|
|
| ### DaisyChain-Web (`web/`) β browser P2P training |
| - **Zero-install nodes** β opening the page IS joining; devices on one |
| network auto-group (Snapdrop-style, by public IP). |
| - **Private cross-network rooms** β `?room=CODE` with **host approval** for |
| every join. |
| - **Full WebRTC mesh** β gradients travel peer-to-peer; the server only |
| signals and serves static files, it never sees weights or gradients. |
| - **Leader-follower runs** β whoever presses Start sets width / sequence / |
| batch-per-device / steps / learning rate for the whole group; config |
| broadcast on the wire. |
| - **Mid-run join** β late devices are synced in (weights + step) and |
| contribute from the next step. |
| - **Bit-identical replicas** β same seeded init, strict roster-order gradient |
| averaging, deterministic Adam with identical state on every peer; verified |
| live by per-step weight hashes. |
| - **Sync guard** β any weight-hash mismatch stops the run instead of |
| training past a fork; the step roster forbids silent partial averages. |
| - **Gradient repair** β a follower missing a roster gradient re-requests it |
| from the leader (8 steps retained), bit-exact, and the run continues. |
| - **Cross-device kernel probe** β every step, every device re-hashes a fixed |
| seeded int8 GEMM through its live kernel; catches broken arithmetic that |
| weight hashes cannot see. |
| - **Hardcoded FineWeb-Edu streaming** β the server reads random slices of |
| the 10BT parquet shards straight off the HF CDN via HTTP range requests |
| (pure-JS `hyparquet`); built-in corpus fallback offline. |
| - **Checkpoints** β download `.pt`, upload β **broadcast to the whole |
| group**; validated (magic, dims, tokenizer vocab) before accepting. |
| - **Inference kit** β one self-contained HTML file with the trained weights |
| baked in; generations offline, anywhere. |
| - **In-page generation** β prompt box on the trained model. |
| - **Old-hardware tier** β no WebGPU? The identical units run on CPU (same |
| bits, so CPU and GPU devices co-train in one group). There is no |
| plain-float path. |
| - **Large-message fragmentation** β multi-MB gradients/checkpoints chunked |
| at 48 KB over the data channels. |
|
|
| ### Verified compute & kernels (web) |
| - **Verified INT8 units everywhere** β block-scaled int8 GEMM: exact LUT |
| products, exact int32 accumulation, bit-exact f32 epilogue with a pinned |
| rounding schedule; scales derived in JS f64 (division never runs on GPU). |
| - **Backends, best-first** β DP4A hardware int8 dot β LUT compute shader β |
| CPU mirror; every kernel **exact-gated at init** (bit-level compares) and |
| demoted to the mirror on any mismatch. |
| - **Continuous random-cell audit** at live training shapes. |
| - **Fused attention kernels** β gather/scatter head-strided qΒ·kα΅ and aΒ·v |
| straight from BTΓC layout (CUTLASS ex. 36/52 style). |
| - **QKV dual-GEMM fusion** β shared left operand quantized once, one batch-3 |
| dispatch (ex. 45); bit-identical. |
| - **B2B MLP chain** β both MLP GEMMs back-to-back on GPU with fused per-row |
| absmax reduction and on-device quantize (ex. 13 + 23); WGSL-exact respec |
| with a fround-stepped JS mirror; fma-contraction-immune by construction. |
| - **Dispatch-optimized backward** β overlapped independent GEMMs, batch-3 |
| sibling fusions (ex. 05/24); bit-identical gradients; optional int8 STE |
| backward path (dormant, 1.21Γ vs float). |
|
|
| ### Verification stack (web) |
| - **Exact init gates** on every kernel, every device, every boot β including |
| gates that "gate the gate" with discriminating boundary inputs. |
| - **IEEE-754 binary32 oracle** in exact BigInt arithmetic β proves the JS |
| epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs). |
| - **Metamorphic property suite** β reference-free relations + definitional |
| absolutes; **4/4** on an externally-authored bug corpus, matching the |
| differential gate. |
| - **RDNA2 ISA audit hardenings** β bit-level (β0-aware) gate comparisons; |
| proof that FMA contraction cannot change the quantize. |
| - **Twelve-suite test chain** (`cd web && npm test`) β convergence, replicas, |
| oracle, gates, properties, external corpus, **self-corpus** (the instruments |
| scored against my own bugs), B2B, optimizer, transformer LM, int8 backward, |
| non-finite quantize refusal; |
| results in [web/TEST_RESULTS.md](web/TEST_RESULTS.md). |
| - **Dirty-buffer gate** β the pool is poisoned before a re-sweep so state bugs |
| (a kernel assuming zeroed memory) are caught deterministically rather than |
| by ordering luck. |
|
|
| ### Documentation |
| - Python: [QUICKSTART](docs/QUICKSTART.md), [LIMITS](docs/LIMITS.md), |
| [CUSTOM_TASK](docs/CUSTOM_TASK.md), [TAILSCALE](docs/TAILSCALE.md). |
| - Web: [Getting started](web/docs/GETTING_STARTED.md), |
| [Architecture](web/docs/ARCHITECTURE.md), |
| [Verification](web/docs/VERIFICATION.md), |
| [Troubleshooting](web/docs/TROUBLESHOOTING.md). |
|
|
| --- |
|
|
| ## Quick start |
|
|
| ### Docker (most reliable β one command) |
| ```bash |
| docker compose -f docker/docker-compose.yml up --build |
| # open http://localhost:8080 |
| ``` |
| Brings up a 3-node demo cluster + dashboard on one machine. |
|
|
| ### Python (real machines) |
| On every machine (`pip install -e .`): |
| ```bash |
| export MASTER_ADDR=100.101.102.10 # coordinator IP (Tailscale 100.x recommended) |
| export MASTER_PORT=29560 |
| export WORLD_SIZE=3 |
| export RANK=0 # 1, 2, ... on the others |
| export GLOO_SOCKET_IFNAME=tailscale0 # your mesh / LAN NIC |
| daisychain-train |
| ``` |
|
|
| ### Windows helper |
| ```bat |
| scripts\setup.bat |
| ``` |
| An interactive menu: Docker, Python node, or just install deps. |
|
|
| Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**. |
|
|
| ### π SpikeWhale control panel (sliders β real training) |
| ```bash |
| python -m daisychain.spikewhale_panel |
| # open http://localhost:8899 |
| ``` |
| A web control panel: pick model size / training settings with sliders, choose any |
| HuggingFace dataset you have access to (default: streamed FineWeb-Edu), hit |
| Start, and watch the live loss. Stop and re-adjust any time with |
| **β Back to settings**. Launches the real DaisyChain training underneath. |
|
|
| ### π DaisyChain-Web (train by opening a browser tab) |
| ```bash |
| cd web && npm install && node server.js |
| # open http://localhost:8787 on every device |
| ``` |
| Zero-install browser training: devices on the same network auto-group |
| (Snapdrop-style) and train a shared model **peer-to-peer over WebRTC**, computing |
| through the same verified INT8 units (WebGPU, with the identical units on CPU |
| for machines without it β there is no plain-float path). Private cross-network |
| rooms via `?room=CODE` with **host approval** β the room creator accepts each |
| device before it can join. Includes gradient averaging with a deterministic |
| Adam optimizer (identical state on every peer, nothing extra over the wire), |
| checkpoint **download** (.pt) and **upload β broadcast** so one device can |
| restore the whole group after a failure. |
|
|
| **Live demo:** https://huggingface.co/spaces/Quazim0t0/DaisyChain-Web |
|
|
| ### Recent updates (July 2026) β DaisyChain-Web |
|
|
| **Verification stack** β the browser trainer's correctness is now *checked by |
| things that run*, not argued ([full results](web/TEST_RESULTS.md)): |
| - **IEEE-754 oracle** (`web/test_ieee.js`): a binary32 oracle built from the |
| standard in exact BigInt arithmetic proves the JS epilogue mirror is |
| spec-correct β and rejects the old round-once mirror on 34% of inputs. |
| - **Metamorphic properties + oracle mutation scoring** (`test_metamorphic.js`, |
| `test_corpus.js`): properties needing no reference implementation, scored |
| against an externally-authored bug taxonomy β **4/4**, matching the exact |
| differential gate's 4/4. Relations own the loop bugs; two definitional |
| absolutes (ReLU output range, a unit-scale integer anchor) own the value |
| bugs no relation can see. |
| - **Exact kernel gates on every live kernel**, a continuous audit at live |
| shapes, and a cross-device kernel probe (same seeded int8 GEMM, same hash on |
| every honest device, any backend). The audit's sampling was rebuilt against |
| a named bug class: its old constants (6 cells, 2% of GEMMs) bounded an |
| overhead that had never been measured (auditing *every* GEMM costs <0.01% of |
| a step), and uniform random cells cannot see a last-row/column bug at a |
| 16512-wide output. Sampling is now **stratified** β the first cells are the |
| structural danger points, chosen deliberately. On a last-column bug, same |
| cell budget: uniform caught **5/300** audits, stratified **300/300**, with |
| zero false positives. |
| - **RDNA2 ISA audit**: reading a real GPU's shader ISA against our |
| determinism assumptions confirmed three of them on silicon (exact packed |
| int8 dot; correctly-rounded f32 add/mul; 1-ULP reciprocal β division stays |
| off the GPU) and produced two hardenings. (1) Real ISAs have non-IEEE |
| variants that **flush β0 to +0**; JS `!==` can't see that (`-0 !== 0` is |
| false), so all gates and audits now compare **bit patterns** β exactly what |
| the replica hash sees. (2) FMA contraction of the quantize's `xΒ·inv + 0.5` |
| (one rounding instead of two) turned out to be **floor-invisible by |
| construction** β proven in `test_b2b.js` with 175k+ last-ulp anomalies at |
| binade edges, zero surviving `floor()`. Rounding mode and denorm flushing |
| are runtime driver state on real hardware, which is why every device |
| re-runs the exact gates at every init. |
|
|
| **Training data** β FineWeb-Edu (10BT sample) is the hardcoded dataset. The |
| Space reads random slices of the parquet shards straight off the HF CDN with |
| range requests (pure-JS `hyparquet`, SNAPPY) and serves plain text at `/data` |
| β no dependency on the datasets-server rows API and its 503s. |
|
|
| **Resilience** β the sync guard now *repairs* instead of halting: a roster |
| gradient that reached the leader but not some follower (asymmetric WebRTC |
| mesh) is re-requested from the leader, bit-exact, and the run continues. The |
| guard still stops anything that would fork the weights. |
|
|
| **CUTLASS-style kernel work**, each step proven bit-identical or exact-gated: |
| - **Dispatch-optimized backward** (ex. 05/24): independent GEMMs overlapped, |
| sibling trios fused into batch-3 dispatches β bit-identical gradients, |
| dormant int8-backward path down from 1.63Γ to 1.21Γ vs float. |
| - **QKV dual-GEMM fusion** (ex. 45): q/k/v share one left operand β quantized |
| once, one batched dispatch, zero changed bits. |
| - **B2B MLP chain** (ex. 13 + 23): both MLP GEMMs back-to-back on the GPU with |
| a fused per-row absmax reduction; the intermediate is quantized on-device |
| via a WGSL-exact respec (`floor(f32(xΒ·invScale)+0.5)` β no GPU division) |
| whose fround-stepped JS mirror keeps mixed GPU/CPU fleets bit-identical. |
|
|
| **Profile-driven speed work** β every change below is bit-identical (gradient |
| and loss hashes unchanged), so none of it trades correctness for wall clock: |
| - **Buffer pooling**: GPU buffers are recycled by size bucket instead of being |
| created and destroyed per dispatch (~19 per MLP call, per layer, per step). |
| **6β10% faster**, every hash unchanged. |
| - **Shared-operand embedding GEMMs**: profiling put two f32 backward GEMMs at |
| **55% of the entire step**, and both consumed the same `dlogits` operand β |
| ~17 MB at the 16512-token vocab, uploaded *twice*. One upload, one encoder, |
| one submit: that pair went 205 β 90 ms and the step **12% faster**. The |
| fusion is gated bit-for-bit against the two calls it replaces. |
| - **A negative result, kept on purpose**: the remaining hot kernel looked |
| cache-hostile (adjacent lanes wrote 66 KB apart), but making the writes |
| contiguous changed nothing. Two probes explain why β holding the output at |
| 17 MB while cutting compute 32Γ barely moved the time. The logits GEMM is |
| **transfer-bound, not compute-bound**, and the readback cannot be removed |
| because softmax must stay in JS (WGSL's `exp` is not correctly rounded, and |
| a per-vendor `exp` would fork replicas). The real lever there is the |
| vocabulary, not the kernel. |
| - **An init backend race was tried and removed**: it tied on the shipped path, |
| cost ~430 ms of init, and made the backend vary between page loads, which |
| silently invalidated three A/B comparisons before it was caught. A knob that |
| changes what you are measuring is worse than a fixed choice. |
|
|
| **Dirty-buffer gate β and the assumption it falsified.** Pooling introduced a |
| bug class the gates predate: a pooled buffer is *not* zero-initialized, so a |
| kernel that assumes zeros is right on step one and wrong on step two. That is |
| a **state** bug, where no single call is wrong and the *sequence* is, which is |
| the family no oracle can reach. The assumption was that the gates were blind |
| to it. Mutation-testing the gate proved otherwise: deleting the zeroing made |
| the plain gate fail at its *second* shape, because the sweep's own shapes |
| recycle each other's buffers. The suite had **incidental** coverage nobody |
| designed, which is coverage nobody can rely on β shorten the shape list and it |
| evaporates with the gate still green. It is now deliberate: the pool is |
| poisoned with 1e4-magnitude residue before a re-sweep, so detection no longer |
| depends on ordering luck. ~90 ms one-time. |
|
|
| **Scoring the oracles against my OWN bugs** (`web/test_selfcorpus.js`) β the |
| external corpus measures kernel bugs someone else wrote down, so this suite |
| asks the harder question: what do the instruments score against the four real |
| bugs of the month? Properties **0/2** on the data-plane pair (the `cΒ·out` |
| theorem again), differential **2/2** β but half the bugs were not in the |
| kernels at all. A dead gate is a bug in a *checker*, caught only by mutating |
| the gate; a stalled roster gradient is a bug in the *protocol*, where every |
| computed value on every peer was correct, so no data oracle could fire. Those |
| needed different instruments, not better oracles. |
|
|
| All thirteen test suites (`cd web && npm test`) pass; results with methodology in |
| [`web/TEST_RESULTS.md`](web/TEST_RESULTS.md). |
|
|
| --- |
|
|
| ## How it works |
|
|
| Each machine runs the **same** command; they form a cluster and train one shared |
| model. Two things happen: |
|
|
| 1. **The compute runs through the emulated GPU logic.** By default the model is |
| built from `VerifiedLinear` layers, so every forward multiply / requantize / |
| ReLU is done by the **bundled verified INT8 units** (`daisychain/verified/`) β |
| the emulated GPU math. Rank 0 prints **cluster-wide unit-invocation counts** |
| so you can see the emulated logic doing the work. |
| 2. **The machines are pooled data-parallel.** Each node trains on its own shard; |
| gradients are capacity-weighted and combined into the exact full-batch |
| gradient, so replicas stay **bit-identical**. Faster machines automatically |
| take a bigger share. |
|
|
| ``` |
| old machine A ββ |
| old machine B ββΌββΊ each runs the emulated GPU logic on its shard ββΊ one model |
| old machine C ββ (gradients combined across the cluster) |
| ``` |
|
|
| ## Bring your own model |
| DaisyChain-Train trains any **Task** (`build_model` / `sample` / `loss`). Copy |
| `examples/my_task_template.py`, set `DAISY_TASK=your_module:YourTask`. Use |
| `VerifiedLinear` (see `daisychain/verified_task.py`) to run your model's compute |
| through the emulated units. See **[docs/CUSTOM_TASK.md](docs/CUSTOM_TASK.md)**. |
|
|
| ## Plain-float alternative |
| To skip the emulated units and train with normal float math on each machine, set |
| `DAISY_TASK=daisychain.example_task:ExampleTask`. Same cluster, same pooling β |
| the model math just runs as ordinary float instead of through the verified units. |
|
|
| ## The dashboard |
| `daisychain-dashboard` (or the Docker service) serves a Tailwind page at `:8080` |
| β readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan |
| (per-node device, weight, batch), and live training loss. |
|
|
| ## Networking |
| Use **Tailscale** for a P2P mesh so machines on different networks get stable IPs |
| on one interface β **[docs/TAILSCALE.md](docs/TAILSCALE.md)**. |
|
|
| --- |
|
|
| ## Layout |
| ``` |
| daisychain/cluster.py capacity-weighted CPU/GPU data-parallel trainer |
| daisychain/train.py entry point (daisychain-train) |
| daisychain/verified/ bundled trained N/N units + VerifiedLinear (train through them) |
| daisychain/verified_task.py default task: forward runs on the verified units |
| daisychain/example_task.py plain-float alternative task |
| daisychain/task.py the Task interface + loader |
| daisychain/dashboard/ agent + P2P scanner + Tailwind server |
| docker/ Dockerfile, dashboard image, compose (demo cluster) |
| scripts/setup.bat / setup.sh interactive setup helpers |
| config/ nodes + cluster env examples |
| examples/my_task_template.py starting point for your own model |
| test_verified_units.py full-domain checks of every shipped verified path (25) |
| docs/ QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE |
| daisychain/spikewhale_task.py trains the real SpikeWhale on streamed HF datasets |
| daisychain/spikewhale_panel.py slider control panel (localhost:8899) |
| web/ DaisyChain-Web: P2P browser training (WebRTC + WebGPU) |
| export_luts_web.py regenerates web/public LUTs from the trained units |
| ``` |
|
|
| ## Recent updates (14 October 2026) β one arithmetic, whatever the device |
|
|
| Lessons from making an emulator give one set of semantics on two backends, applied here and |
| measured on real hardware: |
|
|
| - **Mixed CPU + GPU clusters forked at step 1.** On identical gradients, CUDA Adam and CPU Adam |
| differ in about 2% of weights after 20 steps. That holds for the default, `foreach=False` and |
| `fused=True` alike: same rule, two implementations, two roundings. `replica_diff` only ran |
| after training, so nothing noticed. Now: |
| - In a mixed fleet, GPU ranks step the optimizer on CPU master copies, so every rank runs one |
| implementation (`DAISY_CPU_OPT=0/1` overrides the choice). |
| - Every rank compares a hash of its parameter bits at the logging cadence and stops on any |
| difference. |
| - `test_cluster_mixed.py` (real 2-rank gloo, one CUDA rank and one CPU rank) checks that the |
| old behaviour is stopped by the check and that the new one ends bit-identical. |
| - **Host floating-point mode is pinned.** With flush-to-zero on, the same seed gives different |
| weights (measured), and any library built with `-ffast-math` can switch it on just by loading. |
| `pin_fp_mode()` turns it off in `configure_cpu` and `fit`. |
| - **Subnormals in WebGPU.** WGSL permits flushing f32 subnormals, and the JS mirror never |
| flushes. The f32 GEMM gate's inputs (~0.1) never produced one. Measured in a real browser: the |
| Intel gen-9 iGPU **flushes**, and the old gate called it "bit-exact". The gate now includes a |
| subnormal case, so a flushing GPU keeps its backward in JS instead of forking from |
| non-flushing peers. The app also asks for the `high-performance` adapter now; the default |
| picked that iGPU over a discrete GPU. |
| - **Capacity in one unit.** A node whose benchmark failed reported its *core count* as |
| capacity, which was then weighed against other nodes' matmuls/sec. Mixed units now fall back |
| to core counts for every node, with a warning. |
| - **The kernel probe can no longer pass by not running.** If a device's probe threw, it kept its |
| previous hash (or 0), and a `phash` of 0 made the peer comparison *skip* that device. A failed |
| probe now reports 0, and 0 halts the run (`web/test_probe_unproven.js`). |
| - **Partial fragments of a closed channel are dropped.** Before, they were kept on the side that |
| doesn't start the reconnect. Repeated or out-of-range fragment numbers are rejected instead of |
| being counted. |
| |
| Verified in-browser: 2 peers over WebRTC, 100/100 steps, loss 9.76 β 6.26, no weight, kernel or |
| audit halts. |
|
|
| ### The f32 backward GEMM is audited too |
|
|
| Three gaps from a review of which checks actually cover each live WebGPU kernel: |
|
|
| - **The transposed f32 GEMM was compared only with itself.** The init gate checked the plain |
| split-K shape against the exact mirror. The transposed index pattern (`A[p*m+row]`, used for |
| the embedding gradient) was only ever compared as `fgemm2`'s leg against an `fgemm` call that |
| runs the same shader, which is circular. It is now gated against `fgemmMirror`, including a |
| transposed split-K shape (k > 4096). |
| - **The f32 GEMM had no live-shape audit.** Every int8 kernel samples cells at live shapes. The |
| f32 pair, whose output *is* the embedding gradient, had been checked only at 7Γ4500Γ5 and |
| 9Γ300Γ6. It is now audited by `auditFgemm` (structural corners first, then random cells) on |
| the mirror's partition and rounding schedule. The device's schedule (`fgemmFma`) is recorded |
| at init and passed through. |
| - **Dirty-pool re-gates for the attention kernels and the f32 GEMM.** The bgemm and MLP gates |
| already re-ran on pooled buffers poisoned with large residue. The attention and f32 kernels |
| were only ever gated on fresh buffers; now they are re-gated on poisoned ones too. |
|
|
| `web/test_fgemm_audit.js` (in `npm test`) checks that the audit accepts the mirror at live-like |
| shapes under both schedules and catches a 1-ulp error in a corner and in an interior cell. It |
| also checks that the two schedules really differ on split-K data, so the flag is load-bearing. |
|
|
| ## Recent updates (30 September 2026) β reductions refuse what they used to accept |
|
|
| Six fixes, each found by applying a lesson from differential-testing an RDNA2 shader |
| translator: *a second implementation nobody checks is not verified*, *inactive or absent |
| participants must not leak into a reduction*, *a max over NaN is undefined*, and *know your |
| accumulator width*. Every one is covered by a check that fails on the old code. |
| `python test_verified_units.py` + `python test_cluster_reductions.py` and |
| `cd web && npm test` (thirteen suites) reproduce all of it. |
|
|
| | # | where | what was wrong | now | |
| | --- | --- | --- | --- | |
| | 1 | browser, WebGPU MLP chain | the row max comes from an `atomicMax` over `bitcast(abs(h))` and never passed the non-finite guard the CPU path has; a NaN/Inf `h1` became `inv = 0`/NaN and the GPU quantized `floor(NaN)` (implementation-defined) where a CPU peer threw | `scalesFromAbsMax` refuses a non-finite max. NaN and Inf bits order above every finite value as u32, so the row max is non-finite **iff** some element is β one check covers the whole row (checked on 2000 random rows) | |
| | 2 | browser, `averageGrads` | a gradient of a different length was silently averaged over a prefix (typed arrays ignore out-of-range writes); a NaN/Inf gradient went straight into DaisyAdam, poisoning `m`/`v` on **every** peer identically, so no fork check could see it | refuses both with `GradientError`; the training loop halts at that step with weights and Adam state untouched (still a valid checkpoint) | |
| | 3 | browser, int8 GEMMs (JS and WGSL) | the `Int32Array` and WGSL `i32` accumulators wrap past K = 131,071 β CPU and GPU agree, on a wrong sum | refused with `AccumulatorBoundError` at every GEMM entry point; the same bound as the Python kernel's `MAX_INT32_K` | |
| | 4 | Python, logged cluster loss | `sum(loss_i) / world` β an equal-weight mean of **unequal**, capacity-weighted batches | `cluster_mean_loss` = Ξ£ wα΅’Β·lossα΅’ with the gradient's own weights. Measured on 3- and 13-sample nodes: old **1.418**, true full-batch loss **0.959**, new 0.959 | |
| | 5 | Python, gradient all-reduce | one node's NaN was summed into every replica and applied | `check_finite_grads` refuses the reduced gradient before `opt.step()`; every rank holds the same sum, so every rank stops at the same step with no extra communication | |
| | 6 | Python, `pick_device` | with `CUDA_VISIBLE_DEVICES` set but **empty** (a common "CPU only"), torch reports `is_available() = True` and `device_count() = 0`, and the node crashed at startup | a device must actually exist; falls back to CPU | |
|
|
| Unchanged for valid inputs: finite gradients, finite activations, and K β€ 131,071 produce |
| bit-identical results to before, so updated and older peers still co-train. A 2-rank |
| end-to-end `DaisyCluster.fit` run stays bit-identical across replicas (`replica_diff 0.0`). |
| The WebGPU-side changes are host-side JavaScript (checked by `node --check` and by the |
| Node suites through the shared mirror functions); the WGSL kernels themselves are unchanged. |
|
|
| ## Recent updates (September 2026) β requant rounding, bounded units, non-finite guards |
|
|
| Everything below is measured, and `python test_verified_units.py` (25 checks) plus |
| `cd web && npm test` (twelve suites) reproduce it. |
|
|
| **The requant rounds instead of truncating.** `NeuralRequant16` was |
| `sat_int8(x >> 8)`, and Python's `>>` on negatives is an arithmetic floor, so every |
| requantized activation carried a systematic bias. Over the full int16 domain: |
|
|
| | requant | mean error | disagrees with the other on | |
| | --- | --- | --- | |
| | floor `x >> 8` (old) | **-0.4981 LSB** | 32,640 / 65,536 | |
| | round-half-up `(x + 128) >> 8` (now) | +0.0020 LSB | | |
|
|
| The shipped `requant16.pt` implemented the floor exactly, so this was real behaviour, |
| not a stale docstring. It was **retrained** against the new reference and saved only |
| at 65,536/65,536 (a 99.9% unit would silently break the LUT equivalence the browser |
| relies on), and `web/public/requant_lut.bin` was **re-exported** to match: the |
| invariant is *neural forward == LUT == native integer op*, so a table that lags its |
| unit is a fork. Tie rule, stated precisely: this is half-UP; Python/numpy/torch |
| `round` is half-to-even. The two differ on exactly the 128 exact ties (0.2% of |
| inputs) and both are unbiased to within 0.002 LSB. The Python fleet quantizes with |
| `np.round` and the browser with `floor(x + 0.5)`; the two fleets never co-train, so |
| that cannot fork a group. |
|
|
| **Bounded elementwise units.** The GEMM backends already capped their temporaries; |
| the units right after them did not. `relu_array` / `requant_array` received the |
| whole activation matrix in the proof path at ~1.5β2 KB per element (262,144 |
| elements = **407 MB**), and the multiply-LUT build pushed 262,144 rows through the |
| atom net at startup (**277 MB RSS on every node**). Both are now blocked: |
| 407 MB β **2 MB, flat with N**; 277 MB β **1 MB**. Elementwise blocking is exact |
| (no accumulation order), verified bit-identical at N = 1 / 999 / 32768 / 32769 / |
| 100000. `dataset()` builders are vectorised (65,536 Python iterations β one numpy |
| pass, 0.015 s), which is what makes retraining a unit practical at all. |
|
|
| **A NaN or Inf no longer quantizes to zeros.** A float β int8 conversion has no |
| answer for NaN/Inf, and both trainers gave a silent wrong one: |
|
|
| | path | input | old result | |
| | --- | --- | --- | |
| | Python `qat.py` | `[0.5, nan, 3.0]` | scale = NaN β **`[0, 0, 0]`** (only a RuntimeWarning) | |
| | Python `qat.py` | `[0.5, inf, 3.0]` | scale = Inf β **`[0, 0, 0]`** | |
| | browser quantizers | `[0.5, Inf, 3]` | scale = Infinity β **`[0, 0, 0]`** | |
| | browser quantizers | `[0.5, NaN, 3]` | NaN skipped by the \|max\| scan, stored as **0** | |
|
|
| One bad value zeroed the whole layer's int8 input and training carried on. Now |
| `VerifiedLinear` raises `FloatingPointError`, and every browser quantizer |
| (`quantize`, `quantizeRows`, `quantizeCols`, `rowAbsMax`, `quantizeHeadCols`, and |
| the transformer's `quantizeColsAsRows`) throws `NonFiniteError`, naming how many |
| values were bad. Finite inputs are untouched, so builds with and without the guard |
| still co-train. Both new tests fail against the old code (Python: 2 checks; web: |
| `test_nonfinite.js`, 12) and pass now. |
|
|
| ## Recent updates (August 2026) β verified compute path |
|
|
| Three changes to `daisychain/verified/`, each measured rather than asserted. |
| `python test_verified_units.py` (24 checks) reproduces all of it. |
|
|
| **Bounded GEMM memory.** `LUTBackend.gemm` and `NeuralBackend.gemm` materialized |
| the whole `(m, n, k)` product block before reducing it, so peak memory was cubic |
| in layer width. At the SpikeWhale panel's maximum (hidden 768, sequence 512) a |
| single layer's forward allocated **2.4 GB** β on the spare hardware this project |
| exists to use. Both now block the `m` axis. The contraction axis `k` is untouched, |
| so the sum and its order are unchanged and results stay bit-identical: |
|
|
| | GEMM | before | after | |
| | --- | --- | --- | |
| | 64x64x64 | 2.3 MB | 2.3 MB (unchanged path) | |
| | 256x256x256 | 135.8 MB | 68.9 MB | |
| | 512x768x768 (panel max) | 2426.9 MB | **77.2 MB** | |
| | neural backend, 64x96x96 | 519.2 MB | **57.5 MB** | |
|
|
| Caps are class attributes (`LUTBackend.max_block_bytes`, |
| `NeuralBackend.max_products`), tunable per instance. Below the cap the LUT path |
| takes the original single-block branch unchanged. |
|
|
| The two backends need *different* accounting, which is worth knowing before tuning |
| them: a LUT product costs 8 bytes, but a neural product is split into four nibble |
| pairs through a 128-wide net and costs ~880 bytes. Applying the LUT's byte budget |
| to the neural path computes a block 110x too large and never splits at all. |
|
|
| **The multiply LUT is certified, not trusted.** `build_luts()` returned its tables |
| unchecked while the docs described a self-certify gate. Because the table *is* the |
| complete finite domain, certifying it **is** the exhaustive verification: |
| `LUTBackend.__init__` now checks all 65536 entries against signed integer multiply |
| and raises on mismatch. Cost: **0.5 ms**. |
|
|
| **Full-domain tests for the paths that ship.** The units' exhaustive verification |
| covers the scalar entry points (`NeuralMul8.verify()` walks `mul()`); production |
| runs the batched ones β `mul_array`, `relu_array`, `requant_array` β and the LUTs |
| built from them, which are different code. `test_verified_units.py` checks every |
| shipped path against golden integer arithmetic over its complete domain. All pass |
| today; nothing would have caught a regression before. |
|
|
| **Two guards against silent no-ops.** `gemm_int8` enforces the int32 accumulator |
| bound (`MAX_INT32_K = 131071`) instead of asserting it in a comment, since layer |
| width is user-selectable and an overflow would produce wrong numbers rather than |
| raise. `instrument.require(**minimums)` fails when a verified unit was |
| under-invoked, and refuses to report at all while counting is disabled β a zero |
| from a probe that was switched off is not evidence that the units did not run. |
|
|
| --- |
|
|
| ## Install |
| ```bash |
| pip install torch numpy psutil |
| pip install -e . # exposes: daisychain-train, daisychain-agent, daisychain-dashboard |
| ``` |
| Requires Python β₯ 3.9, PyTorch β₯ 2.0. Multi-node is reliable on **Linux/macOS**; |
| on **Windows use Docker/WSL** (see [Limitations](docs/LIMITS.md)). |
|
|
| --- |
|
|
| ## Links |
| - **DaisyChain on Hugging Face:** https://huggingface.co/DaisyChainAI |
| - **This model:** https://huggingface.co/DaisyChainAI/DaisyChain-Train |
|
|
| **License:** MIT Β· **Author:** Dean Byrne (Quazim0t0) Β· **Org:** DaisyChainAI |
|
|
| ## Citation |
| ```bibtex |
| @misc{byrne2026daisychain, |
| title = {DaisyChain-Train: An Old Hardware Training Pipeline}, |
| author = {Byrne, Dean (Quazim0t0)}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}}, |
| note = {Chain spare/old machines into a data-parallel training cluster} |
| } |
| ``` |
|
|
| **Dean Byrne (Quazim0t0)** Β· 2026 |
|
|
