Kreuzzelg/qwen36-35b-a3b-colibri-i4

Qwen3.6-35B-A3B — colibri int4 container (runs out of the box)

colibri container for Qwen3.6-35B-A3B (35B total / 3B active, hybrid Gated Attention + Gated DeltaNet, 256 experts/layer, Apache-2.0), quantized to true int4 (per-row symmetric scales, ~20 GB). Runs with the pure-C qwen36 engine — not loadable with transformers.

Why this matters: with this container and colibri's CUDA expert tier, a single 8 GB GPU runs Qwen3.6-35B at 9.9 tok/s — almost the speed that Ollama needs two 8 GB GPUs for (10.5 tok/s), and 32 % faster than Ollama on the same single GPU (7.5 tok/s). Heat-based expert placement puts the 43 % of experts that serve ~95 % of routed traffic into VRAM; the rest is computed on the CPU, overlapped with the GPU. With two 8 GB GPUs the tier reaches 100 % VRAM hit rate and 11.3 tok/s.

This container is self-contained: unlike earlier conversions it bundles tokenizer.json and ships a flat config.json (Qwen3.6 VL checkpoints nest the text dims under text_config, which the engine cannot read). The original HF config is preserved as config.hf.json. Produced with the updated c/tools/convert_qwen36.py from the qwen36 engine PR (https://github.com/kreuzzelg/colibri/tree/qwen36-engine); conversion is deterministic — you can reproduce the shards from the base model at any time.

Run it (CPU)

git clone -b qwen36-engine https://github.com/kreuzzelg/colibri && cd colibri
make -C c qwen36
SNAP=$(python -c "from huggingface_hub import snapshot_download; \
print(snapshot_download('Kreuzzelg/qwen36-35b-a3b-colibri-i4'))")
echo "Explain MoE routing in 100 words." > prompt.txt
SNAP=$SNAP N_NEW=200 OMP_NUM_THREADS=<physical cores> ./c/qwen36 256 4 prompt.txt

Run it fast (CUDA VRAM expert tier, 1–2 GPUs with ≥8 GB each)

git clone -b qwen36-cuda-tier https://github.com/kreuzzelg/colibri && cd colibri
make -C c qwen36 CUDA=1 CUDA_ARCH=native
COLI_CUDA=1 COLI_GPUS=0,1 HEAT_FILE=heat.bin \
OMP_NUM_THREADS=<physical cores> OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close \
SNAP=$SNAP N_NEW=200 ./c/qwen36 256 4 prompt.txt

Hot experts are placed into VRAM by measured routing heat (colibri's "route → place → overlap → learn"); HEAT_FILE persists the placement, so the second run starts fully warm.

Measured (Threadripper 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, 200-token decode)

configuration	decode tok/s	VRAM hit rate	peak RSS
CPU only	~2.5	–	~13 GB
1× 8 GB GPU	9.9	95 %	40 GB
2× 8 GB GPUs	11.3	100 %	29 GB
(Ollama q4_K_M, same box)	10.5	–	~40 GB

Numerics: logits cosine ≥ 0.999 vs the f32 CPU reference; GPU-vs-CPU on this container is bit-near identical (cosine 1.0000001).

Container format

Directory of safetensors shards (one per layer + globals): dense weights as f32-loadable tensors, each routed expert as model.layers.{L}.mlp.experts.{E}.merged_weight (int4, 2 nibbles/byte, layout gate|up|down) + .qs (f32 per-row scales), plus qwen36_meta.json (all engine dimensions). Details: docs/qwen36-phase01.md in the engine branch.

Credits

• Base model: Qwen/Qwen3.6-35B-A3B (Apache-2.0)
• colibri engine & concept: JustVugg/colibri
• Original qwen36 engine + converter: @minne100 (PR #602); this container adds the tokenizer/config fixes and was produced with the updated converter.