Gemma4-Coder
Collection
Custom gemma models for coding • 6 items • Updated
▚ Quick start
llama-server -hf pearsonkyle/gemma4-e4b-coder-gguf:IQ4_XS \
-c 32768 -ngl 99 --jinja --host 0.0.0.0 --port 8000
ollama run hf.co/pearsonkyle/gemma4-e4b-coder-gguf:IQ4_XS
▚ Which rung
Every file below is in this repo. The 2-bit builds are not — see Quality.
▚ Deployment — how many tokens you can actually hold
Only 4 of 42 layers grow their KV cache with context (the rest either reuse
shared KV or run a fixed 512-token sliding window), so a sequence costs
16 KiB x ctx + 20 MiB at bf16 — 0.52 GiB at 32 K, 2.02 GiB at the full
131,072. --cache-type-k q8_0 --cache-type-v q8_0 roughly halves it. Add the
rung's own file size for the total footprint:
llama-server -hf pearsonkyle/gemma4-e4b-coder-gguf:IQ4_XS \
-c 131072 --parallel 8 -ngl 99 --jinja \
--cache-type-k q8_0 --cache-type-v q8_0
-cis the TOTAL KV pool, shared across--parallelslots — each slot gets-c / --paralleltokens. That one knob is what decides "max tokens": context length and concurrency trade against a single budget.
Derived from
config.json, not measured. The arithmetic is inscripts/kv_budget.pyin Quant-Tuner. llama.cpp implements its own SWA-aware cache — read its startup line (KV self size = …) and trust that over the table.
▚ Quality
Use IQ4_XS. It is statistically indistinguishable from the bf16 GGUF on this suite. If you need smaller, IQ3_M-awq (2.36 GiB) is the smallest rung that still calls tools reliably; plain IQ3_M ships as its baseline.
The 2-bit rungs are measured here but not uploaded. IQ2_M scores 0.0000 on every metric — 102 of 107 turns emit no tool call at all — and AWQ scaling does not change that. They are reported in full below so the ladder's floor is visible, but shipping 1.84 GiB of a build that cannot call a tool would only invite someone to download it.
Every arm is scored through llama-server on the same 107 held-out turns, and so
is a bf16 GGUF — the control. That control is not ceremony: the same bf16 weights
score 0.8692 in the transformers harness and 0.9065 here, a 3.7-point gap on
identical weights from the stack alone. That is the same size as the effects being
measured, so every rung is read against the bf16 row in this table and never
against the base model's published numbers.
Degradation is not gradual — it fails in two stages. The failure reasons say more than the rates do:
That is why IQ3_M's selection p-value is an unremarkable 0.15 while its schema p-value is below 0.001: it has not forgotten which tool to reach for, it has lost the ability to fill the call in. IQ2_M's 97 discordant schema turns split 97/0 — there is not one turn where it succeeded and bf16 did not.
IQ2_M also destabilised the server: 3 of 36 sessions ended in API errors (104 of 107 turns scored, against 107 for every other arm). Its row is reported over the turns it completed; treat the rate as generous rather than harsh.
AWQ scaling: a real gain at 3-bit, nothing at 2-bit
Both sub-4-bit rungs were also built with AWQ per-channel scaling folded in before quantization, each re-calibrated with its own imatrix over the same 15 M-token corpus at the same context — so the head-to-head isolates the scaling. Both AWQ builds are byte-identical in size to their plain twins, so nothing here is bought with bits.
At 3-bit AWQ works, and significantly. It recovers 21 turns the plain rung fails while losing 8, p = 0.024. That closes most of the distance to the reference: against the bf16 GGUF, IQ3_M-awq is no longer significantly worse on schema validity (27 disc, 18/9, p = 0.12, versus p < 0.001 for the plain rung) and is at parity on tool selection (15/13, p = 0.85). Parameter accuracy is the one place a real gap survives: −0.084, 95 % CI [−0.160, −0.007], which excludes zero.
Against IQ4_XS it shows no detectable difference on any metric (schema p = 0.12, selection p = 0.86, param CI spans zero) — but that is a statement about this suite's power at n = 107, not proof of equivalence, and schema is directionally behind (18 turns to 9). IQ4_XS remains the recommendation because it matches bf16 exactly; IQ3_M-awq is the choice when 0.32 GiB matters more than that certainty.
At 2-bit AWQ changes nothing — not one of 107 turns differs, and 106 of them emit no
tool call at all. This is a trustworthy null: the fold was verified exact to
rel = 1.0e-05 before quantization (an earlier attempt folded in bf16 at rel = 0.189,
six times over awq.apply's own 0.03 gate, and its results were withdrawn). The reason is
structural. AWQ folds 84 groups — 42 layers × (attn, mlp), the entire trunk and nothing
else. At IQ2_M the vocabulary tensor is 0.56 GiB, 30.3 % of the file, and AWQ never
touches it. A trunk-only intervention cannot repair damage concentrated in an embedding
table, and the QAT conditioning that makes 4-bit nearly free offers nothing this far below
its lattice. Tellingly, the better-folded build rambles more, not less: turns running to
the token cap went 43 (plain) → 59 (bf16 fold) → 67 (fp32 fold).
Neither 2-bit build is shipped. Plain IQ2_M and IQ2_M-awq both score 0.0000 and are
absent from this repo by choice; their per-turn results are in eval/toolcall.csv if you
want to verify the floor yourself. Both imatrices are published, so either can be rebuilt
locally with llama-quantize --imatrix.
What this does not say. The suite resolves differences of roughly ±0.09 at n = 107, so "IQ4_XS is indistinguishable from bf16" means exactly that and not "lossless" — its parameter accuracy is 0.4415 against 0.4905, a gap this holdout cannot separate from noise but which is not evidence of no gap.
▶ How the quality numbers were measured
Every rung is scored through llama-server, and so is a bf16 GGUF — because the published bf16 numbers for this model came from the in-process transformers harness, and those are different stacks. A reference model can score differently on each for reasons that have nothing to do with quantization, so each rung is read against the bf16 GGUF measured here, never against the number on the base model's card. Cross-stack comparisons are not valid and are not published as if they were.
Three defaults would otherwise have made these incomparable with the base model's table, so they are pinned: --no-stop-on-fail (the default stops a session at its first wrong tool, giving each arm a different denominator — that is how an earlier run scored 32 and 70 turns against a 107-turn table), --ctx 65536 (the default 8192 truncates these sessions, the most likely cause of a spurious zero), and --max-tokens 2048.
Sessions come from the test slice of the seeded split — never used for training or calibration.
▶ How it was made — imatrix, the converter patch, calibration corpus
imatrix at ctx 32,768, matching the corpus's pack context. A corpus packed at 32 K and measured at 512 would split nearly every session, and the statistics would then describe fragments — and a middle fragment has no system prompt and no tool schemas, a context that never occurs at inference. The -ctx32k calibration split admits only conversations that fit whole inside 32,768 tokens: 457 chunks over 15,001,898 tokens.
This model has no separate output.weight. tie_word_embeddings is true, so the GGUF carries one token_embd.weight doing double duty as input lookup and output projection — verified directly against the bf16 GGUF's tensor list. --process-output therefore has nothing to act on here, and the vocabulary tensor is quantized without imatrix guidance (llama.cpp holds it at higher precision than the trunk for exactly that reason).
What the imatrix does cover is 342 tensors — every quantizable matmul in the trunk. A detail worth reading off it: attn_k and attn_v appear for only 24 of 42 layers, not all 42, because layers 24–41 are num_kv_shared_layers and carry no key/value projection weights at all.
Calibration: the same corpus as the W4A16 build — 11,982 conversations, token-balanced with no source above 6 %, drawn only from the train slice of the seeded split so the evaluation holdouts stay clean. 12,175 tool calls and 2,940 tool-schema declarations survive templating. Corpus SHA-256 is recorded alongside the artifacts.
The converter needed a patch. transformers 5.16.1 writes Gemma 4's attention geometry as per_layer_config: {"05": {"head_dim": 512}, …} on exactly the 7 full_attention layers, where 5.6.2 wrote a single global_head_dim: 512. llama.cpp's gemma.py reads the old key and dies with KeyError: 'global_head_dim'. Patching config.json does not fix it — AutoConfig.from_pretrained().to_dict() normalises the old key back into the new form, so the converter never sees it. The fix derives the value from per_layer_config, verifying the keys are exactly the full_attention layers and that they agree on one head_dim. Patch: patches/llamacpp-gemma4-global-head-dim.patch in Quant-Tuner.
Built with Quant-Tuner against llama.cpp f3e1828.
▶ The restricted vocabulary, and what every quant must preserve
This family does not use Gemma 4's 262,144-token vocabulary. It is pruned to 65,536 tokens chosen by frequency over the training corpus, which removes 5.24 GB of vocabulary-indexed weights before a single layer is quantized:
Two consequences bind every quantization of it:
- The tokenizer ships with the model and is not interchangeable. Each GGUF carries its own, so the normal path is safe — but feeding it ids from a stock Gemma 4 tokenizer produces fluent-looking nonsense with no error.
- The output head stays at high precision. A quantized head over a vocabulary is the classic rare-token failure mode, and a pruned vocabulary concentrates more meaning per row — dropped tokens byte-fall-back into surviving pieces, so those rows carry more of the distribution than they did before.
▶ Stop tokens
Decoding must halt on <eos>, <turn|> and <|tool_response> — the last is how
the model yields after a tool call. Each GGUF carries all three, so the normal path is
correct; a client that overrides them with <eos> alone will see the model keep decoding,
write the tool's output itself, and call another tool. That reads as a broken quant
but is a client config.
▶ Limits
- Text only. The vision and audio towers were removed when the vocabulary was pruned.
- Not suitable for multilingual use — non-Latin scripts fall back toward per-byte encoding under the pruned vocabulary.
- IQ2_M shows where the floor is, it is not a recommendation. Read its row in the quality table before deploying it.
- Perplexity is not the headline here.
llama-perplexityhas no--parse-special, so on a chat-templated eval file the control markers tokenize as plain BPE and absolute perplexity is off-distribution. Quant-vs-quant comparisons on the same file stay valid, which is why KLD leads and PPL does not. - Inherits every limitation of the base model, including that its held-out decision accuracy never improved during training — only format did.
▚ Family
▚ License
Gemma Terms of Use, inherited from google/gemma-4-E4B-it.
- Downloads last month
- 102
Hardware compatibility
Log In to add your hardware
Model tree for pearsonkyle/gemma4-e4b-coder-gguf
Base model
pearsonkyle/gemma4-e4b-coder