WeReCooking/AesCode-32B-EXL3

AesCode-32B EXL3

EXL3 quants of microsoft/AesCode-32B using exllamav3 v1.6.1

KL Divergence vs VRAM

Reference: BF16 original (PPL 6.843). Noise floor KLD 0.000151. Measured with exllamav3 eval/qbench.py on wikitext-2 (20 rows, 2048 ctx). VRAM = decoder + lm_head weights.

KLD spread (median, IQR, noise floor)

Quants

Branch	BPW	Head	VRAM (GiB)	KLD mean	KLD median	PPL	Type
3.0bpw_H6	3.0	6	11.5	0.0360	0.01522	6.987	base
3.73bpw_H6	3.73	6	14.1	0.0157	0.00616	6.902	optimized
4.0bpw_H6	4.0	6	15.1	0.0093	0.00392	6.876	base
4.54bpw_H6	4.54	6	17.0	0.0065	0.00260	6.863	optimized
5.0bpw_H6	5.0	6	18.7	0.0027	0.00122	6.871	base

Bases are direct converts. Optimized = KLD-guided tensor mix of the bases + attention recompiled at 5bpw; on this model they land on the base size/KLD curve, so treat them as in-between sizes and pick by VRAM. Vision tower 6bpw in every branch.

Download and run

Download

pip install -U "huggingface_hub"
hf download WeReCooking/AesCode-32B-EXL3 --revision 4.0bpw_H6 --local-dir ./AesCode-32B-EXL3

Run with exllamav3

Text chat, official exllamav3 CLI (model card sampling). Install exllamav3 first (prebuilt wheels):

git clone https://github.com/turboderp-org/exllamav3 && cd exllamav3
pip install pyperclip prompt_toolkit
python examples/chat.py -m ../AesCode-32B-EXL3 -mode chatml -cs 24576 -maxr 12000 -temp 0.8 -topp 0.95 -topk 20 -minp 0

One shot to file:

python examples/chat.py -m ../AesCode-32B-EXL3 -mode chatml -cs 24576 -maxr 12000 -temp 0.8 -topp 0.95 -topk 20 -minp 0 -prompt "Create a portrait poster infographic about coffee origins as one complete HTML page." -save page.md

With a reference image (how AesCode is meant to be used, based on examples/multimodal.py):

from exllamav3 import Cache, ComboSampler, Config, Generator, Model, Tokenizer
from PIL import Image

config = Config.from_directory("AesCode-32B-EXL3")
vision = Model.from_config(config, component="vision"); vision.load()
model = Model.from_config(config); cache = Cache(model, max_num_tokens=24576); model.load()
tok = Tokenizer.from_config(config)
emb = vision.get_image_embeddings(tokenizer=tok, image=Image.open("reference.png"))
prompt = ("<|im_start|>user\n" + emb.text_alias + "Create a landscape dashboard ... as one complete HTML page."
          "<|im_end|>\n<|im_start|>assistant\n")
gen = Generator(model=model, cache=cache, tokenizer=tok)
print(gen.generate(prompt=prompt, embeddings=[emb], encode_special_tokens=True, completion_only=True,
                   max_new_tokens=12000, stop_conditions=["<|im_end|>"],
                   sampler=ComboSampler(temperature=0.8, top_p=0.95, top_k=20)))

The repo's training system prompt and spec template (inference/prompts.py) give the best results.

Example output (4.0bpw_H6)

Repo example hospital_dashboard with the training prompt template and reference image. Exact text 11/11, chart data 5/5, table 0/1, 1 element outside the 1280x720 canvas. 5097 tokens at 58.0 tok/s on an RTX 5090.

Build Details

How these were made

Base quants: convert.py -b <bpw> (3.0, 4.0, 5.0), head 6bpw, vision 6bpw. Calibration: 250 rows x 2048 (default) for 3.0, 128 rows x 2048 for 4.0/5.0 (32 GB RAM build box)

KLD measurement: util/measure.py -r <bf16> -i <3.0> <4.0> <5.0> (measurement.json on main)

Optimized: util/optimize.py -i <3.0> <4.0> <5.0> -m measurement.json -b <target> then util/recompile.py -or override.yaml with *.self_attn.* from 5.0bpw

Eval: eval/qbench.py, BF16 streamed as reference

Docs: exllamav3 convert.md

Files

main branch: measurement.json, KLD plots, example image, README

Each bpw branch: quantized shards (text + vision) + config + tokenizer + processor configs