README.md

metadata

license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
  - designcoder
  - ui-generation
  - front-end
  - html
  - css
  - javascript
  - code-generation
  - full-sft
  - evaluator
  - reward-model

DesignCoder

Checkpoint collection for DesignCoder, a family of full-parameter SFT models for UI design research and end-to-end HTML/CSS/JavaScript implementation.

The repository holds two kinds of model:

Kind	What it does	Folder prefix
Generation	writes HTML/CSS/JS from a design brief	designcoder_{basemodel}_...
Evaluator	scores a rendered UI screenshot against a rubric	designcoder_evaluator_{basemodel}_...

They are trained on different datasets and answer different inputs — an evaluator will not generate pages, and a generation model will not produce valid rubric verdicts.

Each subfolder in this repository is a self-contained, directly loadable checkpoint.

Naming convention

designcoder_[evaluator_]{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{global_step}

• evaluator: present only for rubric-scoring models; absent means generation
• basemodel / size: base model family and parameter scale
• optimizer: muon or adamw
• bs: global batch size (per_device × grad_accum × world_size)
• extra_axes: any hyper-parameter that deviates from the default recipe, e.g. wd0.05 (weight decay, default 0.0), ep20 (epochs, default 2), or data41287 (dataset revision)
• step: trainer global_step of the exported weights

Dataset revisions

Checkpoints in this repository come from three different dataset revisions. Scores and loss values are only comparable within the same revision.

Tag	Samples	Dataset	Used by
(untagged) data37865	37,865	designcoder_sft_v2 (generation)	*_step1900, *_step3800
data41287	41,287	designcoder_sft_v2 (generation)	*_data41287_step200, *_data41287_step400
data37847	37,847	DesignCoder-evaluate (screenshot scoring)	designcoder_evaluator_*

⚠️ data37847 and data37865 are entirely different datasets — the near-identical sample counts are a coincidence. Never compare loss across them.

Checkpoints

Generation models

Subfolder	Base model	Optimizer	LR	Global batch	Dataset	Step	bench-200 (full, n=200)	Notes
designcoder_qwen3.5_4b_muon_bs32_step1900	Qwen3.5-4B	Muon	1e-5	32	37,865	1900	–	smallest of the first release
designcoder_qwen3.5_9b_muon_bs16_step3800	Qwen3.5-9B	Muon	1e-5	16	37,865	3800	–	optimizer ablation (Muon arm)
designcoder_qwen3.5_9b_adamw_bs16_step3800	Qwen3.5-9B	AdamW	2e-5	16	37,865	3800	–	optimizer ablation (AdamW arm)
designcoder_qwen3.6_27b_adamw_bs32_step1900	Qwen3.6-27B	AdamW	1e-5	32	37,865	1900	–	largest of the first release
designcoder_qwen3.5_4b_adamw_bs256_data41287_step200	Qwen3.5-4B	AdamW	2e-5	256	41,287	200	82.12	best 4B / AdamW
designcoder_qwen3.5_4b_muon_bs256_data41287_step200	Qwen3.5-4B	Muon	2e-5	256	41,287	200	77.62	best 4B / Muon
designcoder_qwen3.5_9b_adamw_bs256_data41287_step200	Qwen3.5-9B	AdamW	2e-5	256	41,287	200	84.40	best 9B
designcoder_qwen3.8_27b_adamw_bs128_data41287_step400	Qwen3.8-27B	AdamW	1e-5	128	41,287	400	87.89	strongest checkpoint in the collection

All four data41287 scores are final full-benchmark runs: 200/200 rollouts, 200/200 screenshot captures, 200/200 judge evaluations per model (no subsetting).

Evaluator models

Rubric scorers for UI screenshots — intended as reward / judge models during rollout, not for page generation. Trained on DesignCoder-evaluate (37,847 samples), evaluated on held-out benchmark screenshots scored by a stronger external vision judge (κ = agreement with that judge; see Evaluator benchmark).

Subfolder	Base model	Optimizer	LR	Global batch	Dataset	Step	OOD κ	Notes
designcoder_evaluator_qwen3.5_9b_adamw_bs256_data37847_step100	Qwen3.5-9B	AdamW	1e-5	256	37,847	100	0.694	1/3 memory, 2× speed; requires exactly 10-item rubrics
designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296	Qwen3.8-27B	AdamW	5e-6	256	37,847	296	0.664	robust to any rubric length — default choice

Which one to use. Counter-intuitively the 9B has slightly higher agreement; the 27B's advantage is format robustness. Verdict-count exact match across rubric lengths:

Rubric items	5	10	15	20	25
9B step100	0.00	1.00	0.92	0.94	0.85
27B step296	1.00	1.00	0.99	1.00	1.00

Training rubrics are always exactly 10 items, so anything else is out-of-distribution. Use the 9B when your rubric is fixed at 10 items; use the 27B otherwise.

Benchmark

bench-200 is the frozen 200-case DesignCoder benchmark (100 Track A landing, 40 Track A dashboard, 30 Track B landing, 30 Track B dashboard; Track A cases specify a style, Track B cases are style-free).

Rubric composition. Every prompt ships with its own reference rubric of 23–25 binary screenshot checks (184 prompts carry 25 checks, 15 carry 24, 1 carries 23 — 4,983 frozen checks in total), all evaluated with check_with=screenshot by a vision judge over the full-page render. Check distribution across dimensions:

Dimension	Checks	Share
Components	1,517	30.4%
Layout	842	16.9%
Aesthetics	782	15.7%
Typography	642	12.9%
Alignment	616	12.4%
Assets	584	11.7%

On top of the frozen checks, the judge scores 5 surface-specific Prompt-Fit items (0–2 each) per case. The reported overall_score (0–100) is the unweighted mean of Prompt Fit and the six rubric dimensions. Judge: gpt-5.6-sol (vision) with structured JSON output.

Full-run results (n=200 per model)

Model	Overall	Landing	Dashboard	Track A	Track B	Prompt Fit	Frozen pass rate	Render fails
27B AdamW step400	87.89	88.55	86.67	86.90	90.20	84.10	88.6%	0/200
9B AdamW step200	84.40	86.31	80.85	84.10	85.09	78.35	85.4%	1/200
4B AdamW step200	82.12	85.07	76.65	82.39	81.50	75.25	83.2%	1/200
4B Muon step200	77.62	81.65	70.14	77.64	77.58	64.60	79.4%	3/200

Scores increase strictly monotonically with scale (all 6 pairwise differences significant, paired bootstrap 10k-resample 95% CI excludes 0 and Wilcoxon p < 0.013 — see eval/significance_tests.json). The gap is far larger on dashboards (+16.5 from 4B Muon to 27B) than on landings (+6.9), and Assets is the weakest dimension for every scale (55–67% pass rate), indicating a data-level bottleneck rather than a capability ceiling.

Evaluation artifacts (eval/)

File	Content
eval/benchmark_summary.csv	per-model aggregates: overall, Track/Surface splits, six dimensions, Prompt Fit, frozen pass rate
eval/benchmark_per_case.csv	long-form per-case scores for all 4 models × 200 cases
eval/significance_tests.json	paired bootstrap (10k resamples) + Wilcoxon signed-rank for all 6 model pairs
eval/rubric_stats.json	rubric composition statistics (checks per prompt, per dimension, per track)
eval/reports.html	self-contained interactive HTML report: model comparison, dimension heatmap, score distributions, per-case tables

Checkpoint selection

The data41287 checkpoints were selected by running the benchmark, not by taking the lowest training loss. In all four runs the best checkpoint sits at roughly 75% of training, and loss kept improving while benchmark scores fell. The table below shows the 8-case selection subset (used only to rank checkpoints, not comparable to the final full-run numbers in the tables above):

Run	Step	Train loss	subset bench (n=8, selection only)	final full bench (n=200)
4B AdamW	200	0.2696	84.22	82.12
4B AdamW	266	0.2682	68.35	–
4B Muon	200	0.3339	83.36	77.62
4B Muon	266	0.3349	81.27	–
9B AdamW	200	0.2518	84.40	84.40
9B AdamW	266	0.2504	lowest of the three	–
27B AdamW	400	0.2067	91.19	87.89
27B AdamW	530	0.2059	86.37	–

The 4B AdamW pair is the clearest example: loss improved from 0.2696 to 0.2682 while the subset score collapsed from 84.22 to 68.35. Do not pick checkpoints from this family by loss. Note also that small subsets systematically overestimate: the subset ranks checkpoints correctly but runs several points above the full 200-case benchmark.

Evaluator benchmark

DesignCoder-evaluate ships no test split, and both evaluators trained on all 37,847 samples for 2 epochs — so there is no clean in-training validation set. Evaluation instead uses held-out inputs: 239 generated-page screenshots from the DesignCoder benchmark, each already scored check-by-check by a stronger external vision judge. A byte-level md5 check confirmed 0 overlap between those benchmark screenshots and the 37,851 training images.

Alignment is possible because the training target's frozen_dynamic_scores.verdicts and the external judge's verdict are both binary and share the same (dimension, point) structure.

Headline metrics (239 held-out cases, original 24–25-item benchmark rubrics):

Metric	9B step100	27B step296
JSON parse rate	1.0000	1.0000
Per-check agreement	0.9226	0.9313
Cohen's κ vs external judge	0.6353	0.6802
Defect recall (verdict = 0)	0.6581	0.6996
Case-level Pearson	0.8417	0.8572

Raw agreement is a weak signal here — 87% of checks are 1, so always-pass already scores ~88%. κ and defect recall are the meaningful numbers.

Reproducing the known quality ordering. The external judge ranks three generation checkpoints 4B 0.833 < 9B 0.884 < 27B 0.905. Both published evaluators recover that ordering to within ~2pp:

Scorer	4B	9B	27B	Ordering
External judge (reference)	0.833	0.884	0.905	–
Evaluator 9B step100	0.855	0.886	0.910	✅
Evaluator 27B step296	0.840	0.893	0.909	✅

Evaluator checkpoint selection

Checkpoints were chosen on a purpose-built out-of-distribution suite (576 prompts: rubric lengths 5/10/15/20/25 × three dimension-grouping shapes), not on training loss. The two families behave differently:

Run	Step	Train loss	OOD κ	Overfit gap (in-dist − OOD)
9B AdamW	50	–	0.6568	+0.004
9B AdamW	100	0.4956	0.6943	+0.007
9B AdamW	150	–	0.6730	+0.018
9B AdamW	200	–	0.6649	+0.011
9B AdamW	250	–	0.6765	−0.019
9B AdamW	296 (final)	0.4358	0.6501	+0.027
27B AdamW	200	–	0.6625	+0.002
27B AdamW	296 (final)	0.4246	0.6640	−0.029

The 9B reproduces the "loss keeps falling, held-out quality degrades" pattern seen in the generation runs — but peaks much earlier (~34% of training vs ~75%). The 27B shows no late-training degradation, so its final checkpoint is published.

Evaluator optimizer ablation

A 9B run with Muon + pure_bf16 (identical data, batch, and steps) failed badly:

	Muon + pure_bf16	AdamW + ZeRO-3
Final train loss	0.7505	0.4358
Truncated generations	14 / 389	0 / 389
Cohen's κ	0.4831	0.6353
Quality ordering	❌ inverted	✅ correct

The Muon arm stalled early (loss 1.376 → 1.138 over 50 steps) and degenerated into repetition loops hitting the 8192-token cap. It is not published. Do not use Muon + pure_bf16 for this task — note this differs from the generation family, where Muon is viable.

Shared training setup

Generation models

• Objective: full-parameter supervised fine-tuning (no LoRA / adapters)
• Dataset: designcoder_sft_v2_train in ShareGPT format (see revision table above)
• Chat template: qwen3_5 with thinking enabled
• Context length: 32,768
• Sequence packing: enabled, with neat packing (no cross-sample attention)
• LR schedule: cosine, warmup ratio 0.1

Evaluator models

• Objective: full-parameter SFT, vision tower frozen (LM + multimodal projector tuned)
• Dataset: DesignCoder-evaluate, 37,847 samples, one screenshot per sample
• Chat template: qwen3_5 with thinking enabled; context length 32,768
• image_max_pixels: 1,048,576 — inputs must be downscaled the same way at inference
• Precision / parallel: bf16 + DeepSpeed ZeRO-3; 2 epochs, global batch 256
• LR schedule: cosine, warmup ratio 0.1

Usage

Generation models

from transformers import AutoModelForCausalLM, AutoProcessor

repo = "xingxm/DesignCoder"
subfolder = "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400"

model = AutoModelForCausalLM.from_pretrained(repo, subfolder=subfolder, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(repo, subfolder=subfolder)

To download a single checkpoint only:

hf download xingxm/DesignCoder --include "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400/*" --local-dir ./DesignCoder

Inference contract

These models are trained as tool-using agents, not single-turn generators. A case runs design_search → (websearch, landing only) → a final answer containing exactly three code blocks in the order html, css, js. Reproduce the system prompts and tool observation format from examples/designcoder/runtime/infer_designcoder.py; prompting with a bare instruction and no tool turns does not match the training distribution and will score far below the numbers above.

Evaluator models

Note the different loader class (AutoModelForImageTextToText) — AutoModelForCausalLM resolves to a text-only shell and generate() will reject the image tensors.

from transformers import AutoModelForImageTextToText, AutoProcessor, AutoTokenizer
from PIL import Image
import torch, math

repo = "xingxm/DesignCoder"
subfolder = "designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296"

tok = AutoTokenizer.from_pretrained(repo, subfolder=subfolder, trust_remote_code=True)
proc = AutoProcessor.from_pretrained(repo, subfolder=subfolder, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo, subfolder=subfolder, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
)

img = Image.open("screenshot.png").convert("RGB")
w, h = img.size                                   # match training preprocessing
if w * h > 1048576:
    s = math.sqrt(1048576 / (w * h))
    img = img.resize((int(w * s), int(h * s)))

user = (
    "<image>\n"
    "Evaluate the attached UI screenshot using the selected visual criteria.\n\n"
    "<surface>\nlanding\n</surface>\n\n"
    "<generation_brief>\n...brief...\n</generation_brief>\n\n"
    '<frozen_rubric>\n{"rubric":{"Alignment":["..."],"Layout":["..."]}}\n</frozen_rubric>'
)
text = tok.apply_chat_template(
    [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}],
    tokenize=False, add_generation_prompt=True,
).replace("<image>", "<|vision_start|><|image_pad|><|vision_end|>")

inputs = proc(text=[text], images=[img], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Evaluator inference contract

• SYSTEM_PROMPT must be the one shipped with the DesignCoder-evaluate dataset — it enumerates the rubric namespace the model was trained against.
• The chat template already ends with <think>\n, so generated text continues inside the reasoning block and carries no opening <think> tag. Split on </think> to reach the JSON payload.
• Downscale images to ≤ 1,048,576 px as shown; larger inputs drift from training.
• Parse frozen_dynamic_scores.verdicts. Ignore the sibling summary string — it miscounts (e.g. reads "24/24" next to 25 emitted verdicts) in both models.
• static_scores / dynamic_scores (40 items, 0/1/2) have no held-out validation — they were only checked against contaminated training samples. Treat them as unverified.

Provenance

Each subfolder additionally ships trainer_state.json / trainer_log.jsonl (and training_loss.png where available) so that the loss curve and exact step schedule of the run can be recovered from the checkpoint itself.