mmnga-o/Qwen3.8-REAP30-Width50-UD-gguf

Qwen3.8 REAP30 Width50 UD GGUF

日本語

An experimental reduced GGUF build of Qwen3.8-2.4T-A95B.

The routed-expert count is reduced by about 30%, then the intermediate width of each retained expert is reduced by 50%. The number of experts selected per token remains 10.

Calibration and selection

The calibration set contains 97 completed English and Japanese samples covering general chat and reasoning, code generation, and tool use. It contains 94,874 profiled tokens, of which 78,061 are semantic assistant tokens used for scoring. Prompt and structure tokens are excluded.

REAP30: expert axis

For expert e, the score is:

REAP(e) = mean[t routed to e](
  top-10-renormalized router weight(t,e)
  × ||unweighted expert output(t,e)||₂
)

Experts are ranked independently in each of the 92 MoE layers. The top 358 of 512 are retained. No experts are reserved manually.

Width50: intermediate axis

After the REAP30 slice, the reduced model is profiled again with the same calibration set. Every intermediate channel is scored independently for each expert:

a(t,e,c)     = SiLU(gate(t,e,c)) × up(t,e,c)
score(e,c)   = Σ[t routed to e](router_weight(t,e) × a(t,e,c))²
               × ||Wdown,e[:,c]||₂²

The top 1,024 of 2,048 channels are retained for each expert. Low-coverage experts use a blend of the activation score and a static weight-energy prior.

GGUF changes

	Original	This build
MoE layers	92	92
Routed experts per layer	512	358
Routed expert FFN width	2,048	1,024
Experts selected per token	10	10

The router and grouped expert tensors are sliced with the same per-layer expert map. Shared experts are retained. MTP tensors are omitted.

Width selection is channel-level rather than 256-channel-block-level. Retained gate/up rows are copied directly. ffn_down_exps.weight is dequantized, gathered by the selected channel map, packed into new QK256 blocks, and requantized to the source GGUF type.

Both folders use the same REAP and Width rankings.

Files

Folder	Shards	Size
UD-IQ1_S	12	about 184.72 GiB
UD-Q1_0	10	about 148.54 GiB

Load the first shard in the selected folder. Each folder also includes slice_manifest.json with the applied ranking hashes and slice settings.

Usage

UD-Q1_0 uses the experimental IQ1_XXXS GGML type. Use the iq1-narrow branch:

git clone --branch iq1-narrow https://github.com/unslothai/llama.cpp
cmake -S llama.cpp -B llama.cpp/build \
  -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build -j --target llama-server

Example for a 32 GB GPU while keeping MoE weights on CPU:

./llama.cpp/build/bin/llama-server \
  -m ./UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \
  --cpu-moe \
  -ngl 80 \
  --ctx-size 8192 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

This configuration was tested with 512 GB system RAM and one 32 GB RTX 5090. It offloaded 80 of 92 layers and used about 30.6 GiB of VRAM. Adjust -ngl and context size for your hardware.

This is a heavily reduced experimental model. Quality and stability may differ from the original model and depend on how closely the calibration distribution matches the target workload.