Inkling 2 Mini GGUF

MirrorNeuronLab independently converted Thinking Machines Lab's original Inkling 2 Mini checkpoint. The conversion decoded its packed NVFP4 and MXFP8 expert weights into a BF16 GGUF intermediate. The Q4_K_M build uses the standard llama.cpp preset without an importance matrix. The BF16 projector provides image input. This release excludes the optional MTP draft model. Original checkpoint

The AD builds were quantized from the same BF16 weights with AtomicChat's published calibration matrix and per-tensor rules adapted to upstream llama.cpp. Expert gate and up weights use IQ1_M or IQ1_S. Expert down weights use standard Q2_0 blocks with calibration-weighted scales. The loop controllers are kept in F32. Across all text tensors, the models average 2.25 and 2.13 bits per weight, respectively. AtomicChat calibration and recipes

File GGUF size Download with vision
Inkling-2-Mini-AD-IQ1_M.gguf 7.35 GB 7.99 GB
Inkling-2-Mini-AD-IQ1_S.gguf 6.96 GB 7.61 GB
Inkling-2-Mini-Q4_K_M.gguf 17.48 GB 18.12 GB
mmproj-Inkling-2-Mini-BF16.gguf 642 MB Shared projector

Verify the files with sha256sum -c SHA256SUMS. The provenance records the checkpoint revision, conversion commands, and tensor types. SHA256SUMS ยท Provenance

Docker Model Runner

Use a Docker Model Runner llama.cpp backend with Inkling 2 Mini (inkling2_loop) support. Support first landed in build b12600. The backend build is independent of the docker model CLI version. llama.cpp release

The Q4_K_M build passed CUDA text and image smoke tests at a 4,096-token context on Docker Model Runner v1.2.9 and an NVIDIA GB10. Q4_K_M public download validation ยท Q4_K_M local package validation

Both AD builds passed the same CLI text and API image tests on CUDA at 4,096 tokens. AD public download validation ยท AD local package validation

Docker Model Runner v1.2.9 leaves the summary columns blank in docker model ls for the AD files. The table gives the measured download sizes.

MODEL=hf.co/MirrorNeuronLab/Inkling-2-Mini-GGUF:AD-IQ1_M
docker model pull "$MODEL"
docker model configure --context-size 4096 "$MODEL"
docker model run "$MODEL" "What is 17 multiplied by 23? Reply with only the number."

Set the tag to AD-IQ1_S for the smaller variant or Q4_K_M for the four-bit build. Docker pulls the corresponding model and the shared vision projector.

On Apple Silicon, use Docker Desktop's native llama.cpp Metal engine with Inkling support (b12600 or newer). That build includes Metal kernels for IQ1_M, IQ1_S, and Q2_0 expert matrix multiplies. Hardware validation for this release was performed on NVIDIA CUDA. Docker Model Runner engine support ยท Metal kernel source

On Linux with NVIDIA GPUs, you can run a separate Docker Model Runner with a compatible CUDA build. Build llama.cpp b12600 for your GPU with -DGGML_CUDA=ON, then pass the build's bin directory to the included launcher. It listens on localhost port 12438 by default. Launcher

bash reproduce/start-docker-runner.sh /path/to/llama.cpp/build-cuda/bin
export MODEL_RUNNER_HOST=http://127.0.0.1:12438

Run the pull, configure, and run commands above in that shell. Image input uses the image_url content type on Docker Model Runner's OpenAI-compatible chat completions endpoint. Docker Model Runner API

llama.cpp

llama-server -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M --jinja -c 4096

When using downloaded files, pass -m Inkling-2-Mini-Q4_K_M.gguf --mmproj mmproj-Inkling-2-Mini-BF16.gguf in place of -hf to enable image input.

Provenance and license

Thinking Machines Lab, Inc. developed the original model and released its weights under Apache 2.0. MirrorNeuronLab converted the model to GGUF and quantized its text weights. See the upstream model card for model capabilities and limitations. License ยท Upstream model card ยท Thinking Machines Lab Model Acceptable Use Policy

To reproduce the AD files, build llama-quantize at b12600 after applying the Q2_0 quantizer patch. Use the pinned importance matrix, the per-tensor rule files, and the commands in the provenance. The patch changes offline quantization only. Provenance ยท Quantizer patch ยท llama.cpp patch license

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2_loop
Hardware compatibility
Log In to add your hardware

1-bit

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for MirrorNeuronLab/Inkling-2-Mini-GGUF

Quantized
(129)
this model