Instructions to use MirrorNeuronLab/Inkling-2-Mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Use Docker
docker model run hf.co/MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "MirrorNeuronLab/Inkling-2-Mini-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MirrorNeuronLab/Inkling-2-Mini-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
- Ollama
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with Ollama:
ollama run hf.co/MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with Docker Model Runner:
docker model run hf.co/MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
- Lemonade
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Inkling-2-Mini-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use MirrorNeuronLab/Inkling-2-Mini-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling 2 Mini GGUF
MirrorNeuronLab independently converted Thinking Machines Lab's original Inkling 2 Mini checkpoint. The conversion decoded its packed NVFP4 and MXFP8 expert weights into a BF16 GGUF intermediate. The Q4_K_M build uses the standard llama.cpp preset without an importance matrix. The BF16 projector provides image input. This release excludes the optional MTP draft model. Original checkpoint
The AD builds were quantized from the same BF16 weights with AtomicChat's published calibration matrix and per-tensor rules adapted to upstream llama.cpp. Expert gate and up weights use IQ1_M or IQ1_S. Expert down weights use standard Q2_0 blocks with calibration-weighted scales. The loop controllers are kept in F32. Across all text tensors, the models average 2.25 and 2.13 bits per weight, respectively. AtomicChat calibration and recipes
| File | GGUF size | Download with vision |
|---|---|---|
| Inkling-2-Mini-AD-IQ1_M.gguf | 7.35 GB | 7.99 GB |
| Inkling-2-Mini-AD-IQ1_S.gguf | 6.96 GB | 7.61 GB |
| Inkling-2-Mini-Q4_K_M.gguf | 17.48 GB | 18.12 GB |
| mmproj-Inkling-2-Mini-BF16.gguf | 642 MB | Shared projector |
Verify the files with sha256sum -c SHA256SUMS. The provenance records the checkpoint revision, conversion commands, and tensor types. SHA256SUMS ยท Provenance
Docker Model Runner
Use a Docker Model Runner llama.cpp backend with Inkling 2 Mini (inkling2_loop) support. Support first landed in build b12600. The backend build is independent of the docker model CLI version. llama.cpp release
The Q4_K_M build passed CUDA text and image smoke tests at a 4,096-token context on Docker Model Runner v1.2.9 and an NVIDIA GB10. Q4_K_M public download validation ยท Q4_K_M local package validation
Both AD builds passed the same CLI text and API image tests on CUDA at 4,096 tokens. AD public download validation ยท AD local package validation
Docker Model Runner v1.2.9 leaves the summary columns blank in docker model ls for the AD files. The table gives the measured download sizes.
MODEL=hf.co/MirrorNeuronLab/Inkling-2-Mini-GGUF:AD-IQ1_M
docker model pull "$MODEL"
docker model configure --context-size 4096 "$MODEL"
docker model run "$MODEL" "What is 17 multiplied by 23? Reply with only the number."
Set the tag to AD-IQ1_S for the smaller variant or Q4_K_M for the four-bit build. Docker pulls the corresponding model and the shared vision projector.
On Apple Silicon, use Docker Desktop's native llama.cpp Metal engine with Inkling support (b12600 or newer). That build includes Metal kernels for IQ1_M, IQ1_S, and Q2_0 expert matrix multiplies. Hardware validation for this release was performed on NVIDIA CUDA. Docker Model Runner engine support ยท Metal kernel source
On Linux with NVIDIA GPUs, you can run a separate Docker Model Runner with a compatible CUDA build. Build llama.cpp b12600 for your GPU with -DGGML_CUDA=ON, then pass the build's bin directory to the included launcher. It listens on localhost port 12438 by default. Launcher
bash reproduce/start-docker-runner.sh /path/to/llama.cpp/build-cuda/bin
export MODEL_RUNNER_HOST=http://127.0.0.1:12438
Run the pull, configure, and run commands above in that shell. Image input uses the image_url content type on Docker Model Runner's OpenAI-compatible chat completions endpoint. Docker Model Runner API
llama.cpp
llama-server -hf MirrorNeuronLab/Inkling-2-Mini-GGUF:Q4_K_M --jinja -c 4096
When using downloaded files, pass -m Inkling-2-Mini-Q4_K_M.gguf --mmproj mmproj-Inkling-2-Mini-BF16.gguf in place of -hf to enable image input.
Provenance and license
Thinking Machines Lab, Inc. developed the original model and released its weights under Apache 2.0. MirrorNeuronLab converted the model to GGUF and quantized its text weights. See the upstream model card for model capabilities and limitations. License ยท Upstream model card ยท Thinking Machines Lab Model Acceptable Use Policy
To reproduce the AD files, build llama-quantize at b12600 after applying the Q2_0 quantizer patch. Use the pinned importance matrix, the per-tensor rule files, and the commands in the provenance. The patch changes offline quantization only. Provenance ยท Quantizer patch ยท llama.cpp patch license
- Downloads last month
- -
1-bit
4-bit
Model tree for MirrorNeuronLab/Inkling-2-Mini-GGUF
Base model
thinkingmachines/Inkling-2-Mini