facebook/MobileMoE-S-Base

MobileMoE-S (Base) Model Card

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-S-Base repository — the pre-trained 0.3B-active base model.

	S	M	L
Active / total params	272M / 1.3B	528M / 2.8B	922M / 5.3B
Layers	20	26	32
Model dimension	768	1024	1280
Heads (Q / KV)	12 / 4	16 / 4	20 / 4
Routed experts	60	60	60
Top-k	4	4	4
INT4 weight memory	0.68 GB	1.48 GB	2.75 GB
Base	MobileMoE-S-Base	MobileMoE-M-Base	MobileMoE-L-Base
SFT	MobileMoE-S-SFT	MobileMoE-M-SFT	MobileMoE-L-SFT
QAT (INT4)	MobileMoE-S-QAT	MobileMoE-M-QAT	MobileMoE-L-QAT

For the detailed technical report: 📝 MobileMoE: Scaling On-Device Mixture of Experts

For more versions, check out the 🤗 MobileMoE Collection

MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute F_inf = 2N_act (GFLOPs) and (b) total parameters N_total (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB).

Key Features

• A new Pareto frontier for on-device LLMs. Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
• Scaling-law-derived architecture. The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile memory and compute constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
• Four-stage recipe. Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.

Model Information

Model: MobileMoE-S-Base (pre-trained + mid-trained)
Active Parameters: 272M
Total Parameters: 1.3B
Layers: 20
Model Dimension: 768
Attention Heads: 12
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 384 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 1536)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Input Modality: Text
Output Modality: Text
Languages: English
Training Stages: Pre-training → mid-training
Context Length: 8,192 tokens
Precision: BF16
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed

Results

All numbers below are for the base (pre-trained) models, re-evaluated under identical settings with greedy decoding using lm-eval; few-shot counts are given in parentheses after the benchmark name, and benchmarks shown without one are evaluated 0-shot.

Foundational benchmarks

Capability	Benchmark	Gemma 3 270M	SmolLM2 360M	MobileMoE-S
Active / total params		270M	362M	272M / 1.3B
Commonsense Reasoning	HellaSwag	41.4	56.5	58.9
	PIQA	68.3	71.7	75.4
	SIQA	40.2	40.7	46.8
	WinoGrande	53.7	59.0	58.6
Knowledge	MMLU (5-shot)	26.7	25.2	43.7
	NaturalQuestions (5-shot)	4.1	7.4	12.6
	TriviaQA (5-shot)	14.3	26.8	33.2
Science	ARC-Challenge (25-shot)	29.4	40.5	46.5
	ARC-Easy	56.8	68.1	73.9
	OpenBookQA	30.4	37.6	34.6
Reading	BoolQ	58.3	61.8	60.2
	DROP (3-shot)	14.2	17.9	39.0
Reasoning	BIG-Bench Hard (3-shot)	29.5	31.7	31.8
	GSM8K (8-shot)	1.8	5.3	36.2
Average		33.5	39.3	46.5

Training

MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 2 (mid-training).

	Pre-training	Mid-training	SFT	QAT
Context length	2,048	8,192	8,192	8,192
Total tokens	~6T	~500B	~126B	~21B
Peak learning rate	4×10-4	4×10-5	4×10-6	4×10-6
LR schedule	Cosine	Linear	Cosine	Cosine
Token dispatch	drop-and-pad	drop-and-pad	dropless	dropless

How to use

MobileMoE uses a custom architecture (model_type: mobilemoe) that is not yet part of upstream transformers, so trust_remote_code=True is required. The modeling code ships in this repo (configuration_mobilemoe.py, modeling_mobilemoe.py).

Requirements

pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"

Verified with the following versions:

Package	Version
torch	2.8.0 (cu128)
transformers	4.57.6
tokenizers	0.22.2
safetensors	0.7.0
accelerate	1.13.0

For batch evaluation we recommend vLLM (≥ 0.10.2) with enforce_eager=True.

Text generation

This is a base model — it has no chat template and is not instruction-tuned. Prompt it with plain text continuation:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "facebook/MobileMoE-S-Base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

prompt = "Why are open-source on-device language models great?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=64,
    do_sample=False,
    temperature=None,
    top_p=None,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Known issues. Loading the tokenizer on transformers 4.57.6 prints a fix_mistral_regex=True warning. Please ignore it and do not set the flag, as MobileMoE uses the Llama-3 tokenizer whose default tokenization is already correct.

Citation

@article{chen2026mobilemoe,
  title={MobileMoE: Scaling On-Device Mixture of Experts},
  author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
  journal={arXiv preprint arXiv:2605.27358},
  year={2026}
}

License

MobileMoE is distributed under the FAIR Noncommercial Research License.