NextTransformer/MoR-49B-Tokens

NextTransformer MoR

Decide which tokens continue at each recursion boundary.

This is the final MoR checkpoint in the NextTransformer 49.152B-token architecture comparison. It has 927,994,042 parameters and a 2,048-token context. The model was pretrained on ClimbMix for 46,875 updates. The 49B in the repository name refers to training tokens.

Final evaluation	Result
Validation BPB	0.6838
CORE-22 (×100)	30.37

Architecture · Run record

Load the weights

Install the tested NextTransformer implementation and the safetensors reader in a Python 3.10–3.13 environment.

pip install "nexttransformer @ git+https://github.com/Interactive-Active-AI/NextTransformer.git@7aa267a88b45e4a19a9739ccb4ba3f196d7aacc0" safetensors

The example encodes a prompt with the checkpoint's tokenizer and prints its most likely next token.

import base64
import json
from pathlib import Path

import tiktoken
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from nexttransformer.models import build_model, config_from_dict

root = Path(snapshot_download("NextTransformer/MoR-49B-Tokens"))
config = config_from_dict(json.loads((root / "model_config.json").read_text()))
model = build_model(config, device="cpu")
model.load_state_dict(load_file(root / "model.safetensors"), strict=True)
if hasattr(model, "set_step"):
    model.set_step(46875)
model.eval()

spec = json.loads((root / "tokenizer.tiktoken.json").read_text())
encoding = tiktoken.Encoding(
    name=spec["name"],
    pat_str=spec["pat_str"],
    mergeable_ranks={base64.b64decode(b): rank for b, rank in spec["mergeable_ranks"]},
    special_tokens=spec["special_tokens"],
)
prompt = [encoding.encode_single_token("<|bos|>")] + encoding.encode_ordinary("The sky is")
with torch.inference_mode():
    logits = model(torch.tensor([prompt]))
print(encoding.decode([int(logits[0, -1].argmax())]))