Speck1-140M

Speck1-140M is a 140.7M parameter English base language model that interleaves global grouped-query attention with gated causal convolution. It was pretrained from scratch on 5B tokens.

This is a base model, not instruction-tuned or specialized in any way. It has no chat template and no safety alignment.

Summary

Property	Value
Parameters	140,652,288
Training tokens	5.0B
Training sequence length	2,048
Configured max context	4,096 (unvalidated beyond 2,048)
Vocabulary	32,000 (Mistral v0.1 SentencePiece)
Release format	BF16 Safetensors
Validation loss / perplexity	2.3655 / 10.649
BananaMind Base Bench Elo	965
CPU decode, batch 1	55.1 tok/s
RTX 3090 decode, batch 1	247.3 tok/s

Architecture

18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.

Component	Value
Hidden width	768
Embedding width	640
SwiGLU intermediate	2,304
Attention heads (Q / KV)	12 / 3
Head dimension	64
Conv inner width	384
Conv kernel sizes	3, 5
RoPE theta	10,000
RMSNorm epsilon	1e-5

Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream via learned projections.

Usage

Speck1-140M works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trust_remote_code=True when loading it.

pip install "transformers==5.1.0" torch sentencepiece safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "specklabs/Speck1-140M"
device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(device)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

output = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)
generated = output[0, inputs.input_ids.shape[1] :]
print(tokenizer.decode(generated, skip_special_tokens=True))

The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support unpadded batches.

Training

Setting	Value
Optimizer steps	76,294
Tokens per step	65,536
Sequence length	2,048
Peak LR	1.5e-3 (cosine decay, 512-step warmup)
Weight decay	0.1
Gradient clipping	1.0
Training time	25.35 hours
Estimated compute	4.97 EFLOP

Muon optimized 2D matrix parameters; AdamW (β 0.9/0.95, ε 1e-8) handled embeddings, norms, and conv kernels.

Training data

Source	Tokens	Share
Ultra-FineWeb	1.98B	39.5%
DCLM Baseline 1.0	1.55B	31.0%
Cosmopedia v2	0.67B	13.4%
FineMath-4+	0.47B	9.5%
Ultra-FineWeb-L3	0.33B	6.6%

The mix shifted toward more Cosmopedia, FineMath, and Ultra-FineWeb-L3 in later training phases. Data was globally deduplicated (128-bit BLAKE2b) and filtered to 200-100,000 characters after NFKC normalization. The included tokenizer is the pinned 32k token Mistral v0.1 SentencePiece model and remains under its upstream Apache-2.0 terms, reproduced in LICENSE.tokenizer.

Evaluation

Scored on all 350 items of BananaMind Base Bench 1.1 using mean conditional token log-probability.

Category	Elo	Accuracy	Weighted acc.
Language completion	982	58.0%	60.8%
Commonsense	953	46.0%	49.8%
World knowledge	906	42.0%	43.3%
Context tracking	899	38.0%	36.6%
Quantitative	799	20.0%	19.6%
Logical reasoning	1019	38.0%	39.2%
Code completion	1157	56.0%	58.6%
Overall	965	42.57%	43.46%

vs. similarly-sized models

Model	Params	Training tokens	Elo	Accuracy	CPU prefill	CPU decode	RTX 3090 prefill	RTX 3090 decode	BF16 memory @2K	BF16 state @2K
BananaMind-2-Pro	139M	100B	1131	67.14%	2,190 tok/s	43.0 tok/s	64,060 tok/s	140.3 tok/s	325.1 MiB	60.0 MiB
SmolLM2-135M	135M	~2T	1119	66.29%	2,201 tok/s	47.4 tok/s	64,814 tok/s	157.7 tok/s	301.6 MiB	45.0 MiB
GPT-X2.5-135M	135M	75B	1106	64.57%	2,042 tok/s	47.2 tok/s	55,346 tok/s	125.0 tok/s	302.6 MiB	45.0 MiB
Supra2-100M-Base	101M	30B	1030	56.29%	3,362 tok/s	56.0 tok/s	113,326 tok/s	298.1 tok/s	216.0 MiB	24.0 MiB
Speck1-140M	141M	5B	965	42.57%	2,252 tok/s	55.1 tok/s	74,323 tok/s	247.3 tok/s	281.3 MiB	12.0 MiB

Reference models saw 6-400x more training tokens, so this is a parameter-adjacent comparison, not a compute-matched one.

Inference speed

Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported throughput is calculated from the median duration.

Memory is unique live BF16 model tensor storage plus cache/state tensor storage after a 2,048-token prefill at batch 1. It excludes framework RSS, CUDA allocator reservations, and temporary operator workspace. FP32 CPU tensor memory is approximately twice the reported BF16 memory. For another context length N, approximate memory as model tensor memory + State@2K × N / 2,048; Speck's small convolution history is fixed rather than context-scaled.

Speck1-140M ranks 2nd among the compared models on both CPU and GPU batch-1 prefill and decode throughput, behind the smaller Supra2-100M-Base. The deduplicated BF16 Safetensors release is 281.3 MB.

Limitations

• Not instruction-tuned: can't reliably follow requests.
• No safety alignment: can produce biased, harmful, or incorrect text.
• Weak at arithmetic and quantitative reasoning.
• Mostly English: multilingual ability untested.
• Validated only up to 2,048 tokens despite a 4,096-token config.
• Trained on web-derived data that may contain bias, errors, or copyrighted text.
• No red-team or misuse evaluation performed.

Reproducibility

Full training and eval code: github.com/alkinun/speck

Citation

@misc{alkinun2026speck1,
  author       = {alkinun},
  title        = {Speck1-140M: A Compact Hybrid Attention-Convolution Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/specklabs/Speck1-140M}},
  url          = {https://github.com/alkinun/speck}
}