mlboydaisuke/OvisOCR2-LiteRT

OvisOCR2 — LiteRT-LM

ATH-MaaS/OvisOCR2 (0.85B, apache-2.0) converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15.

OvisOCR2 is a page-level document-parsing model built by post-training Qwen3.5-0.8B: given a document page image it emits a Markdown transcription in reading order — text as Markdown, tables as HTML <table>, formulas as LaTeX. This package wires the checkpoint's own 12-layer ViT to LiteRT-LM's fast_vlm contract at a static 512×512 input, on the same rail as our litert-community/Qwen3.5-0.8B VL build (the two checkpoints share every config byte; only the weights differ).

File	Recipe	Size
OvisOCR2_int8.litertlm	int8 decoder (dynamic on linears + embedding; convs and the delta rule stay float, fp32 activations declared) + fp16 vision encoder / int8 adapter, static 512×512, six-signature prefill ladder	1.30 GB

What it does on-device (measured)

Transcription of a rendered table page (512×512), CPU backend, greedy — output shown verbatim:

Quarterly Sales Report 2025

This report summarizes unit sales for the first three quarters. Phones remain the
strongest product line across all quarters.

<table border="1"><tr><td>Product</td><td>Q1</td>...</table>

Table 1: Unit sales by product and quarter.

• Table page: every one of the 20 table cells, the title, the paragraph and the caption are transcribed exactly (verified against the page's source text), and the output matches the HF fp32 reference at similarity 1.0000 on the same 512×512 input.
• The GPU backend produces byte-identical output to CPU on this page.
• Formula page: the LaTeX is exact (an inverted Planck law renders symbol-for-symbol) — similarity 1.0000 against the 1-D-position fp32 reference, 0.9968 against the full M-RoPE reference (a one-token $B$/B difference, the position-contract cost).
• Dense academic pages (9 pt text) exceed what a 512×512 input can resolve: the opening is transcribed correctly, then the model invents and eventually repeats. The HF fp32 original does the same on the same input (measured: every variant loops on this page; this build matches the fp32 reference byte-for-byte through the entire legible region and diverges only inside the invented continuation). Upstream's own reference inference code ships a repeat-cleanup post-processor for exactly this behavior. Use the model for pages whose text is legible at 512×512, or tile.
• A note for anyone comparing against stock transformers: the upstream repo has no generation_config.json, so HF's eos is only <|endoftext|> and fp32 generation runs through the turn end and rambles. This bundle declares both <|endoftext|> and <|im_end|> as stop tokens and stops cleanly; add 248046 to eos_token_id on the HF side before comparing.

Scope: this is a document parser, not a chat model

The OCR post-training visibly erodes general chat ability: on a generic 8-question sanity gate the model tends to transcribe the prompt instead of answering it. The fp32 original on the stock HF stack fails the same questions the same way — echoing 17 + 25 back, looping Tokyo, drifting into a numbered list on a French-vocabulary question — measured question-by-question against this build. It is the checkpoint's behavior, not a conversion artifact. Feed it document pages with the transcription prompt below.

One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). A probe asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac). On the same Engine, the _int8 file's replies in the second conversation repeated "The capital of France?" until the cap, in both of its runs (a probe is run a second time when its first run shows a difference or when the fresh Engine misses one of the three expected answers: Paris, 391, blue). On a fresh Engine, its replies began "Paris", "17" and "The quick brown fox jumps over the lazy dog.", and each then repeated the fox sentence until the cap. The probe's questions are outside this model's task: it is a document parser that tends to transcribe a prompt instead of answering it. Create a new Engine for each conversation. The first conversation on an Engine is unaffected.

int8 on the CPU. LiteRT-LM's CPU path quantizes the input of every int8 linear layer to int8 at run time (XNNPACK dynamic quantization). This file keeps the decoder recipe of the Qwen3.5-0.8B-VL_int8.litertlm file in litert-community/Qwen3.5-0.8B published until 2026-10-12 (the delta-rule decay/gate projections int8; that repo re-quantized its files that day). For this file, the CPU transcriptions above (exact on the table page, where the CPU and GPU backends also produce byte-identical output) are the check on that path. A second check asks "What is 17 times 23?" because the sibling Qwen3.5-2B int8 file looped on that question on the CPU path until its 2026-10-11 re-quantization (details on its card). In 3 fresh CPU processes, the _int8 file repeated text until the 400-token cap each time without giving 391 (litert-lm 0.18.0, Mac CPU). That question is outside this model's task: on the sanity gate above the model echoes or loops on such prompts, and so does its fp32 original on the stock Hugging Face stack.

8 GB-class phones and Mali. On Android, this file's GPU engine has been built and run on a Galaxy S26 (Adreno, MemTotal 11,389,756 KiB) only. The transcription itself was not checked on that phone. No Mali phone has run this file. The Qwen3.5-0.8B-VL_int8.litertlm file in litert-community/Qwen3.5-0.8B published until 2026-10-12 shares this file's decoder recipe and six-signature prefill ladder. On an 8 GB Pixel 8a (Arm Mali, MemTotal 7,754,700 KiB), that file's GPU engine creation reached 4 of the 6 prefill signatures before free memory fell below 400,000 KiB, the test's stop line (the reading that stopped it was 300,460 KiB). Whether that build completes without the stop is not measured. So this file's GPU engine is not expected to fit an 8 GB-class phone — an expectation carried over from that file, not a measurement of this one. Use the CPU backend on 8 GB phones.

Tool calls. This bundle's chat template has no tools section, so tool definitions never reach the prompt. On litert-lm 0.18.0 the runtime also parses no Qwen3.5 tool call: such a call comes back as plain text and tool_calls stays empty (my open report). In a test of this file with a tool supplied and the runtime's automatic tool calling turned on, no tool ran (litert-lm 0.18.0 on a Mac).

Usage

litert-lm run ./OvisOCR2_int8.litertlm \
  --attachment page_512.png \
  --prompt "
Extract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <img src=\"images/bbox_{left}_{top}_{right}_{bottom}.jpg\" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: <table>...</table>. Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing." \
  --backend cpu --cache no --temperature 0

The prompt is the upstream reference prompt verbatim. The runtime resizes the attachment to 512×512; pre-rendering your page at 512×512 keeps the aspect ratio under your control.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	2175 tok/s	140.9 tok/s	0.13 s
CPU	659 tok/s	48.9 tok/s	0.43 s

On device (iPhone 17 Pro, iOS 27, litert-lm v0.16.0 vendored build, cold start, single runs):

Backend	Decode	TTFT	Peak memory
GPU (Metal)	48.5 tok/s	1.5 s	3.8 GB
GPU (Metal), image leg	64.7 tok/s	1.5 s	3.8 GB
CPU	18.0 tok/s	1.0 s	0.9 GB

The six-signature ladder loads Metal cleanly (no jetsam, no maxNumTokens override), the same envelope as the Qwen3.5-0.8B VL build. On-device image legs are grounded: shown a poster, it starts transcribing the poster's title; shown a photo without text, it emits its picture-region <img> tag — exactly its document convention.

Conversion notes

• The decoder is the shipped Qwen3.5-0.8B gated-delta hybrid rail (18 linear-attention + 6 full-attention layers, 48 state buffers); the MTP head is dropped.
• The vision tower's fp16-safe LayerNorm scales are calibrated per checkpoint; OvisOCR2's differ from base 0.8B in magnitude (absmax 2803.9 vs 2390.9) though not in shape.
• Vision quantization was gated on document fixtures (a table page, a formula page and an arXiv page), not photographs: the ship shape (fp16 encoder + int8 adapter) holds end-to-end correlation 0.9998+ on all three.
• Stop tokens are <|endoftext|> (248044) and <|im_end|> (248046); the upstream repo has no generation_config.json, so both were taken from config.json + the tokenizer and verified against the tokenizer at bundle build.

Conversion scripts and a step-by-step reproduction: hf-to-litertlm (qwen35vl_work/).