The Dedicated OCR Engine Lost to the General-Purpose Model — 300 Slower

Originally published on hexisteme notes.

I had a 27B vision model running locally (IQ4_XS quantized, 15GB resident) and needed to decide
whether it was worth using for OCR. The comparison was macOS's built-in Vision framework
(VNRecognizeTextRequest) — a dedicated text-recognition engine, free, zero memory
footprint.

My expectation going in: the specialist wins on character accuracy, and the general-purpose
model is reserved for when you need semantic understanding. Slow and expensive, use sparingly.

That expectation was wrong, and it was wrong in a way that would have been invisible in
production.

Method: an image whose answer I already knew

The usual mistake in an OCR comparison is measuring against real documents, where you don't
have ground truth. Then you can't distinguish plausible output from correct output — and
plausible output is exactly what both engines produce when they fail.

So I rendered a 1100×720 test image with the answer fixed in advance:

• A title and date in Korean, a 4-column × 3-row table (model / memory / speed / status), four lines of prose
• One adversarial line: A0-1lO9-B8 — digit 1 next to lowercase l, capital O next to digit 0
• Two empty table cells containing -

Then I looked at it. The first render was wrong — a label came out as tofu boxes (□□),
because the monospace font had no Korean glyphs. If ground truth is broken at the moment you
fix it, every measurement afterwards is void. That check costs thirty seconds and it's the
whole experiment.

The results

Vision's eight errors: IQ4_XS→I04_XS, cloud→cLoud (twice), tok/s→tok/5, two
characters inside a code string, em dash —→-, arrow →→->.

300× faster. On accuracy alone, four times the error rate on a document of this size is
arguably a fine trade.

Accuracy alone is not what decided it.

The difference was structural, not lexical

Vision returned the table decomposed by column. Three model names in a row, then three
memory figures, then the speed and status columns appended at the end of the document.

Which means: you cannot recover which speed belongs to which model from the output. The
row associations are gone. Not garbled — gone. The characters are all there, correctly
grouped, in a well-formed sequence, and the relation between them has evaporated.

The VLM kept the rows. Next to that, 2 errors versus 8 is a rounding difference.

What this actually says is narrower than "the general model is better":

"A dedicated tool beats a general one" depends entirely on where you cut the task.

Vision is dedicated to character recognition. It is not dedicated to document
understanding. My task needed the second and I was picking tools by the first one's benchmark.
The specialist was genuinely better at the thing it specializes in — I had just mislabeled what
I needed.

Why this breaks the cheap-first fallback

The obvious architecture is: run the cheap engine, detect failure, escalate to the expensive
one. Almost everyone reaches for this.

It requires failure to be detectable.

Column-shredded output is syntactically perfect. It has plausible text, plausible structure,
no error signal of any kind. Downstream, it is indistinguishable from a correct read. The
information didn't get corrupted — it got dropped, and dropped information leaves no
residue to detect.

This generalizes past OCR. Any escalation ladder — cheap model then expensive model, cache then
origin, heuristic then solver — is only sound when the cheap tier's failure mode is
observable at the boundary. If the cheap tier can fail by silently discarding a relation
rather than producing a wrong value, "cheap first" isn't an optimization. It's an undetected
data loss path with a cost saving attached.

Both engines failed in exactly the same place

The adversarial string A0-1lO9-B8:

I inspected the enlarged image myself. The font renders digit 0 with a slash through it and
capital O as a plain oval. The two glyphs are visibly different. This isn't image
ambiguity that more pixels would resolve — it's both engines genuinely misreading a
distinguishable character, and resolution doesn't touch it.

So:

No OCR engine can be trusted on strings where homoglyphs change the meaning — codes, IDs,
hashes, addresses, license keys.

That's not a tool-selection problem. It's a property of the entire tool class, which means the
remedy isn't a better engine. It's human confirmation or a checksum. If you're about to build
an OCR path for identifiers, build the checksum first.

What I adopted

Vision as the first pass; escalate to the VLM only for documents where reading order carries
meaning — tables, forms.

With one correction to that rule, from the paragraph above: since column-shredding isn't
detectable downstream, "escalate on failure" doesn't work for tables. If the corpus is mostly
tables, go to the VLM first and eat the 300×. Cheap-first is only valid when failure is
visible.

What would change my mind

• If the target documents are mostly prose, Vision alone is sufficient and the VLM is a 300× waste.
• If Vision gains layout analysis and starts preserving table structure, this verdict is dead.
• The homoglyph failure is common to both engines, so improving one doesn't touch that part of the conclusion.

More notes at hexisteme.github.io/notes.