have been imagined to be learn by a machine. Previous lodge invoices, financial institution statements, payslips, mortgage purposes, medical payments, customs varieties, courtroom filings, work orders.
Most corporations use free instruments alongside paid APIs to attempt to convert these paperwork, and in order for you structured output, APIs like Textract Structured run you as much as round $65 per 1k pages.
In the previous few years, although, numerous new choices have appeared: smaller open-source imaginative and prescient fashions specialised for OCR, common vision-language fashions, and doc parsing instruments like LlamaParse — altering what’s attainable and the fee thereof.
So it felt like a good time to do my very own experiment to check a few of these towards paperwork of various problem.
I scouted 93 docs that might act as a proxy for what corporations use OCR for — handwritten notes, tables, monetary legacy docs, scanned invoices, receipts, charts, previous newspapers, tax varieties — then ran all of them by way of 14 completely different engines.
The concept was to see how they dealt with two issues: textual content restoration and the flexibility to protect helpful desk construction.
The primary query I needed answered: do you really want to pay $65 for 1k structured pages, or are you able to slice that all the way down to a fraction? And does the specialised fashions win over the final ones?
When doing experiments like this you at all times discover fairly a couple of unusual issues, which I’ll cowl too. However to reply that foremost query I’ll take you thru what OCR is (skip if not new), the economics, the check, a few of the outcomes, and what else this confirmed me.
Notice: I didn’t check full area extraction, since that’s more durable to check cleanly throughout fourteen engines.
TL;DR
There is no such thing as a single greatest OCR engine. OCR is a routing downside.
For clear high-volume paperwork, Tesseract continues to be laborious to beat as a result of it’s free and quick. For combined manufacturing paperwork, Gemini Flash was one of the best all-rounder on this check. For tables, Mistral OCR regarded just like the cheaper structured possibility.
The smaller specialist fashions regarded good inside their consolation zone, however failed more durable at paperwork they hadn’t seen. So, for high-stakes or messy paperwork, it is sensible to escalate to a bigger mannequin.
The primary takeaway is financial: don’t pay for costly structured OCR when the doc doesn’t want it. Classify your docs, check engines by yourself knowledge, and route based mostly on value, accuracy, construction, and failure tolerance.
Benchmarks are helpful for discovery, however they won’t inform you what works in your paperwork.
Clarify the OCR house to me
OCR (Optical Character Recognition) is how a machine turns an image into machine readable textual content. Easy in precept, and for simpler docs largely solved, however more durable when issues change into extra human.
Simply to offer you a fast overview, older OCR discovered textual content on a web page, sliced it into characters, and matched each towards a library of recognized shapes. Tesseract has executed this because the Nineteen Eighties.
Fashionable OCR nevertheless (together with newer variations of Tesseract) normally makes use of a neural community that appears on the complete web page without delay and outputs the doc as textual content. So, in case your doc is a clear PDF or a high-quality scan in a normal font, OCR is usually a solved downside.
It stops being solved the second issues get messier: photographed receipts, handwritten notes, bizarre graphs and charts, dense monetary tables, or scanned tax varieties and mortgage purposes.
Corporations want this executed nicely for apparent causes, because it’s one thing each downstream system acts on. The higher OCR will get, the extra paperwork turns into one thing a system can motive over as a substitute of one thing a human has to learn by hand.
There’s additionally the truth that if we feed AI techniques badly parsed docs, the whole lot after will probably be laborious to belief.
I’m all about economics, so this house caught my eye as soon as I noticed how a lot cash is being poured into it. The Clever Doc Processing (IDP) market is projected to develop to someplace between $20 billion and $90 billion by the early 2030s, relying on which analyst you ask.
Most likely pushed by corporations paying $15–25 per bill in guide dealing with prices.
And since I keep near the tech world, I’ve watched a wave of specialised small OCR fashions ship over the previous yr (largely Chinese language), now being utilized by builders in all places.
Which raises the query I needed to check: can the small open-source fashions truly do the work the costly APIs cost for or ought to we truly look in the direction of the final imaginative and prescient fashions to deal with OCR too?
Skip the following part if you wish to perceive what this experiment confirmed. I’ve to undergo the check setup first.
The docs, the engines, and the metrics
This experiment comes down to 3 questions: what engines we used, what docs we examined with, and the way we determined who gained.
For the engines, I needed a lineup that lined all the alternatives I talked about, this meant: previous and new, open and closed, native and cloud, specialised and common.
Tesseract turned the classical alternative. It runs regionally and could be very quick. Then I added two document-parsing pipelines: Docling and Marker. Docling is slower however runs on CPU, Marker is open-weight however needing a GPU to run quick, which reveals up later within the value.
Then for the brand new wave of specialised open OCR fashions: GLM-OCR, PaddleOCR-VL, DeepSeek-OCR, and MinerU 2.5 (a borderline case, actually a pipeline with a VLM inside). I picked them off OpenDataLab’s OmniDocBench leaderboard, the place they ranked first, second, fourth, and fifth.
I hosted them on Modal and served the relevant ones with vLLM, batching to hurry issues up. I counted the scale-up time when measuring latency later.
I additionally added one closed purpose-built mannequin, Mistral OCR, which I’d heard good issues about.
On the open aspect, I used Qwen3-VL (8B, from Alibaba), additionally hosted on Modal with the remainder of the smaller fashions. I ought to flag that I gave it a plain transcription immediate quite than the optimized serving setup it was designed for, so I’ll not have given it a good shot.
On the closed aspect, for the final fashions, I picked Gemini Flash 3.1 Lite (at present first on the IDP Leaderboard, the western counterpart constructed on OmniDocBench v1.5) and Claude Sonnet 4.6, at sixth.
For the cloud doc companies: LlamaParse and AWS Textract, in each its textual content and structured varieties. Structured Textract can do excess of I requested of it. I solely examined its textual content accuracy throughout the board and its desk extraction towards eight of the opposite engines.
Let’s flip to the paperwork. I picked seventeen doc varieties that have been both simple, medium, or laborious. Ninety-three information in all.
Straightforward was the stuff OCR largely solved years in the past: clear invoices and receipts. Medium got here largely from the OmniAI OCR Benchmark dataset: financial institution statements, medical notes, photographed receipts, transport paperwork, tax varieties.
Exhausting was chosen when issues turned harder: charts, varieties, handwritten notes, weirdly scanned monetary tables, authorized papers, newspapers, and previous legacy studies.
Some docs have been actually fairly troublesome, such because the legacy scanned docs you see beneath, and this was simply because I used to be curious if some may truly do it nicely.
A few of these photographs got here with gold floor reality and a few didn’t, and the bottom reality I did have wasn’t at all times constant, some information labeled appropriately, some not, which is why we should always briefly cowl the metrics too.
Since each engine emits completely different markup, the standard scoring didn’t fairly match. One would possibly decide Precision and Recall for a case like this.
Precision appears to be like at how lots of the OCR output’s phrases truly match within the GT whereas Recall measures what number of occasions every GT phrase was captured.
Precision would punish engines that emit markdown construction the GT doesn’t comprise, moreover the GT generally skipped labels fully which might punish the engine unfairly. Recall would measure the phrases however punish the frequency.
So, I added on a 3rd metric known as Protection. I simply needed to measure how a lot of the bottom reality reveals up someplace within the engine’s output. It isn’t excellent, but it surely tells me whether or not an engine caught most of what mattered, with out penalizing it for gaps that have been the bottom reality’s fault quite than the engine’s.
For the paperwork with no gold floor reality in any respect, I fell again on an LLM decide, with Gemini 3 Professional as the bottom mannequin and anybody who’s used one is aware of that is fickle enterprise.
What this experiment confirmed
We mapped each doc towards the Protection metric to construct a scatter chart, and tracked latency on a separate chart. The factor a generalized chart can’t inform you although is that the engines failed in numerous methods.
The bubble graph confirmed that the majority engines fall someplace within the center prime, with two outliers on each side of it.
Gemini Flash and Textract Textual content did very nicely throughout the board with some edge circumstances. The specialised fashions all fell beneath the final fashions and specialised APIs. Sonnet carried out the very best but additionally with a steeper price ticket.
This may occasionally not have been a shock because the check set was extremely uncommon. Among the specialised fashions could not have seen a lot of them. Moreover, this check was on English paperwork and most of those smaller fashions have Chinese language origin.
After we additionally mapped latency, a few of the fashions turned out to be very gradual, however once more most wound up someplace within the center.
The outliers right here have been: Tesseract, Claude Sonnet 4.6, and Docling. Tesseract was extremely quick in comparison with all different engines. It ought to be your go-to for simpler paperwork.
These graphs generalise throughout all of the paperwork, however I did separate the outcomes based mostly on the kind and problem stage.
To begin with the straightforward docs. On invoices, each engine did nicely, Tesseract particularly. Receipts knocked everybody down a bit of.
The one outlier was Docling, which struggled throughout numerous the classes, even the straightforward ones.
Once I regarded into the Docling failures I discovered issues like Ifjointreturn as a substitute of “joint return,” and worse, strings like Metropolis,wrostffielfouaveaoreignadresalcomletacesb. DeepSeek additionally missed key particulars right here like bill quantity and date, which is why its quantity sits low.
The identical sample holds within the medium class, although that’s the place PaddleOCR began degrading on particular varieties: financial institution statements, transport, tax varieties. Tax varieties have been laborious for everybody, however PaddleOCR and Docling wound up on the backside.
Textract was one of the best engine on numerous the medium varieties, together with Claude Sonnet 4.6 and Mistral OCR.
On the more durable varieties, Gemini Flash began rising, beating Textract on varieties and handwritten notes, matching it elsewhere. It did remarkably nicely in all places. Tesseract and Docling failed laborious on handwritten, and varieties have been powerful for them too.
Nearly all of the specialised fashions didn’t pull by way of on these more durable docs besides on monetary tables, the place they held about even.
For the docs with no floor reality (newspapers, authorized, studies, some scanned legacy paperwork) we used an LLM decide. These are genuinely laborious, so it’s no shock virtually everybody failed on the studies and newspapers.
Besides Gemini Flash that did fairly nicely in all places. Mistral OCR additionally did nicely for newspapers. Gemini Flash gained in all places with the decide, although we used Gemini Professional because the decide so take that with a grain of salt (however I did double test myself).
Earlier than rounding off: I additionally ran 8 engines towards Textract Structured to see how they did on monetary tables, extracting an HTML desk. I used Textract Structured’s output as the bottom reality for TEDS (Tree Edit Distance Similarity) and scored Claude Sonnet 4.6, LlamaParse, Mistral OCR, Gemini Flash, Marker, MinerU, DeepSeek-OCR, and Docling towards it.
Mistral OCR, and LlamaParse, and Sonnet did very nicely whereas being less expensive. I additionally ran it by way of an LLM decide, and the winners have been the identical three (even earlier than Textract Structured) although I’d wish to construct that check higher earlier than I totally belief it.
Now, let’s speak about what it prices to scale this up, and what would make sense the place.
When does what make sense
Let’s run by way of what it prices to scale up with these engines, after which based mostly on these docs what you’d selected the place.
First, the price of utilizing these engines fluctuate wildly, as you noticed earlier than. Typically it helps to see the fee not only for one doc, however hundreds as much as 1,000,000.
We’re self-hosting on Modal, so these prices come from precise utilization there. You possibly can run regionally, however my pc wouldn’t permit it and I didn’t wish to strive it.
In the event you have been to only use one engine that handles each simple and laborious paperwork, I might suppose you wound up with a much bigger invoice than obligatory. Utilizing Textract Structured for any paperwork that aren’t wanted would hand you a invoice of $6.5k per 100k docs.
I do surprise what number of corporations go the straightforward approach right here and decide the costly choices for simple in addition to laborious docs and depart some huge cash on the desk.
The important thing thought to take with you right here is that there’s no single greatest engine for each use case, it will depend on doc sort, privateness, desk construction, failure tolerance, value, and so forth.
For the docs we have now right here, Gemini Flash 3.1-Lite is a transparent winner. This one was appropriate from wanting on the leaderboards. Mistral OCR did nicely on structured tables whereas staying low-cost. Claude Sonnet 4.6 did very nicely too, but it surely’s very gradual and costly comparatively.
Docling is so very gradual on my laptop computer. I’m certain there are methods to hurry it up, but it surely additionally failed in ways in which make it inherently unstable (nonetheless a small check although).
The specialised OCR fashions have been a little bit of a headache, particularly on English docs; I noticed output errors in Chinese language that I’ll cowl in a bit, so I’m wondering if that’s a part of it.
Textract is a secure alternative, however structured buys you virtually no extra textual content accuracy so should you’re paying that steep markup for structured output, be sure to truly use it. I’m guessing it’s a fairly good enterprise mannequin for them.
So, typically for this very small check: for clear, high-volume print, simply use Tesseract. For common heterogeneous manufacturing, go Gemini Flash. For a cost-floor with desk construction, check Mistral OCR. For top-stakes docs, path to Sonnet or a bigger mannequin.
Since everybody did nicely in numerous methods you’ll need to contact me for specifics but when you might want to go non-public it might be price it to take a look at fine-tuning a mannequin in your docs. Or, use a small specialised mannequin and escalate on failures.
Let me simply shortly speak about some issues that stood out after doing this experiment.
Different stuff I ought to point out
A handful of issues surfaced from this which are price pulling out on their very own.
First, if you wish to perceive how a mannequin or engine will do in your docs, the one approach is to check on these docs, you possibly can’t depend on benchmarks to inform you. This was the primary perception this confirmed. OCR usefulness relies upon by yourself doc combine, layouts, languages, scans, tables, handwriting, and failure tolerance.
Don’t pay for construction should you don’t want it. I’m wondering what number of are utilizing sure APIs or fashions for a motive they’ll’t justify. Map the fee to grasp what you’re shedding by not utilizing the right engine for the paperwork.
The specialist fashions, as talked about earlier than, have sharp boundaries. That is apparent, they are often glorious inside their coaching distribution however fail exterior of it. That is the place the final fashions will win.
If you wish to fine-tune it might assist, however provided that the stream is secure as it is going to additionally fail whether it is continually launched new doc lessons.
Lastly, the failure modes instructed us greater than the averages.
PaddleOCR had repetition loops, column-merging, fallback into Chinese language textbook template textual content like 书名:___ repeated a whole bunch of occasions. Whereas Docling has character errors, word-merging, and column misalignment all stacking collectively.
DeepSeek OCR has chart blindness and empty outputs on some docs. Tesseract did nice on clear docs (as talked about) however failed on photograph/handwriting altogether outputting rubbish.
Caveats to take into account
Earlier than we spherical up, let me cowl how this check is finally imperfect by naming the problems within the GT, the metrics used, and the pattern measurement.
I lined this in one of many part above, however the floor reality differs between paperwork relying on the dataset the place they have been discovered. Generally tokenization artifacts could make appropriate OCR look worse than it’s.
Most engines have completely different codecs, some return plain textual content, some markdown, some HTML/wealthy markdown and it’s laborious to generalize throughout all.
We’re utilizing Protection, after which additionally another metrics, however these aren’t excellent. Protection gained’t cost the engine if it outputs an excessive amount of textual content or the construction of it’s off. Although I did discover that for the engines that failed, they did so initially or mid-way by way of quite than on the finish.
This implies it’s helpful for rating however not an ideal solution to rating.
LLM judges should not impartial reality: I’ve lined this up to now, however they’re biased and really immediate delicate.
Then I simply have to say that this check is attention-grabbing however not that huge, the pattern measurement is approach too small to make use of this as a factual examine. However, I don’t totally belief these metrics nor the decide so it was the one approach for me to have the ability to double test the outcomes alone with out this turning right into a yr lengthy venture.
So, this check is helpful for path and getting a way of what works, however for getting a way of your use case, you might want to run it by way of along with your particular docs.
Lastly, latency and reproducibility is unstable. Serverless chilly begins make timing noisy, and API fashions can silently change over time, so precise copy is tough.
Like at all times with these articles, it takes fairly a bit to do an experiment like this however I don’t simply do it for content material, I do it as a result of I’m genuinely curious.
What it appears to be like like although is that OCR appears to be a routing downside, and maybe an analysis downside. Classify your docs and run them by way of a number of engines, then attempt to construct an honest router and validator in your pipeline to escalate failures after which log the prices.
If you might want to get the complete outcomes from this experiment otherwise you need me to run it by way of your docs, get in touch.
You possibly can observe my writing on Medium, my website or join with me through LinkedIn.
❤
All datasets used on this benchmark are publicly accessible and sourced from HuggingFace. Licenses embody MIT, CC-BY-4.0, and fair-use frameworks (UCSF Business Paperwork Library) overlaying analysis, scholarship, and schooling. No supply paperwork are reproduced — datasets have been used solely as analysis inputs to measure OCR engine efficiency.
