Benchmarking real-world document extraction

An evaluation of how vision-capable models turn proprietary, private invoices, receipts, and machine-generated documents into reliable, structured data.

494

Documents

Real-world invoices

162

Supplier types

Unique layouts

3

Input formats

Digital, scan, handwriting

80%

Challenging inputs

Scanned or handwritten

Compare model performance

Compare extraction accuracy across document formats with the cost of processing the complete benchmark.

Exact-match accuracy

Switch formats to see how model performance changes.

Higher is better

Full benchmark run cost

Calculated from measured usage across all 494 documents.

Lower is better

For API models, cost = input tokens at the input rate + output and reasoning tokens at the output rate.

InvoiceIQ took 5.7 minutes to complete the benchmark. At $1.50/hour for an NVIDIA L40S instance, the cost is $0.14.

Gemini 3.7 Flash’s $1.50 cost uses introductory pricing of $0.75 per million input tokens and $3.75 per million output and reasoning tokens. At the actual rates of $1.50 and $7.50, the same run costs $3.00.

Benchmark accuracy and cost by model and document format
ModelOverallDigitalScannedHandwrittenCost

Gemini 3.1 Pro

Gemini

81.38%99.00%80.57%70.75%$9.74

Gemini 3.7 Flash

Gemini

80.57%94.00%80.16%72.11%$1.50†

InvoiceIQ

Learn more

Madhi AI

80.16%99.00%82.59%63.27%$0.14*

Claude Sonnet 5

Anthropic

78.74%98.00%82.59%59.18%$3.85

Claude Opus 5

Anthropic

78.54%98.00%79.35%63.95%$10.66

Gemini 3.6 Flash

Gemini

78.54%95.00%80.97%63.27%$4.18

Kimi K3

Moonshot AI

71.26%98.00%70.04%55.10%$17.30

Gemini 3.5 Flash Lite

Gemini

67.81%84.00%70.85%51.70%$0.33

Kimi K2.6

Moonshot AI

67.61%81.00%69.23%55.78%$7.02

GPT-5.6 Sol

OpenAI

58.91%95.00%53.85%42.86%$10.44

GPT-5.6 Terra

OpenAI

52.43%91.00%48.58%32.65%$3.65

GPT-5.6 Luna

OpenAI

48.58%86.00%47.37%25.17%$0.53

Qwen3 VL 8B Instruct

Qwen

47.98%80.00%51.82%19.73%$0.17

Claude Haiku 4.5

Anthropic

31.17%65.00%32.79%5.44%$1.23

Every model was evaluated using the highest reasoning setting available for that model.

One image in, one strict record out

Each model receives the same invoice-extraction task. A response passes only when every scored field matches the reference record.

Scored fields

Invoice numberInvoice dateVATTotal incl. tax

Evaluation rules

  • Dates use a normalized day–month–year format.
  • Invoice numbers are compared without internal spaces.
  • Numeric values require exact equality.
  • Failed requests and invalid responses stay in the denominator.

Performance depends on the document mix

Overall accuracy tells only part of the story. The document format and the field being extracted both change where models succeed, and where human review is still needed.

Digital documents

Clean layouts reveal the extraction ceiling with less noise from image quality and handwriting.

Scanned documents

Compression, skew, shadows, and scan quality test whether a model remains dependable in common workflows.

Handwritten documents

Free-form writing and inconsistent structure expose the largest differences in model robustness.

The weakest field changes with the input

Across all fourteen models, VAT is the most reliable field. Invoice numbers trail the other fields, making them the largest source of field-level errors overall.

VAT
95.5%
Total including tax
90.9%
Invoice date
88.0%
Invoice number
82.3%

Accuracy shifts by format

Each input type creates a different extraction bottleneck.

93%+

Digital documents

Every scored field remains above 93% accuracy.

77.2%

Scanned documents

Invoice-number accuracy falls as blur, skew, and dense layouts make similar identifiers harder to separate.

70.0%

Handwritten documents

Dates become the weakest field, even as invoice-number accuracy recovers to 83.2%.

Where errors come from

Most misses fall into three recurring patterns.

Misidentified numbers

Models may select an account, order, or reference number, or change one character in a long identifier.

Date confusion

Most misses contain a wrong day, month, or year rather than simply using the wrong date format.

Competing amounts

VAT, subtotal, balance due, and grand total can be confused. Of all VAT errors, 43% occur when the correct value is zero.

Because the benchmark uses exact matching, one mistaken character or one incorrect field is enough to fail the complete record.

Madhi AI InvoiceIQ

InvoiceIQ is a supervised fine-tune of Qwen3-VL 8B Instruct, built to turn the base model's extraction capability into consistent exact-match performance.

Why this base model

Among the open base models evaluated during model selection, Qwen3-VL 8B Instruct delivered the strongest extraction accuracy while remaining compact enough to specialize.

What pass@64 revealed

When each document was attempted 64 times, the model often produced an exact answer at least once, even where single-pass accuracy was weaker. That gap signaled capability that could be made more reliable through fine-tuning.

Fine-tuning made the capability repeatable

InvoiceIQ underwent supervised fine-tuning on a specially curated corpus of 8,500 documents. The limited supplier overlap between training and evaluation also tests whether the model can generalize to unfamiliar layouts.

+32.18 pp
Overall accuracy uplift

From 47.98% for the base model to 80.16% for InvoiceIQ.

25%
Supplier overlap

Only a quarter of evaluation suppliers also appear in training.

82.59%
Scanned accuracy

Joint-highest scanned-document result in this benchmark.

Fine-tuning lifted InvoiceIQ to 80.16% overall accuracy, within 1.22 percentage points of Gemini 3.1 Pro, and to 82.59% on scanned documents, tied with Claude Sonnet 5 for the top result and ahead of Gemini 3.1 Pro’s 80.57%. The result suggests that the pass@64 headroom could be converted into dependable single-pass extraction without sacrificing generalization.

Interested in how a workload-specific model performs on your documents? Read the invoice extraction case study.

Need this level of accuracy on your own documents?

We build and fine-tune workload-specific extraction models around your documents, formats, and accuracy requirements.