The Short Answer
This post compares 3 extraction routes and 1 routing design that combines them. They serve as practical reference architectures for AI document processing, and production systems often blend them.
- Document Analysis Service reads the page with OCR, layout analysis and prebuilt or custom trained document models, then maps the output to fields with rules where needed
- OCR Plus Language Model passes structured OCR output to a large language model that finds fields by meaning
- Direct Multimodal Extraction sends the page image or PDF to a model that reads and interprets it in 1 inference call
- Routed Hybrid classifies each document and sends it to whichever route handles that class at the lowest total cost for an acceptable error rate
5 workload inputs shape the choice:
- How much the layout varies across documents of the same type
- How many pages arrive each month and how fast each result must return
- How many languages and scripts appear, and whether they mix on 1 page
- Whether documents arrive as digital PDFs with a text layer or as scans and photos
- What a wrong field costs once it reaches a downstream system
Workloads with several document classes and uneven source quality are strong candidates for Routed Hybrid. The decision matrix in this post maps each input to the route that tends to fit it, and a pilot on your own documents confirms the choice.
The Decision This Architecture Choice Controls
A document processing system turns pages into structured fields that another system can act on. Ingestion, packet splitting, storage, validation and integration all matter, and each one has its own design choices. This post covers 1 choice inside that system, the extraction route that reads the page and returns fields.
That choice shapes 3 numbers you will live with for years. It shapes your cost per 1,000 pages, your share of documents that need a human reviewer, and your exposure when a model provider changes or retires a model.
Start with 1 check before any route. Many invoices, statements and reports arrive as digitally generated PDFs that already carry a text layer. A PDF parser or a layout analysis service reads that text with exact characters and positions, and no image recognition step needs to run. Recognition only becomes necessary for scans, photos, faxes and pages with handwriting.
The post assumes you have already decided to build a custom system and know your document classes. If you have not yet collected real documents, gather a pilot sample of 200 to 500 per class that covers your layouts, vendors, languages and scan quality. Keep a separate holdout set that nobody uses while writing prompts or rules, and judge each route on that holdout.
Fig 1 - 3 Extraction Routes and 1 Output Contract
Route 1 Document Analysis Service
A document analysis service reads the page and returns structured output. Services such as Amazon Textract and Azure AI Document Intelligence return words and lines with bounding polygons, tables with cells, key and value pairs, selection marks such as checkboxes, signature locations and a printed or handwritten flag per word. They also offer prebuilt models for common documents such as invoices, receipts and identity documents, and custom models that you train on your own labelled samples. An open source engine such as Tesseract covers the text recognition part of this route at no licence cost.
Your system then maps that output to your schema. For a fixed layout, rules can map regions or anchor words to fields. For a common document type, a prebuilt model returns the fields directly. For a recurring proprietary form, a custom trained model learns the field positions from labelled examples.
Where it fits - High volume document classes with a stable set of layouts, or a document type that a prebuilt model already covers well. Per page cost is predictable, latency is low and every extracted value carries page coordinates that support review and audit.
Where it breaks - Rule based mapping needs a new rule for every new layout, and a supplier who moves a field causes a wrong or empty value until someone updates the rule. Custom models reduce that maintenance and need labelled training data plus retraining when layouts shift. None of these options reason about context, so a value that needs interpretation, such as a delivery condition written as a sentence, still needs another step.
Economic profile - Low and predictable cost per page, low latency and a maintenance bill that grows with layout count for rule based mapping and with retraining cycles for custom models.
Route 2 OCR Plus Language Model
OCR Plus Language Model runs a document analysis service first and passes its output to a large language model with a prompt and a target schema. The model reads that output, finds the fields and returns structured values. The model locates an invoice total by meaning wherever it sits, so this route removes most template maintenance.
The quality of the handoff decides how well this route performs. When the integration passes tables, cell positions, reading order and coordinates to the model, the model keeps most of the layout evidence. When the integration reduces that output to a plain text stream, table rows merge, checkbox states vanish and numbers lose their columns.
Fig 2 - Flattened and Structured OCR Handoff
Where it fits - Variable layouts with mostly printed text, such as supplier invoices from hundreds of vendors, purchase orders and shipping confirmations. Teams that already run a document analysis service can add the model step without replacing it.
Where it breaks - A misread character or a dropped diacritic in the OCR output reaches the model as fact. The model can sometimes correct an obvious misread from context, and it can also return a confident wrong value built on the error. The route also adds a second call, a second cost line and a second point of failure.
Economic profile - OCR cost plus model token cost on every page, higher latency than Route 1 and far less template maintenance. Error cost tracks OCR quality and the fidelity of the handoff.
Direct Multimodal Extraction sends the page image or PDF to a model that accepts images as input. The model reads the characters, sees the layout and returns fields against your schema in 1 inference call. A separate upload request may come first when the provider uses a files API. For PDFs, some providers combine the rendered page image with the native text layer where one exists, as the Gemini document understanding guide describes.
No separate OCR handoff exists, so no information disappears between 2 systems. The model itself can still misread small text, faint scans and handwriting.
Where it fits - Documents where layout carries meaning or text quality varies, such as scanned forms with handwriting, tables that span columns, checkboxes, stamps and pages that mix 2 scripts. It also suits teams that want 1 vendor contract and 1 cost line per page.
Where it breaks - Models can return bounding boxes when you ask for them, and those boxes come from the model, so you validate them before you treat them as page coordinates. Output can vary between runs on the same page. Versioned model IDs, explicit instructions, structured output, deterministic validation and regression testing reduce that variation, and none of them guarantee identical extraction. Lowering temperature is no fix either, because current Gemini 3 guidance recommends keeping temperature at its default of 1.0 and warns that lower values can degrade output. Your extraction quality also depends on a model version that the provider will eventually retire.
Economic profile - Token cost per page, latency in seconds per page and the lowest template maintenance of the 3 routes. Model version management becomes a standing operating cost.
Routed Hybrid Combines the 3 Routes
Routed Hybrid is a routing design, and it extracts nothing by itself. A classifier identifies the document class, and a routing table sends each class to the route that handles it at the lowest total cost for an acceptable error rate. A known utility bill goes to a document analysis service. An unfamiliar supplier invoice goes to OCR Plus Language Model. A handwritten intake form goes to Direct Multimodal Extraction.
The router also acts on validation results. When an extraction fails a hard check, such as a date that does not parse or line items that do not sum to the stated total, the router can send the page straight to review. When the result passes hard checks and sits in a grey zone on the decision score for its class, the router escalates the page to a stronger route before any reviewer sees it.
Where it fits - Workloads with several document classes, uneven quality across sources or a wide gap between the cheapest and most capable route. Accounts payable, claims and logistics intake workloads often show these traits.
Where it breaks - Routed Hybrid needs more engineering up front. You build and maintain a classifier, a routing table, thresholds per class and a regression set per route. A badly tuned threshold sends too much volume to the expensive route or lets too many errors through the cheap one.
Economic profile - Higher build cost and a standing monitoring cost. At sufficient volume and variety, the blended cost per page can fall below any single route, and your pilot data shows whether it does for your workload.
Fig 3 - Routed Hybrid Extraction Flow
The Decision Matrix
Score each document class against the rows and note which column tends to fit. The cells describe typical behaviour, and your pilot results on the holdout set override them. If different classes favour different columns, you have a Routed Hybrid workload.
Table 1 - AI Document Processing Architecture Decision Matrix
1 Output Contract Across 3 Routes
A single output contract keeps the route decision honest. Every route returns the same fields in the same types, so the target system never needs to know which route produced a record. The conceptual contract below covers a supplier delivery note, a document that mixes printed text, a line item table and a handwritten mark in the receipt box. Your engineering team turns it into a formal JSON Schema with required fields and nullability rules.
output-contract.json
{
"document_id": "string",
"correlation_id": "string",
"document_class": "delivery_note",
"fields": {
"supplier_name": {
"raw_value": "string | null",
"normalized_value": "string | null",
"page": "integer",
"source_region": "polygon | text span | null"
},
"delivery_note_number": "{ same structure }",
"delivery_date": "{ same structure, normalized to ISO 8601 }",
"purchase_order_ref": "{ same structure }",
"signature_mark_detected": "{ same structure, boolean }",
"damage_noted": "{ same structure, boolean }"
},
"line_items": [
{ "sku": "{ same structure }", "description": "{ same structure }", "qty_delivered": "{ same structure, number }", "unit": "{ same structure }" }
],
"validation_failures": ["string"],
"decision_score": "number | null",
"review_status": "auto_posted | escalated | in_review | corrected",
"provenance": {
"route": "document_analysis | ocr_llm | multimodal",
"engine_or_model_version": "string",
"prompt_version": "string | null",
"schema_version": "string"
}
}
The signature_mark_detected field records that a mark appears in the signature box. It confirms nothing about who signed, and any process that needs signer authentication needs a separate control.
Table 2 - How Each Route Fills the Delivery Note Contract
Table 2 describes typical behaviour, and your holdout results replace it for your documents. The provenance block matters more than it looks. When a reviewer corrects a field, provenance tells the engineering team which route, model version and prompt version produced the error, and the corrected document joins the regression set for that route.
Validation State and Decision Scores Decide the Route
A router needs evidence before it can decide what happens to an extraction. That evidence comes from signals of different kinds, and each one answers a different question:
- Hard validation checks whether a value obeys the contract, such as a date that parses or a quantity that is a number. A failure is a fact, and it can send the page to review without any score
- Consistency checks compare fields with each other, such as line item quantities against a stated total
- OCR word confidence measures how sure the engine is about the characters it read, which differs from how sure anyone should be about the field built from them
- Agreement between 2 routes on the same page measures disagreement, and 2 routes can agree on the same wrong value
- A confidence value that a language model reports about its own answer carries no calibration by default
Combine the soft signals into a decision score per field class, and treat it as a ranking signal until you calibrate it. To calibrate, run the router over your labelled holdout set and record the false accept rate at each candidate threshold, meaning the share of auto posted fields that were wrong. Pick each threshold from the false accept rate that the cost of a wrong field in that class can tolerate.
Field risk matters more than document averages. A misread supplier name on a delivery note costs little. A wrong quantity can trigger a payment or an inventory adjustment. Set stricter thresholds, or mandatory review, for the fields that move money or stock. Revisit every threshold whenever a model, prompt or OCR version changes.
Fig 3 shows where these checks sit in the Routed Hybrid flow.
How Each Route Fails
Every route fails in a recognisable way. Knowing the failure in advance tells you which check to build before go live.
Table 3 - Failure Modes and Checks by Route
What the Route Choice Leaves Open
The extraction route is 1 decision in a production system. Several others sit beside it and deserve their own design work:
- Packet splitting and page grouping before classification, for scans that hold several documents in 1 file
- Retries, idempotency, duplicate detection, rate limits and a dead letter queue for pages that fail repeatedly
- Retention, encryption, processing region and each provider's data use terms for documents that carry personal data
- A review screen that shows the source region beside each uncertain field, and a path that feeds corrections back into the regression set
- Evaluation metrics such as field accuracy, table cell accuracy, false accept rate, straight through rate, review rate, 95th percentile latency and total cost per document
What Each Route Costs per 1,000 Pages
On published list prices, a direct multimodal call on a low cost model tier costs about the same as OCR text detection alone, and far less than OCR with form and table analysis. Processing cost therefore rarely decides the route on its own. Review, engineering and maintenance cost usually weigh more.
The estimates below assume a single page document, a 500 token extraction prompt and 400 output tokens of structured JSON. They use about 560 tokens per page image, the figure the Gemini API media resolution documentation gives for its recommended PDF setting. OCR prices come from the Amazon Textract pricing page for the first 1 million pages in US West (Oregon). Model prices come from the Gemini Developer API pricing page at standard paid tier rates.
Table 4 - Estimated Processing Cost per 1,000 Pages by Route
The Gemini Batch API charges 50% of the standard price and targets a turnaround of up to 24 hours. It suits overnight backlogs and reprocessing runs, and it cannot serve a workflow that needs a result while a user waits.
The 400 output tokens must include any thinking tokens the model spends, because the provider bills them as output. If the model reasons at length before answering, the real figure rises above these estimates. Set an appropriate thinking_level for extraction work, and measure real token counts on your own sample. The mid tier Flash prices above are promotional and both input and output double on 1 January 2027, which takes that row to about 4.59 per 1,000 pages at standard rates and puts it above OCR plus a language model. Prices were last checked against the published rate cards on 2 November 2026, so recheck them before you commit a budget.
4 other cost lines usually decide total cost:
- Human review minutes per document multiplied by your review rate, which falls as routing and thresholds improve
- Engineering build cost, which rises from a single route to Routed Hybrid
- Rule and prompt maintenance, which grows with layout count on rule based mapping and with version changes on the model routes
- Integration with the target ERP, CRM or workflow system, which stays roughly constant across routes
A workload of 50,000 pages a month at 2.30 per 1,000 pages spends about 115 USD a month on mid tier model processing. At US labour rates, 1 reviewer hour saved per working day outweighs that line.
A Production Example
Clixlogix moved a legal workflow platform off OCR Plus Language Model when the OCR step began corrupting Nordic party names. Dropped diacritics on Swedish, Norwegian and Danish names caused mismatched records, misrouted cases and false risk flags further down the workflow. The language model could not repair characters the OCR step had already lost. The team replaced the 2 step design with Direct Multimodal Extraction, kept the integration behind a configuration switch for the model identifier and added a regression set before each model change. The full account of that migration from a 2 step OCR pipeline to a multimodal model covers the data residency constraints, the model upgrade path and the results.
Find the Route Each of Your Document Classes Needs
Your document types, monthly volume, languages and destination system decide which architecture fits each class. Share those 4 details with Clixlogix and receive an outline of the processing architecture, the extraction schema and the exceptions that still need a human reviewer.
Plan my document processing architecture
FAQs
