In April 2026, a Wall Street firm’s court filing was flagged for AI generated citations that had to be withdrawn. It was not the first incident of its kind, and going by what has happened across the past 24 months, it will not be the last. The usual fixes get pulled out. Add a human reviewer at the end. Try a better model. Tighten the prompt. And the trouble is, each of those treats the failure as an accident of implementation, when actually it is a property of the design.
The design is the problem. Firms in these incidents deployed AI as a substitute for legal research, then treated the output as the deliverable. The model was asked to read the corpus, identify the relevant authority, judge which version was current, and produce a defensible citation, all in a single pass. When the model had to do all four jobs at once without structured input, it filled the gaps with what read as plausible. What reached the court was fluent, formatted, and wrong.
A different approach was already in production before those headlines arrived. Clixlogix worked with a Nordic legal workflow CRM client whose risk model treated open ended AI reasoning as the exact scenario to design around. Their platform, built with Clixlogix, sequenced AI as one step in a multi step system. Documents were prepared before retrieval touched them. Retrieval surfaced tight, verifiable passages before the LLM was called. The LLM’s job stayed narrow. Synthesis over context that had already been resolved, not open ended reasoning across the corpus. The recent industry failures validated a design decision the Client had made a year earlier.
This piece walks through what “prepared” actually meant in that engagement, using publicly checkable documents any legal team can open for themselves. The earlier piece covered the extraction step of the same engagement, showing how a multimodal model preserved document structure at ingestion. This piece covers what happens after extraction is solved. Getting text out of a document correctly is one problem. Arranging that text so an agentic system can reason over it defensibly is a different one, and it is where the recent public failures actually broke.
For legal CXOs evaluating agentic legal AI right now, the practical question is not which model to license. It is whether the documents feeding the model have been prepared for the way it reads. That question determines whether the system produces work that holds up in court or work that ends up in the Guardian.
Where Legal AI Actually Sits in Firms Right Now
Before getting to what document readiness requires, it is worth being honest about where legal AI sits inside firms today. There are three situations running at the same time, and any legal CXO can probably map their own week to at least two of them.
Some lawyers in the firm are using AI carefully at the edges. Proofreading. Softening tone. Building a Westlaw search string. Summarizing a long inbox thread. A first pass at a 50 state survey the associate will verify against the actual statutes. This works because the lawyer already knows the answer or knows what a wrong answer looks like. The tool accelerates mechanical work. Judgment stays with the human.
Other lawyers, often under time pressure, extend that same casual use into tasks that require infrastructure. A chatbot gets asked to find cases supporting a specific argument. A vendor’s “AI for law firms” product returns citations that get trusted without verification. The moment the model is asked to do more than accelerate work the lawyer already understood, the failure shows up. Fluent output. Fabricated authority. A partner reading about it in the news the next morning.
Then there is what walks in through the door. Clients send discovery responses drafted by ChatGPT. In house counsel arrive at meetings with confident, wrong opinions from a two paragraph prompt. Opposing parties file motions that read like a language model produced them because a language model did. Ethics complaints show up citing rules the AI made up. Every lawyer with a client facing practice can name a version of this from the past 90 days.
These three situations share a single design failure. Model output is being treated as a finished product. Nothing around it carries verification, provenance, or judgment. Careful edge use works because the lawyer supplies that missing workflow manually, one task at a time. It scales as far as the lawyer’s attention scales, which is not far. The public failures happen when the manual workflow gets skipped. The client slop problem is the same failure walking in from the outside.
The question the CXO has to answer is which workflow gets built around the model so that verification, provenance, and judgment stop depending on individual instinct. That workflow starts with the documents. The ones the firm actually holds, uses, and reasons over. Getting those ready for the way an agentic system reads is the design decision that separates the firms whose AI use scales safely from the firms making the news.
What Happens When Documents Are Not Prepared for the Model
The failure shows up any time retrieval fetches a clause without the definition that governs it. The model receives a legally incomplete picture and answers from what it has. The answer is fluent. The citation is real. The gap in reasoning is invisible to the model, because the piece that would have closed the gap was never in the context.
Open EUR-Lex to the GDPR, Article 6.
Fig 1 – GDPR Article 6, retrieved on its own. The term “processing” appears 25 times. Its definition is not on this page.
Article 6 sets out the six lawful bases for processing personal data. Consent. Contract. Legal obligation. Vital interests. Public task. Legitimate interests. Every one of them uses the word “processing” as if the reader already knows what it means. Any lawyer reading Article 6 does. The retrieval system reading Article 6 has only Article 6.
The definition lives one article back.
Fig 2 – GDPR Article 4, the Definitions article. Item (2) defines “processing” and covers 12 distinct operations, any of which changes what qualifies as lawful under Article 6.
A retrieval system that pulls Article 6 in response to a question like “does our processing activity qualify under legitimate interests” hands the model an article whose central term has no boundary. The model will produce an answer using whatever meaning of processing it inferred, and the answer will read as authoritative. Verification is one Ctrl+F away. The definition is not there.
The same failure shows up in commercial contracts. Consider a real NDA filed with the SEC as an exhibit to a 2010 transaction between 3M and Cogent.
Fig 3 – Section 1.1 of the 3M-Cogent NDA. The term “Proprietary Information” is defined here, including three exclusion categories that limit what actually counts as protected. (tap to open at full resolution)
Section 1.1 defines “Proprietary Information” in a full paragraph and adds three exclusion categories. Information already public through no fault of the recipient. Information legitimately obtained from another source. Information independently developed. These exclusions are the entire boundary between protected material and unprotected material.
The obligation clauses that reference the term sit on the next page.
Fig 4 – Section 2 of the same NDA. Four obligation clauses reference “Proprietary Information” multiple times. The definition and its exclusion categories are not on this page. (tap to open at full resolution)
Section 2.1 says the Receiving Party shall not disclose, in any manner whatsoever, in whole or in part, any Proprietary Information. Retrieval pulls it because it matches a query like “what can our team share with the contractor about this project.” The advice returns broad and cautious. Three categories of information that the contract itself excludes from the definition are treated as covered, because the model was reasoning from an obligation whose scope was cut off from it.
Both examples share the property that makes the failure hard to catch. The retrieved passage is real. The citation is correct. The article number is right. Everything visible about the answer looks defensible. What is missing is invisible until someone opens the document and checks. That check is what has been skipped in the recent sanctioned filings.
Four Things Documents Have to Carry
What RAG means
RAG stands for retrieval augmented generation. It is the standard architecture behind AI tools that search your documents before the model answers. Every legal AI tool a firm evaluates today is either a RAG system, a chatbot with no retrieval at all, or a chatbot with retrieval bolted on. When this piece uses the word RAG, that is what it means.
Definitions are one thing documents carry that retrieval has to respect. There are three more that matter almost as much. Which document outranks which, which version is current, and what other sections a given clause depends on. Any agentic legal AI serious enough to be used in a firm has to know all four before it produces an answer.
Fig 5 – The same clause in two states. Above, the document is raw. Below, the document carries the four properties agentic legal AI needs to reason safely: definition links, authority level, version awareness, and cross reference edges.
Hierarchy
Not every document a firm ingests carries the same legal weight. A legal document automation program running inside a firm usually pulls in primary sources (statutes, regulations), secondary sources (case law, regulatory guidance), and internal material (memos, past advice, drafts). Retrieval that treats all of them as a flat set of chunks returns whichever chunk matches the query best on similarity. Similarity does not correlate with authority.
Take GDPR again, this time on employee data processing. GDPR Article 6 provides the lawful bases at the EU level. Germany’s BDSG Section 26 adds more specific rules for employee data processing, under a permission GDPR itself grants in Article 6(2) and (3). Both documents talk about processing employee data. They do not have equal legal weight for a German controller. A retrieval system that returns whichever passage matches the query most closely, without knowing which document outranks which, is producing an answer that a human lawyer would never produce, because the human lawyer knows the hierarchy without thinking about it. A legal RAG pipeline does not, unless the hierarchy was tagged on the document at ingestion.
Version Control
Contracts and regulations both get amended. The amendment does not always replace the whole document. It often changes specific clauses inside a document that otherwise looks the same. If the amended and original versions both sit in the retrieval corpus, and retrieval has no way to distinguish them, the answer depends on which clause the search returned first. For any firm running AI contract review at volume, this is where the failure will show up first.
Look at the 3M-Cogent NDA from the previous section. Section 6 in the original 2008 agreement set the confidentiality survival period at two years. In 2010, an amendment changed it to four.
Fig 6 – Amendment No.1 to the 3M-Cogent NDA (2010). A retrieval system without version awareness returns whichever clause it finds first. Only one answer is current. (tap to open at full resolution)
That is a two year error on the confidentiality clause and a two year error on the no-solicitation clause, on the same contract, produced by the same query, purely because retrieval did not know that the second document supersedes parts of the first. In a live deal that error moves real money. And the model has no way to catch it, because both clauses look real when read in isolation.
Cross References
Legal drafting is compressed by design. A clause references another section, another article, or another statute, and the reader is expected to follow. Article 6 of GDPR references Article 9 (special categories of personal data, like health data and political opinions) and Article 49 (derogations for international data transfers). A question about processing sensitive personal data across borders requires all three articles together. AI legal document review pulling one at a time returns an answer that is correct for whichever article got fetched, and incomplete for the actual question.
An NDA does the same thing. Section 2.5 of the 3M-Cogent NDA references “the attorney-client privilege, work product doctrine or any other applicable privilege.” Section 6, the termination clause, says the obligations “shall terminate on the second anniversary of the Effective Date; provided that Sections 2.5, 2.6, and 2.7 and Articles 4, 5, 6, 7, 8 and 9 shall survive.” That is a compact sentence that depends on the reader (or the model) being able to reach into eight other sections to know what actually survives termination. Retrieval that returns Section 6 in isolation, without those eight referenced sections, hands the model a survival clause with no survivors named.
Fig 7 – Section 6 of the 3M-Cogent NDA references eight other sections that survive termination. Without cross reference edges captured at ingestion, the termination clause reads as a rule with no survivors named.
What to Ask When Someone Claims to Have Prepared Them for You
The four properties are the goal. In practice, most legal CXOs will not build this themselves. They will evaluate what their firm already has, or what a vendor is pitching, or what an internal team is proposing to build. The framework matters as an evaluation tool, in the room where the decision gets made.
Five questions cover it. Each maps to one of the four properties, plus a fifth on what happens when the world changes. Ask them in a vendor meeting, in a product demo, or in a conversation with the head of IT about the document management system the firm already runs. A strong answer names a specific mechanism inside the system. A vague answer about how smart the model is means the mechanism does not exist.
When your system returns a clause that uses a defined term, does it also return the definition, automatically, from the same document?
Strong answer:definitions are indexed and linked at ingestion, so any retrieval that returns a clause containing a defined term also returns the definition.
Weak answer:the model is capable of understanding context. If the vendor cannot describe how the definition gets into the model’s view, the definition will not be there when it matters.
If we upload a regulation and an internal memo on the same topic, how does your system decide which one to surface first?
Strong answer:documents are tagged with an authority level at ingestion, and retrieval scores by authority as well as similarity.
Weak answer:the system finds the most relevant match. Relevance is not authority. A memo that uses the same words as a regulation will win a similarity contest unless something else is telling the system that the regulation outranks it.
When we amend a contract, how does your system know that the amendment supersedes specific clauses in the original?
Strong answer:amendments are linked to the original document with effective dates and clause level supersession metadata, and retrieval filters to the current version by default.
Weak answer:both versions get uploaded, and the model figures out which is current. It does not. It returns whichever clause the search finds first.
When a clause references another section, does your system pull that referenced section into the same answer, or does it leave the model to guess?
Strong answer:cross references are captured at ingestion, usually by pattern matching on standard legal citation formats (“pursuant to Section X,” “as defined in Article Y”), and referenced sections travel with the clause that references them.
Weak answer:the model is trained to follow references. It is not. It answers from whatever text is in front of it.
When a regulation changes or a contract gets amended, what happens?
Strong answer:there is a defined update process, with named roles for who does the update work and a stated turnaround from change to system reflection.
Weak answer:the system stays current automatically. It does not, unless someone is doing the work.
The economics follow from the answers. Preparing a document is a one time cost per document, largely automated by pattern matching for the parts of legal drafting that are patterned by convention, with a light layer of human review for the parts that are not. Maintenance is what happens when the corpus changes. Both costs are meaningful. Both sit far below the recurring per query cost of asking a larger and larger model to reconstruct structure at runtime for every question. For any firm running agentic legal AI at volume, that arithmetic settles quickly.
What This Means Together
All four properties (definitions, hierarchy, version control, cross references) share one requirement. They cannot be inferred by the model at query time. The information is not in the retrieved text, not reliably. It has to be encoded on the document during ingestion, so that when retrieval fires, the model receives the clause and everything that clause depends on to be legally complete. These four are the minimum a document has to carry. Which raises the obvious next question. Can smarter retrieval or a bigger model compensate for a corpus that has not been prepared this way? The short answer, from the same engagement, is no.
Why Smarter Retrieval Does Not Rescue Unstructured Documents
The counter argument that keeps coming up in vendor pitches is that better retrieval mechanics can close the gap without doing the document preparation work. Query rewriting. Query expansion. Hypothetical document embeddings. Step back prompting. Practitioners running these techniques against legal corpora at scale have consistently reported the same result. Grounding gets worse, measurably.
The reason is precision. General domain benchmarks measure retrieval quality largely on recall, on how much relevant material gets surfaced. Legal work punishes recall the moment it drags along wrong material. A semantically similar clause from a different contract is noise. A related provision from the wrong jurisdiction is noise. A superseded version of the right clause is noise. Any of them shows up in the model’s context, and the model does what models do. It reconciles what it was given, produces a fluent synthesis, and hands back an answer that blends the correct source with the incorrect ones. The output reads authoritative and is wrong on the details that matter.
Query rewriting introduces the model’s own inferential drift at the earliest possible point. The model is asked to imagine what the ideal query looks like and search using that. The imagined query is broader than the real one, or narrower, or shifts the meaning slightly. Retrieval returns passages relevant to the model’s rewriting. The final answer sits one step further from the source than it needed to, and there is no way to trace which distance the rewriting introduced.
This is the same failure mode inside the recent sanctioned filings, wrapped in more sophisticated retrieval infrastructure. Ambiguous or partial context goes in. Plausible connections come out. Cited cases that were never decided end up in the final synthesis.
Fig 8 – The pattern behind the recent AI failures in legal work. At every step, the failure comes from what was not carried into the next step.
The Clixlogix AI team has seen the same principle play out at the extraction stage in production. A model that sees the whole document with its structure intact produces better output than a pipeline that strips structure before reasoning. The lesson at the retrieval layer sits in the same place. Retrieval that uses encoded document structure gives the model complete context. Retrieval that leaves the model to reconstruct structure at query time gives the model an assembly job on top of a reasoning job. Legal RAG that holds up in production respects the preparation the documents already carry.
Reranking is useful. Hybrid search is useful. Query classification is useful. All of them work better when the corpus underneath is structured. They do not replace the document preparation work.
Which leaves one honest question. If the four properties are encoded, the retrieval is disciplined, and the model is fed complete legal context, is the system now safe to run without a human? Not exactly. There are two categories of work that no amount of document preparation solves, and any agentic legal AI honest about its limits has to route those through a human.
Where the System Still Needs a Human
Which brings the piece back to the original question. Prepared documents. Disciplined retrieval. Model fed complete legal context. Is the system safe to run without a human? Not exactly. Two categories of work sit outside what document preparation solves, and any legal AI worth deploying has to route them through a human.
Partial Applicability
The system can surface a clause, its definitions, its cross references, its position in the hierarchy, and the current version. Whether that clause actually applies to a new fact pattern is a different kind of question. It is analogical reasoning. It is judgment. And the evidence on how well models handle it, even with well built retrieval infrastructure, is not encouraging.
The Stanford RegLab study led by Varun Magesh and Daniel Ho, published in the Journal of Empirical Legal Studies in 2025, is the benchmark to know here. The team tested LexisNexis’s Lexis+ AI and Thomson Reuters’s Westlaw AI-Assisted Research, both of which use proprietary RAG systems specifically built for legal work. Lexis+ AI hallucinated on 17% of queries. Westlaw AI-Assisted Research hallucinated on 33 to 34%. These are the two most sophisticated commercial RAG systems in the legal market, priced accordingly, and they still get roughly one in five to one in three answers wrong. Preparation helps. It does not eliminate the model’s disposition to fill in gaps when a question requires judgment the retrieval could not provide.
A system honest about this routes applicability questions through a human. The model can pull the clause. It can draft a first position. The determination is a lawyer’s work.
Reproducibility
The second gap is structurally worse for anything that has to withstand an audit. Model generation is probabilistic. Run the same query against the same corpus twice, and the answers are slightly different. Even when the model is configured for maximum determinism, documented output variance of up to 15% across identical runs has been measured, with other studies observing accuracy swings of 70 percentage points across identical conditions in code generation settings.
For research and drafting, this variance is a tolerance the firm can absorb. A lawyer reads the draft, corrects it, uses it as a starting point. For compliance checks, court facing outputs, and regulator facing determinations, that same variance becomes an audit finding waiting to happen. The regulator does not accept “the model said something different last time” as an answer.
The EU AI Act’s approach points at where the industry is heading. It relies on standards bodies to define acceptable output variance for AI systems used in regulated applications, and production deployments are converging on infrastructure that treats reproducibility as a first class requirement. Prompt versioning under change control. Model and retrieval index snapshots per deployment. Averaging across three or more runs to absorb variance. Threshold based quality gates in CI/CD.
The design decision for legal work follows from this. Route audit facing outputs through deterministic logic. Let the model handle research, synthesis, and drafting where variance is tolerable. Firms that confuse the two categories end up with audit findings they cannot explain.
The Boundary
This is the boundary the sanctioned firms failed to draw. The Charlotin AI Hallucination Cases Database has catalogued roughly 1,598 court cases involving AI-fabricated content globally by mid-2026, with US courts imposing over $145,000 in AI-filing penalties in the first quarter of 2026 alone. In April 2026, Sullivan & Cromwell apologized to a federal bankruptcy judge for AI-generated content in a filing. In every case, the failure is the same. Generative output was treated as the deliverable when it should have been one input into a workflow a human owned. Prepared documents and disciplined retrieval move the ceiling. They do not remove the requirement that a lawyer stand behind what leaves the firm.
Fig 9 – The design decision every legal AI deployment makes. Everything on the left is where preparation and retrieval earn their keep. Everything on the right is where a lawyer decides.
The Arithmetic Behind the Design Decision
The decision every legal CXO now has in front of them comes down to a spend allocation. Money into model upgrades. Money into document preparation. The arithmetic makes the case clearer than the argument does.
Start with what a query actually costs. Frontier model API pricing across the three major providers, as of July 2026, sits in a narrow range.
Provider
Model
Input ($/M tokens)
Output ($/M tokens)
Anthropic
Claude Sonnet 4.6
$3.00
$15.00
OpenAI
GPT-4o
$2.50
$10.00
Google
Gemini 2.5 Pro
$3.50
$10.50
Median (used below)
—
$3.00
$10.50
Fig 10 – Frontier legal AI model API rates, July 2026. Numbers are per million tokens.
Take a mid size firm as the working example. 500 contracts under active management. 2,000 attorney queries per month against the agentic legal AI system. Both figures are illustrative. Any CXO reading this can substitute their own volume.
Approach 1 — Model First, No Document Preparation
The retrieved context on each query averages 8,000 tokens because the system drags in redundant, tangentially relevant, and superseded passages. Generation averages 1,000 tokens. At median frontier-model rates, the arithmetic works out this way:
Worked math — Approach 1
Input tokens per query
8,000
× $3.00 / 1M
Output tokens per query
1,000
× $10.50 / 1M
Per query cost
$0.035
Queries per month
2,000
illustrative
Monthly API cost
$70
Annual API cost
$840
3-year API total
$2,520
That number is deceptively small, and it hides the actual cost. Every query with incomplete context produces an answer that a lawyer has to unwind. Every hallucination costs partner review time. Every escalation is a billable hour spent verifying what the system already claimed. The visible line item is $2,520. The invisible line items are what put firms in the news.
Approach 2 — Document Preparation First, Then the Model
Ingestion of the 500 contract corpus is largely automated. Pattern matching handles cross references, definitions, and standard citation formats without a language model touching them. A trained analyst reviews the pipeline’s confidence flags on the 10 percent of documents where automation is ambiguous. Once the corpus is prepared, retrieved context per query drops from 8,000 tokens to roughly 3,000, because retrieval now returns tight, defensible passages. The economics reshape themselves.
Worked math — Approach 2
Upfront ingestion (one-time)
Analyst review, 50 flagged docs × 20 min @ $75/hr
$1,250
Automated tooling + infrastructure
$5,000
Ingestion subtotal
$6,250
Annual API layer (reduced context)
Input tokens per query
3,000
was 8,000
Per query cost
$0.019
was $0.035
Monthly ($38) × 12
$456
Annual maintenance
Amendments, updates (10% corpus/yr)
~$800
3-year total ($6,250 + $456×3 + $800×3)
$10,018
The Comparison
Approach 1 — Model first
~$2,520 / 3 yrs
Plus every hallucinated answer someone has to catch, plus a sanctioned filing if one gets through.
Approach 2 — Structure first
~$10,018 / 3 yrs
Front loaded, predictable, materially lower error rate.
Delta: ~$7,500 more in visible spend for a system that produces defensible output instead of fluent output. Gap closes further as query volume rises, because Approach 1’s API cost scales linearly with queries and Approach 2’s does not.
The Risk Premium
The Charlotin AI Hallucination Cases Database has catalogued roughly 1,598 court cases involving AI-generated content globally by mid 2026. Single matter sanctions have reached $109,700. US courts imposed over $145,000 in AI-filing penalties in the first quarter of 2026 alone. The risk premium on Approach 1 is difficult to price precisely, and it is not zero. It scales with query volume in a way the per query API cost does not.
The pragmatic reading of this arithmetic is straightforward. Model spend and document preparation spend are not substitutes for each other. Structure spend is a one time investment with predictable maintenance and a lower error ceiling. Model spend scales with usage forever and never closes the structural gap it is meant to compensate for. Firms treating this as a sequence, structure first and model second, buy a lower total cost of ownership and a lower audit risk in the same purchase.
The Bottom Line
The recent wave of sanctioned AI filings is not evidence that agentic legal AI does not work. It is evidence of what happens when firms deploy the model as the answer instead of as one step in a workflow that carries verification, provenance, and judgment around it.
The firms getting this right are treating AI investment as an architecture decision, not a purchase. They are preparing documents so retrieval can surface the specific clause, definition, version, and cross reference the model needs, then letting the model handle the narrow synthesis job that models are actually good at. They are drawing a clean boundary between what the system can automate and what must route through a lawyer. And they are running the arithmetic honestly, which shows structure spend as a finite upfront investment against model spend that scales with usage forever and never closes the gap it is meant to compensate for.
The firms getting this wrong are the ones the profession is now reading about.
Structure first, model second. That is the investment sequence that separates the firms whose AI use scales safely from the firms making the news.
Building Agentic Legal AI That Holds Up in Production?
Clixlogix builds agentic legal AI systems for legal teams and legal tech platforms at exactly this decision point. Document preparation pipelines, retrieval architecture, workflow integration, and the sequence of engineering decisions that separate systems that hold up in production from systems that produce the headlines.
When a legal CXO or head of legal operations is trying to answer whether their existing corpus and their existing vendor stack will hold up as usage grows, this is the conversation.
AI hallucinations in legal work happen when the model produces confident output about content it never actually received. In most sanctioned cases, the underlying cause is upstream. Retrieval fetched an incomplete picture of the source document, and the model filled the gaps with plausible sounding invention. A citation gets fabricated because the model had to synthesize an answer without the actual case in its context. A defined term gets misapplied because the retrieval returned the obligation clause without the definitions section. Fixing hallucinations reliably means fixing what the model receives, not just prompting it more carefully.
A better model with the same incomplete context produces more fluent wrong answers, not more correct ones. The Stanford RegLab study published in the Journal of Empirical Legal Studies in 2025 tested LexisNexis Lexis+ AI and Thomson Reuters Westlaw AI-Assisted Research, both built on proprietary RAG systems tuned for legal work. Both hallucinated on 17 to 34 percent of queries. The bottleneck was the retrieved context, not the reasoning model. Model upgrades are the wrong lever until the corpus feeding the model has been prepared for machine reasoning.
Document structure in this context means the four properties documents have to carry into retrieval so the model receives a legally complete picture. Definitions have to travel with the clauses that use them. Documents have to be tagged with their authority level so retrieval knows a regulation outranks a memo. Amendments have to be linked to the original document so retrieval returns the current version. And cross references have to be captured explicitly so a clause that depends on another section pulls that section along with it. Encoding these four properties at ingestion is what makes agentic legal AI defensible in production.
At ingestion, each document gets tagged with metadata indicating its legal weight. A regulation gets tagged as primary authority. Case law gets tagged as secondary. Internal memos and past advice get tagged as advisory. When a query comes in, retrieval scores results by both semantic similarity and authority, so a regulation matching the query surfaces ahead of a memo matching the same query. The tagging step is largely automated using filing metadata and document type classification, with a light human review layer for documents where the automation is uncertain.
Version control at the retrieval layer means tagging each document with an effective date and, where applicable, a supersession relationship. When a contract is amended, the amendment document is linked to the original with clause level supersession metadata. Retrieval then filters to the current version by default, so an attorney asking about a specific clause receives the amended text rather than the superseded original. Without this, both versions live in the corpus with equal weight, and the model returns whichever the search happens to match first.
Legal drafting follows a small set of standard citation formats such as “pursuant to Section X,” “as defined in Article Y,” and “subject to Clause Z.” Pattern matching on these formats during ingestion captures cross references as explicit edges in a document graph. When retrieval returns a clause containing one of these references, it also pulls the referenced section into the same context window. Most legal documents follow patterned enough conventions that automated extraction handles the majority of cross references, with human review reserved for unusual document types or edge cases.
Tasks where the model is doing mechanical work over prepared documents, and where a human is not required to sign off on the specific output, can be fully automated. Locating a defined term across a corpus. Flagging missing required clauses. Summarizing contract obligations. Building the first draft of a fifty state survey. Checking whether a specific provision exists in the document set. These tasks have deterministic success criteria and low tolerance for the model to reason beyond what retrieval provided, which is where automation earns its place.
Tasks that require analogical reasoning, jurisdiction-specific judgment, or output that must be defensible under audit should route through a human. Determining whether a clause applies to a new fact pattern. Resolving conflicts between authorities of different weight. Assessing enforceability in a specific jurisdiction. Any output that will be defended to a regulator. Any output that goes into a court filing. These are decisions where the model surfaces material and drafts a starting position, and a lawyer decides.
A single step approach asks the model to read the corpus, judge authority, decide which version is current, resolve cross references, and produce a defensible citation in one probabilistic pass. Any gap in the retrieved context gets filled by the model with plausible sounding invention. A multi step pipeline separates these decisions. Documents are prepared at ingestion so retrieval can return legally complete context. Retrieval returns the specific clause plus its dependencies. The model then handles the narrow synthesis job with a complete picture in front of it. The failure surface shrinks because each step has one job and inspectable output.
Extraction quality is how accurately the system converts a document into machine readable text at ingestion, preserving characters, layout, and structure. Retrieval quality is how accurately the system fetches the right passages from that extracted corpus when a query arrives. Both matter. A pipeline that extracts perfectly but retrieves badly hands the model incomplete context. A pipeline that retrieves well but from badly extracted documents returns garbled text. The recent sanctioned filings failed at the retrieval layer specifically, on corpora that were technically well extracted but not structurally prepared for machine reasoning.
Share this blog
Summarise this Blog with
ClaudeChatGPTGeminiPerplexity
Written By
Chief Executive Officer @ Clixlogix
Pushker is the founder of Clixlogix. Give him a messy operation and he finds the leverage point, then builds the fix himself. He works at the edge of what AI can actually do inside a business, and writes about what he finds there.
DM Us
