takes a five-minute change and returns eight clear sections. Selections. Motion gadgets. Dangers. Open questions. Every part reads prefer it was written by somebody who was paying consideration.
Learn the underlying transcript, although, and you discover that two of these sections had been inferred from a single ambiguous sentence, one was invented totally, and three had been pattern-matched from the mannequin’s prior on what a gathering abstract ought to comprise. Assured, formatted, structurally indistinguishable from a abstract of a gathering the place these issues truly occurred.
This isn’t a hallucination drawback within the ordinary sense. The mannequin shouldn’t be making up a reality concerning the world. It’s making up a reality concerning the assembly. And the failure mode shouldn’t be seen within the output. It’s simply confident-sounding textual content that the reader can’t simply confirm towards the supply.
There’s a title for this failure mode in one other discipline, and it’s older than language fashions. It’s what occurs whenever you do estimation with out identification.
This text shouldn’t be a brand new summarization benchmark. It’s an argument for a design sample that I’ve not seen handled because the central design constraint in AI engineering literature: deal with LLM-generated summaries as structured claims over a supply, require every declare to declare its assist class, and constrain evaluation levels to allow them to solely weaken unsupported claims slightly than make the output smoother. I’ll stroll by what that appears like in follow, what it produces, and the place it breaks.
The lacking step
Causal inference is the analytical custom that formalizes the distinction between figuring out a amount and estimating one. Identification is the argument that the information you could have can assist the declare you wish to make. Estimation is the process that produces a quantity as soon as identification is settled. The order shouldn’t be negotiable. You can’t estimate a remedy impact you haven’t first argued is identifiable out of your observational information, as a result of the ensuing quantity is meaningless. It seems to be like an impact. It’s not an impact.
Practitioners who work in observational settings spend a considerable fraction of their time on identification. They draw causal graphs. They argue about confounders. They distinguish between what the information can assist and what the information can’t. The estimation step, when it lastly comes, is commonly the straightforward half.
Now contemplate what an LLM summarizer does. It receives a transcript. It produces structured claims concerning the content material of that transcript: selections made, commitments accepted, dangers raised, subsequent steps assigned. Every declare is, in an actual sense, an estimate of a latent amount. The choice was made or it was not. The dedication was accepted or it was not. The abstract is asserting a price for every of those portions.
There is no such thing as a identification step. The mannequin doesn’t ask whether or not the transcript incorporates sufficient proof to assist the declare. It produces the declare as a result of the format requires one.
LLM summarization behaves like observational evaluation, however it’s usually deployed with out something resembling an identification step.
The AI engineering literature has not been silent on the underlying drawback. Hallucination detection, calibrated uncertainty, selective prediction and abstention, RAG grounding, quotation verification, factual consistency, and claim verification: every of those is a severe line of labor, and every addresses an actual layer of the failure. What they’ve in frequent is that they deal with fabrication as a mannequin habits to be measured, scored, or suppressed after the very fact.
Identification is a unique layer. It doesn’t rating the output for trustworthiness. It adjustments what the mannequin is allowed to say within the first place by requiring each declare to declare what it’s and the place it got here from. The 2 layers are complementary. A pipeline that does identification properly nonetheless advantages from calibration and grounding work downstream. A pipeline that does solely the downstream work is filtering output that ought to by no means have been produced within the kind it was produced.
What identification seems to be like for a transcript
Identification in observational information is a query about what the information can assist. Identification for a transcript is similar query, narrowed to a selected supply. Given this transcript, what will be noticed instantly, what will be inferred with acknowledged assumptions, and what can’t be supported in any respect?
That’s the complete transfer. Each declare a summarizer produces ought to declare which of these three classes it belongs to. Noticed claims level to a selected span of the transcript and assert nothing past what that span says. Inferred claims declare the belief being made and the proof the inference is bridging. Suggestions declare that they’re the mannequin’s suggestion, not the contributors’ choice.
A summarizer that can’t place a declare into a type of classes has no enterprise producing the declare. The correct output in that case shouldn’t be a smoother declare. It’s no declare.
That is uncomfortable for the patron of summaries, as a result of it means many sections can be empty when the underlying dialog was skinny. That discomfort is the purpose. It’s data. It tells the reader that the assembly didn’t, in reality, produce eight sections of substance, no matter what the summarizer needed to put in writing.
A pipeline that enforces the self-discipline
The structure follows from the framing. Three LLM levels and a deterministic renderer.
Picture by Creator
The primary stage extracts structured info from the transcript. Speaker turns, specific commitments, specific selections, specific portions. This stage is intentionally conservative. It’s allowed to overlook issues. It’s not allowed to invent them.
The second stage synthesizes these info into declare objects throughout eight sections. Every declare carries a label: noticed, inferred, or suggestion. Every declare carries a pointer to the proof within the extracted info. Synthesis is the place the analytical work occurs, and it’s also the place the mannequin is almost certainly to float.
The third stage audits. That is the stage that does the identification work, and the constraint on it’s the a part of the design that issues most.
The audit stage can’t rewrite the evaluation into one thing smoother. It can’t add a better-sounding suggestion. It can’t invent lacking context.
It’s given a bounded set of operations and forbidden from doing anything. It could delete a declare. It could downgrade a declare from noticed to inferred, or from inferred to suggestion. It could transfer a declare to a extra acceptable part. It could exchange a declare with an specific insufficient-evidence placeholder. It could collapse a whole part when nothing in it survives evaluation.
Something not on this checklist is forbidden, together with writing higher claims.
Picture by Creator
The replace_with_insufficient_evidence operation deserves its personal line. It’s the system actually typing a placeholder into the output the place a assured declare was. That’s identification work made operational. The reader sees, in prose, precisely the place the synthesis stage produced a declare that the supply couldn’t assist.
Why the asymmetry issues. A reviewer that’s allowed to enhance the evaluation turns into one other supply of the identical drawback the system is making an attempt to unravel. A reviewer that’s solely allowed to weaken or take away can solely fail in a single course: by being too cautious. That could be a tolerable failure mode. The other shouldn’t be.
What the design produces, and what it refuses to supply
This isn’t a benchmark. It’s a small fixture-based stress check designed to verify whether or not the structure produces the habits it was constructed to supply. Three transcripts will not be sufficient to make normal claims about LLM summarization. They’re sufficient to verify whether or not a selected design alternative has the implications the design predicted.
The fixtures are: a call assembly during which a pricing mannequin was chosen amongst three actual options, a working session that surfaced a measurement drawback with out resolving it, and a skinny two-person sync that contained virtually no choice content material.
What didn’t occur. Throughout the three runs, the pipeline produced zero fabricated commitments and nil ungrounded portions. That is what the structure is designed to make tougher. A declare can’t survive the pipeline if it doesn’t have a pointer to proof, and the audit stage can’t manufacture proof to maintain a declare alive. The end result shouldn’t be a assure. The deterministic renderer is the one stage that offers ensures. Extraction, synthesis, and audit are nonetheless LLM calls and might nonetheless fail. The purpose is that the structure pushes their failures towards removing slightly than towards fabrication, and the fixtures are in line with that.
What did occur. The end result that I discover extra fascinating is the abstention price.
Throughout three fixture transcripts, the share of empty part slots rose from 17% to 58%.
Throughout all three fixtures: 0 fabricated commitments, 0 ungrounded portions.
Picture by Creator
On the wealthy choice assembly, the pipeline left seventeen % of part slots empty or changed with the insufficient-evidence placeholder. On the working session, the determine rose to 25 %. On the skinny sync, it reached fifty-eight %. The system produced roughly three and a half instances as many empty sections when the enter sign was skinny in comparison with when it was wealthy.
That’s the habits the design is making an attempt to supply. A summarizer that fills the identical eight sections no matter enter shouldn’t be summarizing. It’s producing output that conforms to a template. The template is doing the work, and the mannequin is the beauty end.
A summarizer that abstains in proportion to the thinness of the enter is doing one thing totally different. It’s treating the transcript as a supply whose content material varies, and it’s letting that variation present up within the output. The empty sections will not be failures of the mannequin. They’re the mannequin declining to say what the supply doesn’t assist.
Excerpts from the decision-meeting fixture, with the explicit labels surfaced inline.
Picture by Creator
Studying the end result. The labels will not be ornament. They alter what the reader does with the output. An noticed declare invitations verification towards the transcript. An inferred declare invitations scrutiny of the belief that produced it. An insufficient-evidence placeholder invitations the reader to both take a look at the supply themselves or settle for that the assembly didn’t, in reality, produce a declare of that form.
The objection from the patron
There may be an argument that vacant sections are a usability drawback. The reader anticipated a abstract. The reader received a partial abstract with specific gaps. The reader has to do extra work.
That objection deserves a direct reply. The reader who received a fluent eight-section abstract of a five-minute change was already doing extra work, simply invisibly. They had been going to learn the abstract, act on it, and sooner or later uncover that two of the motion gadgets weren’t truly agreed to and one of many dangers was by no means raised. The price of that discovery is excessive. It’s paid in misallocated conferences, missed commitments, and the gradual erosion of belief within the tooling.
Trustworthy vacancy pushes the associated fee ahead. The reader sees the hole instantly and might determine methods to deal with it. Open the transcript. Ask a participant. Deal with the assembly as inconclusive. Every of these is a greater response than appearing on a assured abstract that was generated from a confidence the supply didn’t earn.
This is similar commerce observational analysts make once they refuse to report some extent estimate with out identification. The buyer would like a quantity. The analyst declines. The choice the patron makes from no quantity is, on common, higher than the choice they’d have created from a quantity the information couldn’t assist.
Generalizing the sample
The structure transfers. Any LLM workflow that produces structured claims from a supply will be reframed as observational evaluation and given an identification layer.
Doc evaluation for authorized discovery. Affected person observe summarization. Buyer name evaluation. Code evaluation summaries. Every of those is presently deployed as a one-shot technology drawback, with a mannequin producing structured output from a supply and the patron trusting the end result. Every of them has a model of the identical failure mode the assembly summarizer has, and every will be made extra auditable with the same structure: an extraction stage that’s conservative about what it pulls from the supply, a synthesis stage that produces labeled claims with proof pointers, and an audit stage that’s forbidden from including or strengthening something. The implementation and the danger profile differ throughout these domains. The sample transfers. The specifics don’t.
The labels and the proof pointers will not be non-compulsory options. They’re the identification step made operational. A declare with out a label shouldn’t be identifiable. A declare with out an proof pointer can’t be audited. The audit stage’s monotonic-weakening constraint is what prevents identification work from being undone by a mannequin that wishes to supply smoother output.
What this implies for the folks constructing these techniques
Calibrated uncertainty estimates are invaluable. Hallucination benchmarks are invaluable. Grounding and quotation work are invaluable. None of them substitute for the self-discipline of refusing to supply a declare that the supply doesn’t assist.
That self-discipline is lacking from many LLM techniques partly for cultural causes. The sphere grew out of machine studying, the place the purpose of a mannequin is to supply an output for each enter. The notion that the suitable output is typically no output shouldn’t be international to the literature, however it’s international to the default disposition of a generative mannequin skilled to fill in what comes subsequent. It’s, nonetheless, native to observational evaluation, the place the suitable reply to many questions is that the information can’t assist a solution.
So the methods for making LLM analytical techniques reliable might not come primarily from inside the LLM literature. They could come from disciplines which have already labored out what it means to do trustworthy evaluation beneath situations the place the supply is the binding constraint. Causal inference is a type of disciplines. Survey methodology is one other. Forensic accounting is one other.
The individuals who already know methods to refuse to estimate with out identification have an unusually good vantage level on what’s improper with present LLM analytical tooling, and what to do about it.
Causal inference taught a technology of practitioners to not estimate what they haven’t first recognized. LLM summarizers make the identical mistake, simply in prose as a substitute of numbers. The repair is not only a greater mannequin. The repair is to place again the step that observational evaluation by no means let go of, and to implement it with an structure that can not be talked out of doing the suitable factor.
Just a few closing pitfalls
- Treating the labels as beauty. If the labels will not be enforced upstream, they’re ornament. They must be assigned at synthesis with a pointer to proof and audited downstream towards that pointer. A synthesis stage that produces a label with out an proof pointer shouldn’t be doing identification work. It’s producing a class that appears like identification.
- Letting the audit stage be useful. That is the straightforward mistake. A reviewer that may add a suggestion, provide lacking context, or rewrite a careless declare feels helpful. It’s also precisely the failure mode the synthesis stage already has, simply dressed up as high quality management. Constrain the audit to a set set of weakening operations. The rest is the system arguing with itself.
- Complicated abstention with low high quality. A summarizer that returns principally empty sections on a skinny assembly shouldn’t be failing. A summarizer that returns assured eight-section output on the identical skinny assembly is failing, simply invisibly. The best way to guage these techniques shouldn’t be abstract completeness, it’s whether or not the abstention price scales with the sign within the supply.
- Reasoning from three fixtures to normal claims. Three transcripts are sufficient to verify whether or not a design alternative produces the habits it was constructed to supply. They don’t seem to be sufficient to make claims about LLM summarization normally. If you happen to construct a model of this, you will want your personal fixture set and your personal definition of what counts as the suitable degree of abstention on your use case.
The asymmetry that issues
A pipeline that may solely weaken its outputs has a single failure mode: it may be too cautious. A pipeline that may strengthen its outputs has each failure mode the literature has been documenting for the final a number of years.
Selecting the primary sort over the second sort shouldn’t be a technical choice. It’s a choice about what the system is for. If the system is for producing fluent textual content, the second sort wins on each metric. If the system is for producing claims a reader can audit earlier than appearing, solely the primary sort is defensible.
Most present tooling is constructed for the primary purpose and deployed as if it had been constructed for the second. Treating that hole as a methodological drawback slightly than a model-quality drawback is what adjustments the out there treatments.
Repository, analysis harness, and instance outputs can be found on GitHub. The complete pocket book walks one transcript by each stage and runs the eval harness throughout all three fixtures.
Workers Knowledge Scientist targeted on causal inference, experimentation, and choice science. I write about turning ambiguous enterprise questions into decision-ready evaluation.
Extra like this on LinkedIn 👇
