AI and systems
How to Stop AI Agents Making Things Up in Business Reporting
In short
An AI agent writing about your business data does not fabricate randomly. It fabricates in three specific places: it asserts causes it cannot see, it fills gaps in coverage rather than reporting them, and it produces confident single numbers where the underlying data only supports a range. Fix those three and the fabrication problem largely disappears.
The fix is architectural rather than a matter of better prompting. Below are eight controls, in the order they are worth implementing.
1. Never let the model produce the number#
The most reliable control is to keep generation and judgement in separate systems. A deterministic engine computes the figure. The model interprets, explains and evaluates, but never produces the value itself.
This generalises well beyond reporting. In a system generating architectural layouts, a constraint solver computes every coordinate and the model only expresses design intent as structured relationships, because a model asked to produce geometry produces plausible geometry that is frequently invalid. The same logic applies to a revenue figure.
If the model is describing something a query already computed, there is nothing for it to invent.
2. Forbid causal claims from aggregates#
The most common failure in AI-generated business reporting is a fluent sentence asserting why something moved. The aggregate does not contain the cause. Margin fell and headcount rose in the same month is a correlation the total cannot resolve, and the reader cannot tell which sentence in the paragraph is grounded and which is invented.
Make the rule explicit: the agent describes what moved and cites the source. It does not assert why. If a cause is genuinely available in the data, it must be cited to the specific record that shows it.
3. Show unavailable rather than estimated#
Anything the system cannot stand behind is displayed as unavailable, never interpolated or approximated.
This feels unhelpful and is the opposite. The first time a dashboard shows a number that turns out to have been inferred, every other number on it loses credibility. Withholding is cheap. Recovering trust is not.
4. Withhold partial coverage instead of footnoting it#
Where a metric is built from several sources and only some reported, do not render the total with a footnote. A footnote does not survive a screenshot, and the figure will be pasted into a board pack without it.
State coverage as a first-class property. One pipeline in production runs at 70 percent coverage of historical time data, with the missing 30 percent identified as records on deleted or inaccessible boards. That is a usable dataset precisely because the gap is known and stated. An unstated 70 percent is a liability.
5. Require four fields or reject the output#
Where an estimate carries real consequence, make completeness a validation rule rather than a style guideline. Every output must carry the tool used, the full assumption set, the precision range, and an explicit statement of what professional review is required. An output missing any of the four is invalid and is not published with a caveat.
The difference matters. Caveats get dropped under time pressure. A rejected payload does not.
6. Wire the confidence score to a gate#
A confidence score that does not change system behaviour is decoration.
Set a threshold below which the system does not proceed and instead surfaces the uncertainty. In one compliance system that threshold sits at 0.6, below which human review is mandatory for high-risk scopes, and every determination is logged with its score regardless. Set the gate before anyone is working to a deadline, because that is when the temptation to lower it arrives.
7. Never execute the user's words#
Convert natural language input into a structured object and validate it against a schema before anything downstream runs. Raw text is never executed as code and never passed directly to an API.
This is most of the prompt injection defence for a business system, and it is architectural rather than clever. If the model's output must survive a schema check before it can act, most injection attempts die at the boundary.
8. Make the audit trail structurally immutable#
Record what the system did and why, in an append-only store where update and delete are revoked at the database level rather than avoided by application logic.
"The application never deletes" is a policy. Revoking the permission is a guarantee. For anything that may need defending later, the difference is the whole point.
Two sequencing rules that prevent the problem entirely#
Do not put an AI layer over incomplete data. An agent reasoning over a partial picture produces confident output about a partial picture, which is worse than no output because it is not visibly partial. Sequence the intelligence layer after the data layer is complete. On one platform this meant deferring the AI insight capability until every data source was integrated, and deferring the financial views separately until an accountant had reviewed them.
Do not let the agent choose its own next step. Autonomy at the task level is fine. Autonomy at the sequencing level is where agent systems go wrong, because a wrong next step compounds for hours before anyone notices. A working pattern is that the agent completes its task, reports back with a pass or fail table per check, and stops. A human decides what follows.
The three-level validation model#
Worth adopting as the organising frame for all of the above. Errors block the pipeline. Warnings recommend and are surfaced. Information is observed and logged.
Most agent systems have one implicit level, which is log it and continue. Splitting into three gives a real stop mechanism without making the system unusable, because only the top level halts anything.
Common questions#
Will better prompting fix hallucination in reporting? It reduces frequency and does not change the failure mode. The controls above are structural, which is why they hold when the prompt does not.
Should the agent ever state a cause? Only when the cause is present in the data and cited to the specific record. Never inferred from an aggregate.
What if the business wants a single number, not a range? Give the number with its precision tier attached. Users do not object to a wide range. They object to discovering that a confident figure was a wide range in disguise.
How do we handle a data source that failed overnight? Record it as a failed run so the gap is visible, and show that period's coverage as partial. Never interpolate across it.
Does this slow the system down? The deterministic checks are faster than the model calls they replace. Routing most traffic away from the model is usually a performance improvement and a cost reduction at the same time.