Stopping an AI agent inventing things

A language model that does not know something does not experience not knowing. It produces the most plausible continuation, and a plausible continuation of "what does that cost?" is a number.

You cannot eliminate this. You can make honest refusal the path of least resistance, which in practice removes almost all of it.

1. Say "I don't know" is allowed

The most effective single line in most prompts:

If you do not know, say you do not know and offer to have someone follow up. Never guess.

Without it, not knowing reads to the model as a problem to be worked around rather than a valid answer. This one sentence removes more invention than any other change on this list.

2. Ground it, and say so

Attach a knowledge base and instruct it to answer only from that plus the prompt. Retrieval alone is not enough — you have to tell it that the retrieved material is the boundary, not a suggestion.

The base has to be written for retrieval, which is not how help centres are written. One question per document, answer in the first two sentences, facts stated as facts.

3. Name the categories it must never improvise in

Generic caution does not survive a specific question. Be concrete:

• Prices not on the published list.
• Delivery dates it cannot look up.
• Whether something is covered, included, or under warranty.
• Anything medical, legal or financial about this individual.
• What a competitor does.

For each, say what to do instead. A refusal with no alternative is a dead end, and the model will invent its way out of a dead end.

4. Give it tools instead of memory

The most reliable fix for "what's my order status" is not a better prompt. It is a function that looks the order up.

Facts the agent can fetch are facts it cannot get wrong. Facts it has to recall from a prompt are facts it can approximate. Move anything specific, current or per-customer into a function call, and the surface area for invention shrinks to almost nothing.

5. Make handover cheap

An agent with no exit will improvise, because improvising is the only move left. An agent that can hand over will hand over.

Name the conditions, and make the handover a real thing that lands in front of a real person — handover is a feature, not a failure.

Testing for it

Happy-path testing will not find this. Deliberately ask:

1. Something plausible that is not in the knowledge base at all.
2. A price for a product you do not sell.
3. A question with a false premise — "when you cancelled my order last week…"
4. A question about a competitor.
5. Something in a language you did not configure.

The correct answer to all five is a graceful "I don't know / that's not something I can help with, let me get someone". If any of them produce a confident answer, you have found your problem before a customer did.

Then read the transcripts

Testing catches the categories you thought of. Reading fifty real conversations catches the ones you did not, and it is the only method that reliably works. Pay particular attention to any answer that sounds more specific than your knowledge base could support — that specificity came from somewhere, and it was not you.

You are not aiming for an agent that is always right. You are aiming for one that is never confidently wrong.