AI · CX Operations

The most important thing your AI can say is “I don’t know”

By Roberto La Rosa · Fractional CX Director · October 2026 · 7 min read

This month I put an assistant on this website. Every answer in it is one I approved. The part I spent the most time on is the one where it says “I don’t know”.

When a question goes beyond those answers, it says so and repeats the question back. Then it offers two ways to reach me: a call, or an email that already contains the question. After two misses it stops trying and hands over.

It sounds like a detail. It’s the part most AI projects in customer service get wrong, and recent research explains why.

Machines are trained to guess

In September 2025 OpenAI published a paper on why language models hallucinate. The answer is less mysterious than it sounds. Models are graded mostly on accuracy, the share of questions they get right. Under that scoring, guessing always beats admitting you don’t know, the same way a student gains nothing by leaving a multiple-choice answer blank.

The numbers in the paper make the point. On the same factual test, an older model (o4-mini) abstained only 1% of the time and got 75% of its answers wrong. A newer one (gpt-5-thinking-mini) declined to answer 52% of the time and got 26% wrong. Accuracy was almost identical: 24% against 22%. The older model looked better on the scoreboard because it guessed.

OpenAI’s proposed fix is about the scoreboard: penalise confident errors more than uncertainty, and give partial credit for an honest “I don’t know”.

The “I don’t know” was there by default

Anthropic looked at the same problem from inside the model. In March 2025 it published research tracing how Claude decides whether to answer. It found a circuit that is on by default and makes the model say it doesn’t have enough information. When the model recognises something it knows, another signal switches that default off.

Hallucinations happen when that switch misfires. The model recognises a name it has seen, the default “I don’t know” gets suppressed, and it fills the gap with something plausible and false.

So in a sense the “I don’t know” is the starting point. What we build around the model decides whether it survives.

What a confident wrong answer costs

In February 2024 a Canadian tribunal ruled against Air Canada. Its website chatbot had told a customer he could claim a bereavement discount after travelling. The airline’s policy said otherwise. Air Canada argued that the chatbot was responsible for its own actions. The tribunal disagreed: the company is responsible for all the information on its website, chatbot included. The damages were a few hundred Canadian dollars. The principle is worth far more: what your bot says, you said.

In April 2025 Cursor, a fast-growing software company, watched its support bot invent a policy. Users asked why they were being logged out when switching devices, and the bot explained a rule that didn’t exist. Some cancelled their subscriptions. The co-founder had to reply in public: “We have no such policy.” Since then, AI replies in their email support are labelled as AI.

In May 2025 Klarna’s CEO told Bloomberg that cost had been “a too predominant evaluation factor” in how they organised AI customer service, and that “what you end up having is lower quality”. The new line: customers must know there will always be a human if they want one.

What customers are actually afraid of

Gartner surveyed 5,728 customers at the end of 2023. 64% said they would prefer that companies didn’t use AI in customer service. Their top concern was that AI would make it harder to reach a person. Wrong answers were on the list too.

Read together, these cases point the same way. Customers accept that a bot has limits. They react badly when it hides them and blocks the way to a person.

Does admitting uncertainty cost trust?

A little, and that’s useful. In a 2024 study by researchers at Princeton and Microsoft Research, 404 people used an AI search tool to answer medical questions. When the AI said “I’m not sure, but…”, people agreed with it less often (from 80.9% to 74.8%) and were less confident in its answers. They also got more answers right, because they leaned less on the wrong ones.

Trust went down slightly. Accuracy went up. In customer service I’d take that trade every time. I want customers to trust the right answers and question the shaky ones.

We trained people to guess first

None of this is new to anyone who has worked the floor. I started as an agent, measured on handle time, without anyone explaining why. When the clock is the KPI, “let me check and get back to you” feels like failing. A confident answer closes the contact.

That’s the same incentive OpenAI describes in its models. If the scorecard rewards closing and ignores being wrong, people learn to guess, and so do machines. Customer happiness is downstream of organisational happiness. Here it’s also downstream of what we choose to measure.

How to design a good “I don’t know”

This is what I built into the assistant on this site, and what I’d ask of any AI in a support operation.

  1. Say it early. Microsoft’s design guidance for conversational agents suggests no more than two fallback attempts before sending the user elsewhere. After two misses, stop trying.
  2. Show what you understood. Repeat the customer’s question back, so they know it was read.
  3. Hand over with the question attached. Nobody should have to explain their problem twice.
  4. Set an expectation. Say who will answer, and roughly when.
  5. Measure it. Count how often the AI says “I don’t know”, read those questions every week, and turn the frequent ones into new approved answers.

The last point is the one most teams skip. The share of conversations closed without an agent says little on its own. I’d look at how many of those conversations come back the next day.

Do you know how many times a day your AI says “I don’t know”? And how many times it should have?

Sources

Related: AI deflection fails when it is implemented under pressure · Your CSAT might be lying to you

Roberto La Rosa Fractional CX Director and founder of Happiness Harbor, Milan. 20+ years leading customer operations across European scale-ups, with teams of over 1,850 agents in 4 countries. Started his career as an agent. Author of "AI Alone Is Not Enough" and the white paper "CX Is Not Delegated. It Is Built." Published in CMI Magazine. Currently studying organizational psychology. LinkedIn

Considering bringing in a fractional CX director to lead this end to end?

How often does your AI say “I don’t know”?

45 minutes to look at your fallbacks and handoffs together, no cost, no deck.

Book a free call