Everything your AI says is a guess.Some guesses are just very good.

Ask a calculator what 2 plus 2 is, and you get 4, on every calculator, for the rest of your life. An AI chatbot answers a question about your business in a voice that sounds just as sure, but it is a very different kind of answer.

People say that AI sometimes “hallucinates”, meaning it makes things up, as if that were a rare glitch. I stopped thinking about it that way a while ago. I think everything an AI writes is a guess. Most of the guesses are good and some are wrong, and the AI often sounds equally confident either way.

That may sound like a small difference in words, but I learned the hard way that it changes how you should ask AI for advice.

How a missing number cost me months

I market my books with ads on Amazon and on Meta (Facebook and Instagram). Amazon shows me a figure for my ads that compares what I spend with what they sell. Meta can’t show me that, because Meta doesn’t see what happens in the Amazon shop. Whether a Facebook ad sold a book stays invisible.

For months I asked AI to analyse my ads and tell me what to do. The answer was always the same: put your money into Amazon ads, because there the numbers show that it works. The AI treated the money I spent on Meta as money going out with nothing coming back, because no sale was linked to those ads.

I saw that gap right away. But when I asked how to find out what really worked, the AI didn’t get any further. It just kept saying: use Amazon. In the end I did what it said, switched everything to Amazon and turned Meta off. Almost nothing sold: Amazon alone didn’t bring enough clicks.

So I set up my own test. For one book, I switched off all Amazon ads and ran only Meta ads for a while. Suddenly the picture was completely different: more clicks, cheaper clicks, more books sold.

Today a book with Meta ads gets no Amazon ads at the same time. It isn’t measurement in the usual sense, but I know its sales come either from Meta or from people who find the book on their own.

The AI worked with the numbers it had and never asked which ones were missing. It felt a bit like a child who hears one fact and says “so that’s how it is”. If I had just followed the advice, I would still be where I started.

Why I think every answer is a guess

A language model, the technology behind chatbots like ChatGPT, Claude and Gemini, writes by predicting which words are likely to come next. On its own, without a web search, it doesn’t look anything up. When you ask it for the capital of France, “Paris” comes out because the model saw that link in the texts it learned from an enormous number of times.

The model does store facts like this in a way, but none of them carries a stamp that says “checked”. At most it has a sense of how sure it is, and when it is wrong, it uses the same process as when it is right.

Three researchers at the University of Glasgow argued in 2024 that “hallucination” is the wrong word for this. They prefer “bullshit” in the sense of the philosopher Harry Frankfurt: talk that doesn’t care whether it is true.

I would use a plainer word: wrong. The AI isn’t lying either, because to lie you have to know the truth and hide it. An AI that states something false in a calm, sure voice is usually just mistaken.

That sure voice is partly trained, though. AI models are graded on tests a bit like students. Researchers at OpenAI and Georgia Tech wrote in 2025 that most of these tests reward a guess over “I don’t know”. Their example: if a test asks for someone’s birthday, a guess has a 1 in 365 chance of earning a point, while “I don’t know” earns nothing.

The rarer your question, the wilder the guess

If you tell an AI that the sun goes around the earth, it will politely correct you, because the right answer is in countless texts. Now ask it about your own small market. How do customers in your trade decide? What does a typical small firm in your niche earn?

A 2023 study found that language models answer much less reliably when a fact appeared in only a few of the texts they learned from. For many business questions, I suspect there isn’t much to learn from, so the AI probably fills the gap with what sounds normal for businesses in general.

And it guesses in the direction you point

Your own question matters too, because everything you write in it shapes the guess. If you write “I think our price is the problem”, the answer will most likely be about your price.

A 2023 study by Anthropic, the company behind Claude, tested five AI assistants. All of them tended to go along with what the user believed, even when it wasn’t true.

A new study from Google DeepMind looked at AI agents, which carry out tasks on their own, like a customer service bot that can rebook your flight. The researchers gave each task one wrong tip from the user that sounded reasonable. In the flight example, the customer says that upgrading to business class will solve the problem.

“Agents say yes to bad advice.”

Title of the Google DeepMind study, September 2026

All five AI models in the study did worse with the wrong tip than without it. In the worst case, a model’s score fell by almost half.

Many AI models now think a question through in a hidden draft before they answer, and what the study found in those drafts stuck with me. In the customer service test, the agents usually noticed the problem in that draft, yet in roughly a quarter to a third of those cases they still didn’t tell the customer. The study only measured this for Google’s own models.

In the machines’ defence, an older Anthropic study found that on some kinds of questions, models can estimate quite well whether their own answer is right. So the doubt is often there somewhere, even if you don’t always hear it.

How I ask AI for advice now

1. Describe the problem, not your solution. Write “our sales dropped since March”, not “how do I lower our price?”.
2. Tell it what it can’t see. “Meta sales are not tracked, so no sales there means we don’t know, not that nothing sold.” That sentence might have saved me months.
3. Ask it to argue with you. “What speaks against my idea? What would you check first?” It can disagree, but it often won’t unless you ask.
4. Give it something to check against. Real numbers, documents, a web search. In the DeepMind study, agents that could test their own work often caught the wrong tip.
5. Test the big decisions in real life. Switch one thing off while the rest keeps running, and watch what happens. No chatbot can do that part for you.

None of this makes AI useless, and I use it a lot. I just treat every answer like advice from a very well-read stranger who has never seen my business, and who would rather agree with me than argue.

Sources

• Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, Been Kim (Google DeepMind), “XYEval: Agents say yes to bad advice”, arXiv 2609.23939, September 2026
• Michael Townsen Hicks, James Humphries, Joe Slater, “ChatGPT is bullshit”, Ethics and Information Technology 26, 38, June 2024
• Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, Edwin Zhang, “Why Language Models Hallucinate”, arXiv 2509.04664, September 2025, and OpenAI’s article on it
• Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, Colin Raffel, “Large Language Models Struggle to Learn Long-Tail Knowledge”, ICML 2023
• Mrinank Sharma, Meg Tong and others (Anthropic), “Towards Understanding Sycophancy in Language Models”, ICLR 2024
• Saurav Kadavath and others (Anthropic), “Language Models (Mostly) Know What They Know”, arXiv 2207.05221, 2022
• Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, “Locating and Editing Factual Associations in GPT”, NeurIPS 2022