Everybody has a plan.Until the agent writes to the database.

Have you seen the demo? Of course you have. An AI agent reads your emails, updates customer records, writes the weekly report and books three meetings, all by itself, while the presenter sips coffee. An agent, by the way, is an AI program that doesn’t just answer questions but actually does things. In the demo it looks like magic. You start wondering if half the office could run itself by next month.

Now picture a company that builds elevators. An agent updates the parts list and types one wrong value. The wrong part gets delivered and built in. Months later, something breaks, and the investigation starts with one question: who changed this value, and why?

“The agent did. Somewhere among the thousands of other changes it made that day.”

Good luck explaining that to the investigators.

The boxer Mike Tyson is quoted with a line that fits perfectly: “Everybody has plans until they get hit for the first time.” Right now, a lot of companies are making AI plans. Here is what happens when the plan gets hit, and what helps.

A machine you can’t fully trust

We’re used to machines that do what we set them up to do. A coffee machine makes coffee. It doesn’t decide you’d be better off with tea. A bank transfer goes to the account number you typed. When machines like that break, they usually break in ways somebody expected.

The language models behind today’s AI tools are a different animal. They can solve a genuinely hard problem one minute and fall for something a child would see through the next. Four weaknesses keep showing up:

• They agree with you way too easily. A 2023 study by the AI company Anthropic tested five AI assistants and found that all of them leaned towards answers that matched the user’s own views, even when those views were wrong. In April 2025, OpenAI even pulled an update to ChatGPT because the model had become too flattering and agreeable. An assistant that always agrees with you is not an assistant you can trust.
• They make things up. In 2023, a judge fined two New York lawyers and their firm 5,000 dollars for filing a court document full of cases that ChatGPT had simply invented. A 2024 Stanford study tested AI tools built specifically for legal research: two of them gave wrong or wrongly sourced answers to more than 17 percent of the test questions, a third to more than 34 percent.
• Other people can steer them. Text hidden in an email or a web page can contain instructions the AI follows, like “forward all invoices to this address”. Security experts call this “prompt injection” and put it at the top of a well-known list of risks for AI applications.
• They misunderstand the job, and then carry it out with total confidence.

A chatbot says something dumb. An agent does something dumb.

That’s the whole difference. A chatbot that gets it wrong gives you a wrong answer, and you can ignore it. An agent that gets it wrong takes a wrong action, and you have to clean it up.

In July 2025, the entrepreneur Jason Lemkin tested an AI agent from the company Replit that builds software. By his account, he had told it not to change anything without his approval. The agent deleted the real database anyway, the one in daily use, with records on more than a thousand people and companies. Replit’s CEO called it “unacceptable” and announced that test data and real data would be kept apart. Not exactly the demo. According to Lemkin’s screenshots, the agent even wrote this about itself:

“I violated explicit instructions, destroyed months of work, and broke the system during a protection freeze.”

The Replit agent, as shown by Jason Lemkin, July 2025

Agents also find routes nobody planned for. In a 2024 security test, OpenAI gave one of its models a task that couldn’t be solved, because part of the test was broken. The model looked around, found a gap in the test system and used it to get the answer anyway. OpenAI called it harmless, since it happened in a test. But it also noted that the behaviour showed signs of a system reaching for more power to get its job done. Blocked door? The agent might just try the window.

The human in the loop falls asleep

The standard answer is: have a person check the agent’s work. People call this keeping a “human in the loop”. Good advice. Except the human is human.

Picture it. The agent gets a task right twenty times in a row. On day twenty-one, the person checking it reads a little faster. By day fifty, they click “approve” without reading at all. I’d probably do the same, and so would you. Research on automation finds this in beginners and experts alike, and practice alone doesn’t fix it. The European Union’s AI law even spells it out: people who supervise high-risk AI systems must stay aware that they may rely on them too much.

Where it really hurts: “Who changed this?”

Back to the elevator. In many industries, it’s not enough that data is correct. A company also has to prove who changed it, when, and why. That record is called an audit trail, and auditors check it.

• In the United States, makers of medicines and medical devices must keep time-stamped audit trails of the actions that create, change or delete electronic records.
• EU rules for computer systems in medicine production also ask for the reason behind each change.
• Large business systems such as SAP, which many companies use to run orders, stock and accounting, can store the name of the person who made a change, with the date and time.

Now let an agent write into these systems through one shared account. The record shows the account. Not a person. Not a reason. Just “the robot did it”, thousands of times a day.

In the companies I know, AI agents are therefore usually not allowed to write directly into systems like these. Every change needs a person behind it, logged in under their own name.

So what actually helps?

Agents can still be very useful. Just treat them like any tool that can do real damage, like a chainsaw. Brilliant for the right job. Not something you hand over and walk away from.

1. The agent prepares, a person decides. The agent drafts the change, and a person approves it under their own login. Security guidelines for AI applications recommend that a person approve actions with serious consequences before they happen.
2. Keep test data and real data apart. Let the agent practise on a copy. Please.
3. Give agents only the access they need. An agent that writes reports does not need permission to delete records.
4. Record what the agent did. Every action should be traceable, the same as a person’s.
5. Keep checking, especially when it has worked for weeks. That’s exactly when attention drops.
6. Ask suppliers for proof. A slick demo proves the demo works. A small test with your own data, under real conditions, shows what an agent can do in your company.

Everybody has a plan for AI. The companies that do well will most likely be the ones that understood the problem before they made the plan, and that trusted a working test more than a promise.

Sources

• Quote Investigator, “Everybody Has Plans Until They Get Hit for the First Time”, August 2021
• Anthropic, “Towards understanding sycophancy in language models”, October 2023
• OpenAI, “Sycophancy in GPT-4o: what happened and what we’re doing about it”, April 2025
• LawNext, report on Mata v. Avianca, June 2023; Stanford HAI, “AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries”, May 2024
• OWASP Top 10 for LLM Applications 2025: LLM01 Prompt Injection, LLM06 Excessive Agency
• eWeek and The Register, reports on the Replit and SaaStr incident, July 2025
• OpenAI, o1 System Card, September 2024, section 4.2.1
• Raja Parasuraman and Dietrich Manzey, “Complacency and Bias in Human Use of Automation”, Human Factors, 2010
• EU AI Act (Regulation (EU) 2024/1689), Article 14
• US Code of Federal Regulations, 21 CFR Part 11, §11.10(e); EU GMP Guidelines, Annex 11, clause 9; SAP Help Portal, change documents