Your AI agent vendor delivered something that doesn't work — here's how to tell if it's salvageable

A KL logistics company paid a vendor a five-figure sum for an AI agent to handle order enquiries. In the demo it was flawless. Three months into production it was quietly making things up — quoting delivery dates it had no way to know, occasionally emailing customers the wrong address, and failing on anything that wasn't a textbook order. The vendor had gone quiet. The operations lead was stuck with something that looked like an AI agent but couldn't be trusted with a real customer.

If some version of that is happening to you, the first question isn't "who do I blame" — it's "can this be fixed, or do I have to start again?" This is a field guide to answering it, and to knowing when a fix to your AI agent is worth it and when it isn't.

Why agents fail after the demo

The uncomfortable truth is that a demo proves almost nothing. Demos run on happy-path inputs the builder chose. Production runs on whatever your customers actually type, at 2am, in three languages, with edge cases nobody scripted. The gap between those two is where most agent projects quietly die. The failure modes are remarkably consistent:

• Confidently wrong. The agent answers everything — including things it has no basis to answer. Without guardrails and grounding, a language model will invent a plausible delivery date or policy rather than say "I don't know."
• Never evaluated. There's no test suite, no golden set of real cases, no measure of how often it's right. Nobody can actually tell you its accuracy, because nobody measured it.
• Brittle integration. It half-connects to your systems. It reads, but writes fail silently; a retry creates a duplicate order; an API change breaks it and no one notices for a week.
• No documentation, no owner. The prompts, the logic and the "why" live in one freelancer's head. When they move on, so does any hope of maintaining it.
• Runaway cost. It's wired to the most expensive model for every task, or it loops, and the token bill balloons for work a cheaper setup would do fine.

None of these are exotic. They're the predictable result of building an agent as a one-off deliverable instead of a governed system — which is exactly why Anthropic's own guidance stresses starting simple, adding tools deliberately, and measuring (Anthropic — Building effective agents). Most failed agents skipped the measuring.

The rectify-or-rebuild decision

Here's the framework we use when someone hands us a broken agent. It comes down to one question: is the foundation sound, or is the foundation the problem?

Lean toward rectify when:

• The core approach is reasonable — the right task, a sane model choice, real access to your systems — but it's missing guardrails, evals, or error handling.
• The code and prompts are legible enough to understand and extend.
• The failures are specific and reproducible (it breaks on X), not diffuse ("it's just unreliable").

Lean toward rebuild when:

• There's no separation between "read" and "write" actions, no logging, and no way to reconstruct what it did — you can't safely change what you can't see.
• The whole thing is a thin wrapper around a single mega-prompt with no structure, so every fix breaks something else.
• It's fundamentally aimed at the wrong problem, or the "agent" is really a chatbot that was never going to complete the task.

The honest answer is sometimes that a rebuild is cheaper than a rescue — patching a broken foundation costs more over a year than replacing it. A good partner will tell you that up front rather than bill you to prop up something that won't hold. That honesty is the whole point of a proper audit.

What a real quality bar looks like

Whether you rectify or rebuild, the destination is the same: an agent that would pass the bar it should have been held to in the first place. Concretely:

• It's measured. Scored against a golden set of your real cases — we use the Agent GPA framework (Goal, Plan, Action) — so its accuracy is a number, not a vibe. It doesn't go live until it beats your manual baseline.
• A human stays in the loop on anything that matters. The agent evidences and drafts; a person approves the decisions with consequences.
• It's grounded and guarded. It works from your real data, resists prompt injection, and is honest when it doesn't know — rather than inventing an answer.
• It's observable and safe. Every action is logged and reversible-by-design; a retry can't create a duplicate; failures route to a human, not into a void.
• Data is handled properly — PDPA/PDPC-aligned, least-privilege access to your systems.

That's not gold-plating. It's the difference between a demo and something you can put in front of a customer. We wrote up how we build to that standard, use case by use case, in our AI agent use-case directory.

What it costs to find out

The good news: finding out where you stand is cheap. A proper AI agent audit — someone experienced going through the agent, its integration, and its (missing) evals and giving you a plain rectify-or-rebuild verdict — is the highest-leverage RM you can spend, because it stops you pouring money into either a doomed patch job or an unnecessary rebuild.

We offer that audit free, precisely because the verdict should be honest and not shaped by a quote. If a rescue makes sense, our handover runs RM 2,000–5,000 in Malaysia (S$1,000–3,000 in Singapore) to take ownership and stabilise the agent — priced to how much is broken — with ongoing maintenance scoped to its real load. If a rebuild is the right call, we'll tell you that instead — and you can see what building an AI agent properly involves.

The one thing not to do is keep paying to keep a broken agent limping along in front of real customers. Every confidently-wrong answer it gives is a customer you're quietly losing.

Frequently asked questions

Sources

• Anthropic — Building effective agents: why agents should start simple, add tools deliberately, and be measured.

This article is general guidance, not a warranty — get a proper assessment of your specific agent before deciding to rectify or rebuild.