Podium Cut Engineering Fixes 90%: The AI Agent Testing Playbook for Malaysia

AI Summary

Podium, a US software company, used LangSmith — LangChain's platform for tracing, evaluating and improving LLM applications — to bring its "AI employee" agent to 98% F1 response quality while cutting the need for engineering intervention by 90%. The method was unglamorous: curate datasets from real interactions, test the agent continuously across its development lifecycle, and fine-tune based on what the tests expose. The story is less about Podium than about a discipline most Malaysian companies deploying chatbots and agents currently skip: evaluation. Teams that adopt it ship agents that hold up in production; teams that don't end up paying engineers to babysit theirs.

Key Takeaways

• Podium improved its AI agent's F1 response quality to 98% and reduced engineering intervention by 90% using LangSmith for dataset curation, lifecycle testing and fine-tuning.
• The core method is borrowed from software engineering: build a test set of real cases, score every change against it, and only ship what passes. Most agent projects in Malaysia have no equivalent step.
• F1 balances precision and recall — catching cases the agent should handle while not fumbling the ones it does. An agent can look accurate in demos and still fail F1 badly in production.
• "90% less engineering intervention" is the number that matters for CFOs: it is the difference between an agent that runs itself and one that becomes a permanent cost centre.
• Caveat worth noting: this is a case study published by the vendor (LangChain) on its own blog, so read the numbers as best-case, not typical-case.

What Happened

Podium built what it calls an AI employee — an agent designed to perform work a human staff member would otherwise handle, end to end. Like most agents of this type, the hard part was not getting it to work once. The hard part was getting it to work reliably, every time, across the messy range of real conversations.

According to the case study published on LangChain's blog, Podium used LangSmith across the full development lifecycle of the agent. That meant two things in practice: dataset curation — collecting real interaction data and shaping it into structured evaluation sets — and fine-tuning, where the model is further trained on your own examples to sharpen its behaviour on your specific workload.

The results the company reported are specific. Agent F1 response quality reached 98%. And the need for engineering intervention — engineers manually stepping in to fix, patch or handle what the agent got wrong — dropped by 90%.

A note on terms, since they appear throughout this article. F1 score is a standard machine-learning metric that blends precision and recall. Precision means: when the agent responds, how often is it right? Recall means: of all the things it should have handled, how many did it actually handle? F1 forces both to be good. You cannot game it by being right on easy cases while dodging hard ones. That is why 98% F1 is a more demanding claim than "98% accuracy."

Why It Matters

Most AI agent projects do not fail at the demo stage. They fail quietly, weeks after launch, when the agent starts mishandling edge cases and someone in engineering gets assigned to babysit it. That babysitting cost is the hidden line item nobody budgets for. Podium's 90% reduction in engineering intervention is, in my read, the most commercially important number in the story — more so than the 98% F1 — because it converts an unpredictable support burden into a system that mostly maintains itself.

The second reason this matters is the method itself. Software went through this maturation decades ago: unit tests, regression suites, continuous integration. You do not merge code into a bank's core system without automated tests. Yet companies routinely deploy AI agents — which speak on their behalf to customers — with no equivalent testing layer. Podium's playbook treats agent behaviour like code: curate a golden dataset, score every change against it, fine-tune on failures, and re-test before shipping. It is boring, and that is the point. Boring is what production looks like.

Third, the lifecycle framing deserves attention. Testing "across the lifecycle" means evaluation is not a one-time gate before launch. It runs during development, at every model or prompt change, and against production behaviour. This is the shift the industry is working through right now: from prompt-and-pray to measure-and-ship. Teams that make this shift early will compound an advantage, because every conversation their agent handles becomes training signal for the next version. Teams that don't will keep restarting from zero.

What This Means for Malaysia

Malaysian businesses are deploying AI agents fast — mostly WhatsApp chatbots for sales, bookings and customer service, because that is where Malaysian customers actually are. What is missing locally is the evaluation layer. A retail chain in Klang Valley can launch