How to Evaluate AI Voice Agents Using LangSmith: A Practical Guide

AI Summary

LangChain recently detailed a practical framework for evaluating AI voice agents using LangSmith. Because voice agents operate in real-time and rely on audio, they fail in ways that text chatbots do not. To fix this, developers must test these systems across three distinct layers: execution mechanics, business outcomes, and caller experience. By combining LangSmith traces, code-based evaluators, LLM judges, and targeted human review, technical teams can find out exactly why a voice agent misunderstood a word or dropped a call. For Malaysian businesses building automated customer service lines, this methodology provides a necessary blueprint for deploying reliable voice AI.

Key Takeaways

• Voice AI requires testing across three distinct pillars: execution (did the system run properly), outcomes (did it solve the problem), and experience (was it pleasant to talk to).
• LangSmith traces allow developers to look inside the AI agent to see exactly where a failure occurred, such as in the speech-to-text transcription or the tool-calling logic.
• Code evaluators handle strict, binary rules, while LLM judges can evaluate conversational tone and nuance automatically.
• Human review remains necessary to catch cultural context, local slang, and edge cases that automated evaluators miss.
• Malaysian companies must apply this framework to local accents and languages, such as Manglish and Bahasa Melayu, before trusting an agent with real customer calls.

What Happened

LangChain, the company behind the popular LangChain framework for building large language model (LLM) applications, published a technical guide on how to evaluate voice agents. The guide focuses on their observability platform, LangSmith. As companies move away from text-based chatbots and toward voice-driven AI, the complexity of the technology increases significantly. A text chatbot only processes typed words. A voice agent must transcribe human speech, reason about it, decide on an action, and generate a spoken response in real-time.

The article outlines a structured approach to testing these complex systems. LangChain breaks evaluation down into three specific categories. First, execution: this measures the technical mechanics of the agent, such as how accurately it transcribed the audio, how fast it responded, and whether it successfully called an external database. Second, outcomes: this measures if the agent actually achieved its goal, like successfully booking an appointment or answering a billing question. Third, caller experience: this evaluates the human side of the interaction, judging whether the agent interrupted the caller, used a polite tone, or sounded robotic.

To measure these three categories, the guide highlights four specific tools available in LangSmith. LangSmith traces act as a map, showing developers the exact step-by-step journey of a single call. Code evaluators run automated, strict checks, like verifying that a phone number contains exactly the right amount of digits. LLM judges use a separate, highly capable AI model to grade the conversation based on specific prompts. Finally, human review involves actual people listening to calls to catch the subtle nuances that automated systems cannot understand.

Why It Matters

Text-based AI applications tolerate slight delays. If a chatbot takes three seconds to reply, the user simply waits. Voice agents do not have this luxury. If a voice agent takes three seconds to process a response, the human on the phone assumes the call dropped, gets frustrated, and hangs up. Voice interactions happen in real-time. This makes the underlying technology much harder to build and much more prone to silent failures.

Furthermore, voice agents hallucinate out loud. If a text chatbot makes a mistake, the user can easily scroll up and read the correction. When a voice agent confidently states a wrong bank balance or books a flight on the wrong date, the interaction is fleeting. The user might not catch the error until much later. This creates a high-risk environment for businesses. Deploying a voice agent without a rigorous evaluation framework in place is a liability. It can damage brand reputation and create security vulnerabilities.

The LangSmith framework matters because it brings discipline to a technology that often feels like magic. Voice AI requires moving beyond basic user testing. By establishing clear metrics for execution, outcomes, and experience, engineering teams can systematically improve their agents. Instead of guessing why a caller got angry, developers can look at a LangSmith trace, see that the speech-to-text engine misunderstood a word, and fix that specific parameter. This shifts voice AI from an unpredictable experiment to a manageable enterprise tool.

What This Means for Malaysia

Malaysia has a massive Business Process Outsourcing (BPO) and shared services sector, concentrated heavily in Kuala Lumpur and Penang. Local operations currently handle thousands of hours of voice calls daily for global banks, telecommunications companies, and tech giants. As AI voice agents mature, these Malaysian BPO centres will lead the charge in deploying them to handle first-line support, tier-one troubleshooting, and appointment scheduling. The LangSmith evaluation framework is highly relevant for local technical teams tasked with making these automated agents reliable.

The specific challenge for Malaysia is linguistic diversity. A voice agent deployed locally cannot just understand standard American or British English. It must navigate Bahasa Melayu, Manglish, and various dialects of Chinese and Tamil. It must understand a Malaysian user who code-switches between English and Malay in the same sentence. Local AI builders must use the human review and LLM judge components of LangSmith to specifically train and evaluate their systems on Malaysian voices. If an agent is only evaluated on western accents, it will fail the moment a user in Subang Jaya or Georgetown picks up the phone.

For Malaysian small and medium enterprises (SMEs), this evolution means customer service is changing. A local clinic, legal firm, or logistics company will soon be able to deploy a voice agent to handle after-hours calls. However, under the Personal Data Protection Act (PDPA), these businesses must ensure the AI agent does not expose private information. Using code evaluators to strictly enforce data access rules will be vital for local compliance. Additionally, government agencies pushing the MyDIGITAL agenda can use these evaluation frameworks to build better public service hotlines, ensuring citizens receive accurate information without the frustration of traditional automated phone menus.

How Your Business Can Use This

If your company is planning to build or deploy an AI voice agent, you must demand an evaluation strategy from your technical team or vendor. Do not accept a demo as proof of readiness. A controlled demo rarely reflects the messy reality of actual customer phone calls. Start by defining your critical metrics based on the LangSmith three-pillar approach. For execution, you might require a response latency of under one second. For outcomes, you might measure the percentage of calls where the agent successfully updated a customer relationship management (CRM) record. For experience, you might track how often the agent interrupted the speaker.

To get started, build a small test group. Before letting an AI agent answer your main customer service line, route a small percentage of calls to the AI while human agents listen in. Use LangSmith to generate traces for every single one of these test calls. Set up LLM judges to automatically grade the transcripts for politeness and accuracy. This automated grading saves your human quality assurance team hours of manual work.

When the system fails, use the traces to fix the root cause. If the agent books the wrong appointment, look at the trace. You might find that the background noise on the caller's end confused the speech-to-text tool. You can then adjust the microphone sensitivity settings. If the agent sounds robotic, adjust the text-to-speech parameters. This continuous cycle of testing, tracing, and adjusting is the only way to safely transition from human operators to autonomous voice systems.

The Agentic AI Angle

This development goes directly to the heart of Agentic AI. A standard chatbot follows a strict, pre-written script. An autonomous AI agent plans, reasons, and uses tools to accomplish a goal. When a Malaysian customer calls a logistics company to reschedule a delivery, an agentic voice system does not just offer a menu of options. It accesses the delivery database, checks the user's address, verifies the driver's schedule, and actively executes the rescheduling process via API calls.

LangSmith traces are specifically designed to monitor this autonomous behaviour. Because the agent acts independently, the human operator needs a reliable way to see exactly what the machine decided to do. The