Monitoring AI Agents in Production: What Malaysian Businesses Must Get Right

AI Summary

LangChain, a company whose tools are widely used to build LLM applications, has published guidance on monitoring AI agents in production. Its core argument: conventional monitoring cannot see inside an agent's decisions, so teams need dedicated observability — full traces of every agent run, continuous evaluation of outputs, and a structured loop to improve agents at scale. For Malaysian businesses, the practical read is blunt. An agent that impressed in a demo can still fail quietly in production, and without monitoring you will only find out when a customer complains. This piece explains the framework, why it matters, and what to do about it this quarter.

Key Takeaways

• Conventional monitoring watches infrastructure — uptime, latency, error rates. Agent monitoring watches decisions: every prompt sent, every tool called, every conclusion drawn. LangChain's position is that the second category needs purpose-built observability tooling.
• A trace is the basic unit of debugging. One agent task can involve many chained steps — retrieval, tool calls, reasoning — and you need the full sequence recorded to find where things went wrong.
• Evaluation does not stop at launch. Real user inputs drift, prompts get edited, models get updated. Continuous evaluation catches regressions before your customers do.
• Monitoring is not just insurance; it is the fuel for improvement. Failure traces tell you exactly which step to fix — a prompt, a tool description, or a missing guardrail.
• Analysis: agent traces will almost certainly contain personal data — customer queries, order details, staff instructions — which brings your logs under Malaysia's PDPA, not just your main database.

What Happened

LangChain published a guide on production monitoring for AI agents, aimed at teams running LLM-based systems in live business environments rather than experiments. The guide's premise is that agents are a new category of software with a new category of failure, and that monitoring them requires new observability tools rather than recycled ones from traditional software operations.

The guidance is organised around three pillars: trace, evaluate, and improve. Tracing means recording the complete execution of every agent run — what the agent was asked, what it did at each step, which tools it called, and what it produced. Evaluation means scoring those runs against a definition of "good," on an ongoing basis rather than as a one-off pre-launch test. Improvement means using what the traces and evaluations reveal to fix the agent — refining prompts, adjusting tools, tightening guardrails — and then verifying the fix worked.

The scope is production at scale. This is guidance for systems handling real volumes of real tasks, where a small defect rate multiplied across thousands of runs becomes a serious business problem. It reflects a maturing conversation in the LLM engineering world: the hard part is no longer building an agent that works once. The hard part is keeping it working, week after week, as inputs, models, and business conditions change.

Why It Matters

Agents fail differently from normal software. A crashed server shows an error and stops. An agent can complete its task confidently, produce a polished answer, and be wrong — a wrong refund amount, a wrong delivery date, a wrong clause in a quotation. Nothing crashes. Nothing alerts. This is silent failure, and it is the specific risk that monitoring exists to catch.

The compounding problem makes it worse. An agent chains steps together, so an error made at step two — say, retrieving the wrong customer record — gets built upon at steps three through nine. By the end, the output looks coherent while resting on a bad foundation. Without a full trace, you cannot tell whether the agent reasoned badly or simply started with the wrong input. That distinction determines what you fix.

There is also a market signal here, and this part