Numan's Eval-First AI Health Coach: The Playbook Malaysian Teams Should Copy

AI Summary

LangChain has published a case study on Numan, a digital health platform that built "Nu," an AI health coach, using LangGraph to run the agent and LangSmith to evaluate it. The story here is not the chatbot — it is the method. Numan took an eval-first approach: writing tests for each category of coach behaviour, running simulated conversations before real users ever touch the product, and installing safety gates that stop the agent when a request crosses from coaching into regulated medical territory. For Malaysian businesses building AI agents in banking, insurance, healthcare, or government service, this is the most practical blueprint currently public for shipping generative AI in Malaysia without betting the company on it.

Key Takeaways

• "Eval-first" means writing the test cases and pass criteria before or alongside the agent itself — the same discipline software engineers apply with unit tests, applied to LLM behaviour.
• Per-category evals matter because one AI agent is really several products in one; a single average quality score hides failures in the category that could hurt you.
• Simulated conversations let you test an agent against thousands of scripted user scenarios — including difficult and edge-case users — before a real customer meets it.
• Safety gates are hard checks between the agent's draft reply and the user, enforcing red lines such as "never diagnose or prescribe" — this turns a policy document into running code.
• The case study comes from LangChain, which sells LangGraph and LangSmith, so treat it as a vendor showcase — but the underlying pattern is sound and tool-agnostic.

What Happened

LangChain, the company behind the LangGraph agent framework and the LangSmith evaluation platform, published an engineering case study on Numan's AI health coach, called Nu. Numan operates in health, which means its AI product sits directly on a regulatory line: on one side is lifestyle coaching — encouragement, information, habit support. On the other side is medical practice — diagnosis, treatment, prescription — which is regulated territory where an AI must not wander.

According to the post, Numan built Nu on LangGraph, a framework for constructing agents that run through multiple steps, hold state across a conversation, and decide when to call tools or hand off to a human. LangSmith is the companion platform for tracing what an agent did and scoring how well it did it. Together they let the Numan team see every decision the coach made and grade it.

Three practices stand out in the write-up. First, per-category evals: rather than one overall quality score, each distinct type of coach behaviour gets its own evaluation set and its own pass criteria. Second, simulated conversations: the team tested Nu against scripted, synthetic user conversations before exposing it to real patients. Third, safety gates: hard checkpoints that intercept the agent's output when