How Similarweb Evaluates Agent Reports with LangSmith

AI Summary

Similarweb, the digital intelligence company, has shared how it uses LangSmith — LangChain's evaluation and observability platform — to assess the quality of long-form research reports generated by AI agents. Their approach combines structured rubrics, faithfulness checks, execution traces, and baseline comparisons to systematically measure whether agent outputs are accurate, grounded in real data, and improving over time. For Malaysian businesses building or buying AI agent systems, this case study offers a practical blueprint for solving one of the hardest problems in agentic AI: knowing whether you can trust the output.

---

Key Takeaways

• Agent evaluation is not optional. Long-form AI research reports are prone to hallucination, omission, and inconsistency. Without a structured evaluation pipeline, you cannot distinguish a useful agent from an expensive liability.
• Rubrics provide measurable quality standards. Similarweb scores reports against defined criteria rather than relying on gut feel, enabling consistent, scalable assessment.
• Faithfulness checks verify that agent claims are grounded in source data. This catches fabricated facts before they reach a decision-maker's desk.
• Execution traces reveal how the agent reasoned. LangSmith's tracing lets engineers see every step, tool call, and intermediate decision — essential for debugging complex agent workflows.
• Baseline comparisons turn evaluation into a continuous improvement loop. By comparing new agent versions against previous benchmarks, teams can prove whether a change actually made things better.

---

What Happened

Similarweb, a company known for providing competitive digital intelligence — web traffic data, market analysis, and consumer insights — has been building AI agents that generate long-form research reports. These reports are not simple chatbot responses. They are multi-page analytical documents that synthesise data from multiple sources, draw conclusions, and present findings in a structured format.

The challenge Similarweb faced is one that any organisation deploying agentic AI will recognise: how do you know the report is good? A human reading a ten-page AI-generated market analysis might spot obvious errors, but subtle hallucinations, missing context, or logical inconsistencies can easily slip through. When the report influences business strategy, the cost of a bad output is high.

To solve this, Similarweb turned to LangSmith, the evaluation and observability platform built by LangChain. LangSmith is designed to help developers test, monitor, and improve LLM (Large Language Model) applications. It provides tools for tracing agent execution, running automated evaluations, and comparing performance across versions.

According to the LangChain blog post, Similarweb's evaluation methodology rests on four pillars:

1. Rubrics. Similarweb defines explicit scoring criteria for what makes a good report — completeness, accuracy, structure, relevance, and similar dimensions. Each agent-generated report is scored against these rubrics, producing a quantifiable quality metric rather than a subjective impression.

2. Faithfulness checks. This is a specific evaluation technique that verifies whether claims in the report are actually supported by the source data the agent accessed. If the agent writes "Company X saw a 30% traffic increase in Q3," the faithfulness check examines whether the underlying data actually supports that statement. This is critical because LLMs can generate confident-sounding statements that have no factual basis — the well-documented hallucination problem.

3. Traces. LangSmith records the full execution path of the agent — every prompt, every tool call, every intermediate reasoning step. This trace data lets engineers understand not just what the agent produced, but how it arrived at that output. If a report contains an error, the trace shows where the agent went wrong.

4. Baseline comparisons. Similarweb compares new agent versions or prompt changes against previous performance benchmarks. This means every modification — a new data source, a rewritten prompt, a different model — can be measured against a known baseline to confirm whether it actually improved output quality.

---

Why It Matters

This case study matters because it addresses the single biggest gap in enterprise AI adoption today: trust at scale.

Many organisations have experimented with AI chatbots or copilots. Far fewer have deployed autonomous agents that produce work product — reports, analyses, recommendations — that feeds directly into business decisions. The reason is not technology. The technology exists. The reason is confidence. A CEO will not act on a market entry strategy generated by an AI agent unless she trusts that the analysis is grounded, complete, and free of fabrication.

What Similarweb and LangSmith demonstrate is that trust is not a feeling. It is a system — a repeatable, measurable process of evaluation that produces evidence of quality. This shifts the conversation from "is AI reliable?" (unanswerable in general) to "is this specific agent reliable for this specific task, and here is the data proving it?"

The four-pillar approach — rubrics, faithfulness, traces, baselines — is becoming the de facto standard for serious AI evaluation. Companies that adopt this disciplined approach will deploy agents faster and with less risk. Companies that skip evaluation will either deploy unreliable agents (and pay the consequences) or stall in pilot purgatory, unable to prove value.

This also signals a maturation of the AI tooling ecosystem. The first wave of AI tools helped you build agents. The current wave — LangSmith, and similar platforms — helps you govern them. Evaluation infrastructure is now as important as the agent itself.

---

What This Means for Malaysia

For Malaysian businesses, the implications are direct and immediate.

For SMEs and mid-market companies, the message is that you should not deploy an AI agent for research, reporting, or customer-facing tasks without an evaluation process. If you are a logistics company in Port Klang using an agent to generate daily route optimisation reports, or a fintech in Kuala Lumpur using an agent to compile regulatory compliance summaries, you need to know the output is reliable. The Similarweb approach — even implemented in a lightweight way — gives you a framework.

For Malaysian enterprises and GLCs, this is relevant to AI governance frameworks being developed under the National AI Roadmap and MyDIGITAL initiatives. As regulators and boards demand accountability for AI-driven decisions, having a documented evaluation pipeline — with rubrics, faithfulness scores, and trace logs — becomes evidence of due diligence. Under the PDPA and emerging AI governance guidelines, organisations may need to demonstrate that they have taken reasonable steps to ensure AI outputs are accurate and safe.

For the local AI builder ecosystem — startups in MRANTI, MDEC's Malaysia Digital initiative, or the Penang tech corridor — LangSmith represents an accessible, practical tool for building trustworthy agents. Malaysian AI companies pitching to enterprise clients can differentiate themselves by showing not just their agent's capabilities, but their evaluation methodology. In a market where many vendors still sell AI on hype, a vendor who can show faithfulness scores and trace logs will win the serious clients.

For government agencies exploring AI for policy research, public service delivery, or data analysis, the Similarweb case demonstrates that agent-generated reports must pass quality gates before they inform decisions. A ministry using an AI agent to draft industry briefing papers needs the same safeguards Similarweb applies.

---

How Your Business Can Use This

Here is a practical approach to applying Similarweb's evaluation framework in a Malaysian business context:

Step 1: Define your rubric. Start by writing down what a good output looks like for your specific use case. If your agent generates market research reports, your rubric might include: factual accuracy, source attribution, completeness of coverage, logical structure, and actionability of insights. Score each on a simple 1–5 scale. You do not need sophisticated tooling to start — a spreadsheet works.

Step 2: Implement faithfulness checks manually first, then automate. Before investing in LangSmith or similar platforms, have a human reviewer verify that key claims in agent outputs are supported by source data. Once you understand the failure patterns, automate this check using an LLM-as-judge approach — a second model evaluates whether the first model's claims are grounded.

Step 3: Start tracing. If you are building agents using LangChain or a similar framework, enable execution tracing from day one. The trace data is invaluable for debugging and will form the backbone of your evaluation pipeline.

Step 4: Establish a baseline before making changes. Run your current agent against a fixed set of test inputs and record the scores. Any future change — new prompt, new model, new data source — is measured against this baseline. If scores drop, you revert.

Step 5: Choose your tooling. LangSmith is well-suited for teams already in the LangChain ecosystem. Alternatives exist, but the key capability you need is trace-level observability combined with automated evaluation scoring.

---

The Agentic AI Angle

This story is fundamentally about agentic AI — autonomous systems that plan, reason, and execute multi-step tasks. Unlike a chatbot that answers a single question, an agent producing a long-form research report might execute dozens of steps: searching databases, retrieving documents, extracting data points, synthesising findings, drafting sections, revising based on internal logic.

Each of those steps is a potential failure point. An agent might retrieve the wrong document, misinterpret a data point, or skip a critical source entirely. LangSmith's tracing capability is what makes agentic AI viable in production because it exposes the full chain of reasoning, not just the final output.

For Malaysian businesses, this means agentic AI is not a future concept — it is deployable today, but only with evaluation infrastructure. A practical example: a Malaysian property developer could deploy an agent that autonomously researches land valuation data, zoning regulations, comparable transactions, and market trends, then produces a site feasibility report. Using the Similarweb approach, every claim in that report would be faithfulness-checked against source data, the agent's reasoning would be traced for audit purposes, and each report version would be scored against a rubric before it reaches the investment committee.

This is the difference between AI as a novelty and AI as infrastructure.

---

Risks and Limitations

Several caveats are worth noting. First, evaluation frameworks reduce but do not eliminate risk. A faithfulness check verifies that a claim is supported by retrieved sources — but if the agent retrieved the wrong sources in the first place, the check passes on inaccurate data. Evaluation is only as good as the data pipeline feeding the agent.

Second, rubric-based evaluation involves subjective judgement in designing the rubric itself. Two organisations might define "accuracy" or "completeness" differently, leading to different quality assessments for the same output.

Third, LangSmith and similar platforms require technical expertise to implement effectively. SMEs without in-house AI engineering talent may need to partner with a specialist or consultant, adding cost and complexity.

Finally, LLM-as-judge approaches (using one model to evaluate another) have their own reliability issues. Judges can be biased, inconsistent, or influenced by surface features like writing style rather than substantive accuracy.

---

The Bottom Line

Similarweb's use of LangSmith to evaluate agent-generated reports is not just a technical case study — it is a template for responsible AI deployment. The four-pillar framework of rubrics, faithfulness checks, traces, and baseline comparisons answers the question that every Malaysian business leader should be asking before deploying agentic AI: "How do I know this is working?"

The action for this quarter: if your organisation is using or planning to use AI agents for any report generation, research synthesis, or analytical task, build an evaluation pipeline before you scale. Start simple — a rubric in a spreadsheet, a human reviewer checking factual claims, a log of agent outputs. Then graduate to platform-based evaluation as your use cases mature. The organisations that build evaluation discipline now will deploy AI faster, safer, and with far greater confidence than those who skip this step.

---

FAQ

What is LangSmith and do I need it for my Malaysian SME? LangSmith is an evaluation and observability platform for AI applications built on LangChain. If your SME is building custom AI agents or LLM applications, it helps you test and monitor quality. If you are only using off-the-shelf tools like ChatGPT, you do not need it yet.

How is faithfulness checking different from regular proofreading? Faithfulness checking specifically verifies that every factual claim in an AI output is supported by retrieved source data. It catches AI hallucinations — confident statements that sound plausible but are fabricated. Regular proofreading checks language quality but may not catch factual errors.

Can Malaysian companies access LangSmith, and what does it cost? LangSmith is available internationally, including from Malaysia. Pricing varies based on usage tiers. Malaysian AI builders should evaluate it alongside alternatives based on their specific tech stack and budget — the principles of evaluation matter more than the specific tool.

---

Sources / References

• LangChain Blog: "How Similarweb Evaluates Long-Form Agent Research Reports with LangSmith" — Primary source for all facts about Similarweb's evaluation methodology, including rubrics, faithfulness checks, traces, and baseline comparisons. Available at: https://www.langchain.com/blog/how-similarweb-evaluates-long-form-agent-research-reports-with-langsmith