Token Counts Are the Wrong AI Metric — Start Measuring What Ships

AI Summary

MIT Technology Review argues that token counts — the standard way businesses measure AI usage and cost — have become the wrong yardstick for agentic AI. A coding agent given a single task plans, writes code, runs tests, and repeats the loop to fix errors, consuming 10 to 50 times more tokens than a basic chat prompt. The better and more useful the agent gets, the more calls it makes, so consumption rises as capability rises. The piece's core recommendation: measure AI by delivered outcomes — what actually ships — rather than by token throughput. For Malaysian companies budgeting for AI automation this year, that means rebuilding how you forecast, approve, and evaluate AI spend.

Key Takeaways

• One agentic coding task burns 10 to 50 times the tokens of a single chat prompt, because the agent plans, generates code, runs tests, and loops back to correct its own errors.
• Agent usefulness drives consumption up, not down — a more capable agent makes more calls per task, so costs scale with value delivered, not with queries typed.
• Token counts measure activity, not output. A cheap chatbot that answers questions is not the same unit of work as an agent that ships working software.
• Finance teams budgeting for AI based on chatbot-era assumptions will miss agent-era costs by an order of magnitude — and may wrongly conclude agents "failed" on cost.
• The fix is an outcome-based unit metric: cost per shipped feature, per resolved ticket, per processed document. Tokens become a diagnostic, not the scoreboard.

What Happened

MIT Technology Review published an argument that the AI industry — and by extension, every business buying AI — is measuring the wrong thing. The article describes how agentic AI actually works when a developer hands it a coding task. The agent does not answer once and stop. It plans the work, generates code, runs tests against that code, detects failures, and repeats the cycle until the errors are corrected.

Each step in that cycle is a call to the underlying model. The article puts a number on it: a single agentic task can use 10 to 50 times more tokens than a basic chat prompt. That is the difference between asking an LLM "how do I fix this bug?" and having an autonomous agent actually fix the bug — plan, write, test, fail, rewrite, test again.

The article's second point is the one that catches most buyers off guard. The more useful the agent becomes, the more calls it makes. Capability and consumption climb together. A weak agent gives up early and costs little. A strong agent keeps iterating until the job is done, and the job done is what you actually wanted.

Hence the title's instruction: stop counting tokens, start counting what ships. If the unit of value is a completed, working deliverable, then tokens are just an input cost — like electricity in a factory — and obsessing over them is measuring the wrong layer.

Why It Matters

This matters because measurement drives behaviour. If your board reviews a monthly AI report built around token spend, every incentive pushes teams toward cheap, shallow usage — single-turn chatbots that look affordable and change nothing. If the report is built around cost per completed outcome, teams are incentivised to let agents iterate fully and finish the job, even when the token meter spins.

Think of it in manufacturing terms. No factory manager reports monthly electricity consumption as the headline KPI. You report units produced, defect rates, and cost per unit. Electricity is a line item inside that. The MIT Technology Review argument says AI has now crossed the same line: agents produce finished work, so finished work is the metric.

There is a market signal here too. My read, as analysis: if useful agents consume 10 to 50 times the tokens of a chat prompt, then demand for compute and the semiconductors behind it keeps growing as agents improve — better agents will not shrink AI infrastructure demand, they will expand it. That has implications for the global AI supply chain, data centre buildouts, and pricing models. Expect vendors to eventually sell agents on outcome pricing — per shipped feature, per resolved task — because token-based billing makes their best product look like their most expensive one.

The comparison to watch is cloud computing in the early 2010s. Companies that budgeted for servers struggled with cloud bills; companies that switched to cost-per-workload thinking adopted cloud faster and got more from it. Token anxiety is this decade's version of that same budgeting error.

What This Means for Malaysia

Most Malaysian organisations are still in chatbot-era AI. Generative AI adoption locally has largely meant staff typing prompts into a chat window, with costs that are small and predictable. The moment a Malaysian company — a software house in Penang, a bank in Kuala Lumpur, a manufacturing group in Johor — deploys a genuine agentic workflow, that cost model breaks. The 10-to-50-times multiplier applies immediately, and finance will notice at the next invoice.

This is also a procurement and governance question. Government-linked entities and agencies working under MyDIGITAL and MDEC-led AI adoption agendas should be careful that internal approval frameworks do not judge AI projects on token spend, because that benchmark structurally favours trivial use cases over transformative ones. PDPA matters here too: an agent that plans, acts, and iterates touches more systems and more data than a chatbot does, so autonomy and data governance have to be designed together, not bolted on after.

For Malaysian SMEs in software services, outsourced development, and business process work, the stakes are competitive. If global clients start measuring vendors on what ships — and their own internal agents are shipping faster — Malaysian firms that adopt agentic AI and outcome metrics can defend their margins. Firms that stay on chatbot-era habits will compete on labour cost alone, which is a shrinking advantage.

How Your Business Can Use This

Start with one workflow that has an unambiguous definition of "done". In software, that is a feature that passes tests. Outside software it works the same way: a support ticket closed without reopening, an invoice reconciled and posted, a claim processed end to end.

Then run a four-week pilot with two numbers on the scoreboard: completed work units, and cost per completed unit. Log tokens underneath as diagnostic data — useful for spotting an agent stuck in a wasteful loop — but do not let tokens become the headline. Compare the cost per outcome against the fully loaded labour cost of the same outcome done manually. That comparison, not the raw AI bill, tells you whether the agent earns its place.

Practical steps for this quarter:

1. Pick one repetitive workflow with a clear finish line.
2. Define the work unit precisely (a shipped feature, a closed ticket, a posted document).
3. Let the agent run the full plan-execute-test-correct loop, with caps on spend per task so a runaway loop cannot burn budget.
4. Report cost per outcome weekly. Trend it down.
5. Use the same unit metric when negotiating with AI