Measurement
A token bill says what you spent, not what you got. How to measure agents so finance and engineering agree.
Oct 22, 2026 · 6 min read
A monthly token bill says what you spent, not what you got. Teams that run agents well measure cost per outcome and tie it to a business result chosen before the build.
Attach a business KPI to every candidate agent at intake: cycle time, error rate, cost per transaction or time to resolution. It shapes the evaluation suite, the monitoring and the release decision, and it stops ROI from becoming a story assembled after the money is spent.
The unit that holds up is total agent cost divided by successful tasks. It captures retries, rework, escalations and review time. A model that is cheaper per call can be more expensive per outcome if its work needs cleanup, while a pricier model that finishes in fewer steps can win.
A single-run pass rate hides inconsistency. Splunk's guidance on agent evaluation reports pass^k, success across repeated attempts, next to pass@k. In one cited computer-use study an agent reached about 78% pass@10 but only about 36% pass^10: it looked capable and was still too inconsistent to ship. Measure by task type, since healthcare scheduling needs near-perfect consistency while a content workflow may tolerate retries.
Start with task completion rate, cost per session and one quality metric matched to the main failure mode: groundedness for retrieval-backed agents, tool-call accuracy for agents that take actions. Add per-feature spending ceilings and fallbacks, so a runaway agent shows up before the invoice does.
Value per completed task minus cost per completed task, times the number of completed tasks. For the cleanest answer compare against a holdout group, because people who choose to use an agent often differ from those who do not.
Tell us about one process. We'll show you how an agent could handle it, and what to measure to confirm it's working.