Evaluating AI Agents Before Production: Test Datasets, Tool-Call Tracing, Reliability, and Cost Monitoring

Evaluating AI Agents Before Production: Test Datasets, Tool-Call Tracing, Reliability, and Cost Monitoring

Serious AI agent evaluation before production covers four things: a test dataset built from real usage, graders that check what the agent produced, reliability measured across repeated runs rather than one lucky pass, and cost and latency tracked per task once you ship. Miss any of the four and you’re not shipping an agent, you’re shipping a demo with a billing account attached.

The gap is measurable. In LangChain’s State of Agent Engineering survey (1,340 respondents, published June 2026), 89% of teams had observability in place, but only 52.4% ran offline evals. Most teams can see their agents; far fewer can prove they’re correct. And quality was the top barrier to production, cited by 32% of teams. Repeated evaluations help establish whether the agent is ready for real users.

What AI agent evaluation has to cover before production

An eval is simple to define: give the agent an input, then apply grading logic to its output to measure success. Anthropic’s guide on demystifying evals for AI agents (January 2026) breaks the structure into tasks, trials, transcripts, and graders, and that vocabulary maps onto what you need before launch:

• Test datasets: real tasks with defined success criteria, built from production traces and observed failures.
• Graders: code, model-based, or human logic that scores each trial.
• Reliability measurement: multiple trials per task, because agents are non-deterministic.
• Tracing: a record of every model call and tool call, so failures are debuggable.
• Production monitoring: online evals, guardrails, and cost tracking for whatever offline testing can’t predict.

The discipline is running all of them continuously instead of spot-checking outputs by eye.

Why a single run tells you almost nothing

Agents are probabilistic. The same task, same prompt, same model can succeed on Monday and flail on Tuesday. Testing each task once and moving on measures luck, not capability.

Two metrics separate capability from consistency. pass@k is the probability the agent succeeds at least once in k attempts; it measures capability. pass^k is the probability that all k attempts succeed; it measures consistency, which is what production users actually experience. Anthropic’s example: a 75% per-trial success rate sounds fine, but over three trials the all-succeed probability is 0.75 cubed, roughly 42%. Your “pretty good” agent fails a quarter of the time.

The τ-bench benchmark from Sierra Research makes the same point: GPT-4o, then a frontier model, completed under 50% of retail customer-service tasks, and its pass^8 dropped below 25%. Consistency collapses much faster than capability.

My take: report pass^k alongside pass@k for anything customer-facing. A capability eval with a low pass rate tells you what to build next. A regression suite should sit near 100%, and anything below that is a bug you’re about to ship.

Build the test dataset from real traces

Where do good eval tasks come from? Not from brainstorming. Both Anthropic and LangChain’s agent evaluation readiness checklist (March 2026) converge on the same starting point: manually review 20 to 50 real traces, then start with 20 to 50 simple tasks drawn from observed failures.

That surprises people who assume evals need thousands of cases. Early on, it doesn’t, because the effect sizes are huge. If your agent fails a category of tasks half the time, a handful of tests will catch it. Volume matters only when you’re detecting small improvements. As a rough rule from statistical writeups like Cameron Wolfe’s on LLM eval math: a few dozen cases detect big swings, while reliably detecting a five-point improvement takes hundreds of examples.

The process LangChain recommends, which matches what strong teams do:

1. Pull production traces, mixing a random sample with error cases, user complaints, and outliers.
2. Spend 60 to 80% of your eval effort on error analysis: read traces with a domain expert, open-code the failures, and build a failure taxonomy until no new categories appear.
3. Convert failures into tasks. Every bug a teammate finds becomes an eval case permanently.
4. Give every task a reference solution proving it’s solvable, and include negative cases where the correct behavior is to refuse or do nothing.
5. Split the suite: capability evals probe what the agent can do and start with low pass rates; regression evals protect existing behavior and should be near-perfect.

LangChain adds a detail worth stealing for multi-turn agents: N-1 testing. Take a real conversation, keep all but the last turn, and evaluate only the agent’s final response. You get multi-turn coverage without simulating entire conversations. And prune evals that no longer reflect production behavior; they waste CI minutes and hide regressions.

Grade outcomes, not paths

Here’s the grading mistake I see most often: writing assertions on the exact sequence of tool calls the agent “should” have made. Anthropic’s guidance is blunt about this; check exact sequences and you build brittle tests that fail valid, unanticipated approaches. Their line: grade what the agent produced, not the path it took. If the task was “book the refund,” check whether the refund exists in the database.

Tool-call checks still belong in your suite, just aimed at properties rather than choreography: did the agent call a destructive API without required confirmation, or hallucinate a tool that doesn’t exist? Those are worth failing a build over.

Three grader types, in order of reach:

• Code-based graders: string checks, unit tests, database-state verification, static analysis. Fast, cheap, deterministic.
• LLM-as-judge: a model scores outputs against a rubric for subjective dimensions like tone. Flexible, but noisy.
• Human review: the calibration standard. Expensive, so use it to validate the other two.

LLM-as-judge deserves its own caution. The research literature (see the large survey by Gu et al.) documents recurring failure modes: position bias in pairwise comparisons, a preference for longer answers, self-preference for a judge model’s own outputs, and plain inconsistency across runs. None of that makes judges useless. It makes them uncalibrated instruments until proven otherwise.

To calibrate one: start with 20+ human-labeled examples, measure agreement, and grow the set to roughly 100. LangChain’s checklist also suggests binary pass/fail verdicts over 1-to-5 scores, an explicit “Unknown” option so the judge can abstain rather than hallucinate a verdict, one isolated judge per rubric dimension, and periodic recalibration, because a model upgrade can silently shift judge behavior.

Tool-call tracing: the receipt for every run

Tool-call tracing is the practice of recording each agent run as a structured trace: one span for every model call, tool invocation, retrieval step, and handoff, with inputs, outputs, token counts, errors, and timing attached. A trace is the full end-to-end run; a span is one operation inside it. Without this, a failed run is a shrug. With it, you can see the retrieval step returned garbage, the model called the wrong API, and the retry burned $0.04 pretending otherwise. Tracing is also where future eval datasets come from.

The plumbing is standardizing. The OpenTelemetry GenAI semantic conventions define spans and attributes for GenAI operations (agent spans, tool operations, MCP) plus provider-specific conventions for OpenAI, Anthropic, and AWS Bedrock. These conventions are still in development, so attribute names may change before they stabilize. Instrument through a library (Phoenix’s OpenInference, Langfuse, LangSmith, Weave, or plain OTel SDKs) so you’re not hand-maintaining attribute names.

Two tracing habits that pay off in evals:

• Isolate every trial. Run each attempt in a clean environment: fresh containers, sandboxed APIs, reset database state. Anthropic recounts their own agent gaining an unfair advantage by reading a previous trial’s git history. Contaminated state corrupts scores quietly.
• Distrust perfect zeros. A task with 0% pass@100 usually signals a broken task or broken grading, not an incapable agent. Anthropic cites CORE-Bench, where a model’s score jumped from 42% to 95% after grading bugs were fixed. When scores move weirdly, audit the grader before the agent.

Which tools for AI agent evaluation in 2026?

The tooling map changed more this year than in the previous two combined, so older guides are badly stale. The headlines: OpenAI is deprecating its Evals platform (read-only for existing users October 31, 2026, shutdown November 30), and it published a migration guide to Promptfoo, the open-source eval and red-teaming CLI OpenAI acquired earlier in 2026. ClickHouse acquired Langfuse in January 2026; the MIT-licensed core remains open source and self-hostable.

The main differences are hosting, trace allowances, and how usage is billed:

How to choose: if you’re in the LangChain ecosystem, LangSmith is the path of least resistance. If open source and data ownership are non-negotiable, Langfuse self-hosted or Phoenix is the pragmatic pick. If you want evals and production analysis in one commercial package with unlimited seats, that’s Braintrust’s model. And almost everyone should have Promptfoo or equivalent in CI regardless, because a local, config-driven eval runner is the cheapest regression net you can build.

What public benchmarks can and can’t tell you

Public benchmarks are useful for picking a base model and nearly useless for proving your agent is ready. The events of 2026 made that case better than any argument.

On October 23, 2026, OpenAI announced it stopped evaluating SWE-bench Verified, the coding benchmark it curated in 2024. Two reasons: contamination (frontier models could reproduce gold patches and verbatim task details, suggesting training-data exposure) and broken tests. In an audit of 138 problems their models failed inconsistently, OpenAI engineers found 59.4% had material issues in the tests or task descriptions.

Of the 138 problems OpenAI audited, 59.4% had material issues in the tests or the task descriptions. — OpenAI, “Why we no longer evaluate SWE-bench Verified,” October 23, 2026

GAIA, the general-assistant benchmark from 2023, tells the second story: saturation. Leading systems now exceed the human baseline the authors published, so it no longer separates good agents from great ones. Meta’s response was Gaia2 and the ARE framework (September 2025), which evaluates agents in asynchronous environments where events happen on a clock whether the agent is ready or not.

The practical takeaway: cherry-pick a handful of benchmark tasks (τ²-bench for tool-plus-policy behavior, Terminal-Bench for coding, GDPval for real professional work) when they match your domain, and treat the leaderboard as background noise. Your production traces are the benchmark that matters.

The workflow, end to end

Pulling this together, here’s the sequence:

1. Instrument everything. Structured tracing first: one span per model call and tool call, with arguments, results, token counts, and latency.
2. Read 20 to 50 real traces. With a domain expert, categorize what went wrong and build the failure taxonomy until it stops growing. Budget most of your eval time here.
3. Write the first dataset. 20 to 50 tasks from real failures, each with inputs, success criteria, and a reference solution. Include negative cases.
4. Define graders per task. Code-based checks on verifiable outcomes first. LLM judges for subjective dimensions, calibrated against 20 to 100 human-labeled examples. Human review for the ambiguous residue.
5. Run k trials per task in clean environments. Fresh containers, sandboxed APIs, reset state. Report pass@1 and pass^k, plus steps, latency, and tokens per task.
6. Wire the regression suite into CI. Cheap code graders on every commit; LLM-judged evals on preview deployments. Start in reporting-only mode, promote to a hard gate once scores are stable.
7. Ship with monitoring, not dashboards. Online evals score sampled production traffic asynchronously; guardrails block bad responses inline. Capture user feedback as structured data.
8. Close the loop. Every production failure becomes a new eval case, and every saturated capability eval graduates into the regression suite. This flywheel is the whole practice.

Watching cost and latency after launch

Agent economics have failure modes that per-request dashboards hide.

The core move is attribution: tie spend to the trace, span, agent, and step, the way Braintrust’s cost-observability guide (May 2026) describes; platforms like Datadog’s LLM Observability estimate per-request cost from token counts and provider pricing. The metric that matters most is cost per completed task, not cost per request. An agent that needs five retries and two fallbacks to finish a booking is a different product from one that gets it in three calls, even if a success-rate chart shows both as identical.

Retries are the silent line item. Every failed tool call, every context-window overflow, every judge invocation in your guardrails is spend, and none of it appears in a clean demo. Tag retry spend separately from productive spend. Watch latency at the percentile level (p50 and p95): an agent that’s 95% accurate but much slower may be a net downgrade.

The stakes aren’t hypothetical. Gartner predicted in June 2025 that over 40% of agentic AI projects would be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. A 2025 MIT NANDA report covered by Fortune found 95% of enterprise GenAI pilots failed to show measurable returns. Evaluation discipline is how you avoid joining either statistic.

One distinction to keep crisp: evaluators run asynchronously and measure quality; guardrails run inline in milliseconds and block bad outputs before users see them. Offline evals catch regressions before deploy; online ones catch the distribution shift no pre-launch dataset predicts.

FAQ

How do I test an AI agent before production? Build a dataset of 20 to 50 tasks from real traces, grade outcomes with code-based checkers plus calibrated LLM judges, run multiple trials per task and report pass@k and pass^k, then wire the regression suite into CI. After launch, keep scoring production traffic with online evals and guardrails.

What is LLM-as-judge, and is it reliable? It’s the practice of using a model to score another model’s outputs against a rubric. It’s reliable enough for subjective dimensions only after calibration: measure agreement against 20 to 100 human-labeled examples, use binary verdicts, give the judge an “Unknown” option, and watch for documented biases like verbosity and position preferences.

How big should my agent eval dataset be? Start with 20 to 50 tasks you’re confident in; that’s enough to catch large failures, and quality beats quantity. Detecting small improvements of a few percentage points reliably takes hundreds of examples, so grow the suite as the agent improves.

What is tool-call tracing? Recording every agent run as a structured trace of spans: one span per model call, tool invocation, and retrieval step, with inputs, outputs, errors, timing, and token counts. It’s how you debug failures, verify tool behavior, attribute cost, and harvest real examples for your dataset. OpenTelemetry’s GenAI conventions define the standard span shapes.

Which tools should I use for agent evals in 2026? LangSmith if you’re in the LangChain stack, Braintrust for a commercial package with unlimited seats, Langfuse (MIT core, now owned by ClickHouse) or Arize Phoenix for open, self-hostable tracing and evals, and Promptfoo for a free CI-native eval runner. OpenAI’s Evals platform shuts down November 30, 2026.

How do I monitor AI agent costs in production? Track token usage and estimated cost per span, attribute spend to agent, task, and workflow, and compute cost per completed task rather than per request. Tag retry and fallback spend separately, since retries are the expense that per-request dashboards hide. LangSmith, Braintrust, Langfuse, and Datadog-style LLM observability all cover the plumbing.

Sources

• Anthropic: Demystifying evals for AI agents (January 9, 2026)
• OpenAI: Why we no longer evaluate SWE-bench Verified (October 23, 2026)
• OpenAI: Working with evals (deprecation notice)
• OpenAI Cookbook: Moving from OpenAI Evals to Promptfoo (June 3, 2026)
• LangChain: Agent Evaluation Readiness Checklist (March 27, 2026)
• LangChain: State of Agent Engineering survey (published June 12, 2026)
• LangChain: LangSmith pricing
• Langfuse: Pricing
• ClickHouse: ClickHouse welcomes Langfuse (January 16, 2026)
• Braintrust: Pricing
• Braintrust: How AI observability helps lower LLM cost at scale (May 6, 2026)
• Arize Phoenix on GitHub
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv, June 2024)
• Meta AI: ARE: Scaling Up Agent Environments and Evaluations (Gaia2) (September 22, 2025)
• OpenTelemetry: GenAI semantic conventions repository
• Gu et al.: A Survey on LLM-as-a-Judge (arXiv, November 2024)
• Gartner: Over 40% of agentic AI projects will be canceled by end of 2027 (June 25, 2025)
• Fortune: MIT report on generative AI pilots (August 18, 2025)
• Datadog: LLM Observability cost tracking