To evaluate an AI agent, give it a set of tasks with clear success criteria, run each task several times in a clean environment, and grade what the agent actually did, not only what it said. Anthropic describes agent evals as harder than single-turn evals because agents use tools across many turns, change state as they go, and let mistakes compound.

Terms

Anthropic defines the vocabulary most guides now reuse:

TermMeaning
TaskOne test with defined inputs and success criteria
TrialOne attempt at a task. Several are run because outputs vary
GraderLogic that scores some aspect of the agent’s performance
TranscriptThe full record of a trial: outputs, tool calls, reasoning, intermediate results
OutcomeThe final state in the environment, such as whether a reservation exists in the database
HarnessThe infrastructure that runs the evals, as distinct from the agent’s own scaffold

Anthropic also notes that evaluating an agent means evaluating the model and its harness together.

Steps

  1. Start from real failures. Anthropic says 20 to 50 simple tasks drawn from real failures are a great start, and that evals get harder to build the longer a team waits. More mature agents may need larger, harder sets. LangChain recommends reading 20 to 50 real traces before building any infrastructure.
  2. Do error analysis first. LangChain puts 60 to 80 percent of eval effort here: gather traces, note issues with a domain expert, group them into a failure taxonomy, and repeat until no new categories appear.
  3. Write unambiguous tasks. Anthropic defines a good task as one where two domain experts would reach the same verdict, and suggests a reference solution for each task to prove it can be solved. With frontier models, a 0% pass rate across many trials is, it says, most often a broken task rather than an incapable agent.
  4. Balance the set. Anthropic advises testing both cases where a behavior should occur and cases where it should not, because one-sided evals produce one-sided optimization. LangChain makes the same point about positive and negative cases.
  5. Isolate every trial. Anthropic recommends starting each trial from a clean environment, because shared state such as leftover files can cause correlated failures or inflate scores. LangChain adds that timeouts, malformed API responses and stale caches often look like reasoning failures, so rule out infrastructure before blaming the agent.
  6. Pick graders per task. See the table below.
  7. Read transcripts. Anthropic says it does not take eval scores at face value until someone has read transcripts, since a failed task can mean the agent erred or the grader rejected a valid solution.
  8. Keep the suite alive. Anthropic says owning and iterating on evals should be as routine as maintaining unit tests, and LangChain recommends one domain expert as the quality arbiter for ambiguous cases.

Choosing graders

Anthropic describes three grader types that are usually combined, and recommends deterministic graders where possible, LLM graders where necessary, and humans for validation.

GraderExamplesStrengthsWeaknesses
Code-basedString match, unit tests, static analysis, outcome checks, tool call checksFast, cheap, objective, reproducibleBrittle to valid variations, little nuance
Model-basedRubric scoring, pairwise comparison, reference-based checks, multi-judge consensusFlexible, handles open-ended outputNon-deterministic, costs more, needs calibration against humans
HumanExpert review, spot checks, A/B testsGold standard, calibrates the model gradersExpensive and slow

Grader details that Anthropic lists:

  • Grade what the agent produced, not the path it took, and give partial credit for multi-part tasks.
  • Give an LLM grader an “Unknown” option so it does not guess.
  • Grade each rubric dimension with its own isolated LLM judge instead of one judge for everything.
  • Calibrate LLM graders against human experts, then review by hand only occasionally once the grader is stable.

OpenAI’s trace grading uses structured scores or labels on an agent’s trace to assess correctness, quality and adherence. OpenAI lists questions it answers, such as whether the agent picked the right tool and whether a handoff happened when it should. For how to validate an LLM grader, see What is LLM as a judge?

What to grade

LangChain argues that scoring only the final output can make an agent look healthier than it is, since a correct answer can hide an unstable or risky path. It splits evaluation into three levels.

LevelChecksLangChain’s suggested use
Single step (run)One tool call or LLM decisionTargeted debugging
Full turn (trace)Final response, trajectory, state changesRelease gating and regression testing
Multi-turn (thread)Intent, outcome and trajectory across a conversationConversational agents

State changes deserve their own check. LangChain gives the example of a calendar scheduling agent, where an evaluator should assert that the agent called edit_file on the correct document and that the file actually contains the new preference, in addition to checking that the final message confirmed the update. Anthropic makes the same distinction between a transcript and an outcome, and tau-bench built its scoring on comparing the final database state with an annotated goal state.

For multi-turn agents, LangChain describes N-1 testing: take real conversation prefixes from production and let the agent generate only the final turn, which avoids scripted conversations that break when the agent takes a valid but unplanned path.

By agent type

Anthropic says the same techniques apply across agent types, with different emphasis.

Agent typeWhat to emphasizeBenchmarks Anthropic names
CodingDeterministic tests, plus transcript and code quality rubricsSWE-bench Verified, Terminal-Bench
ConversationalEnd-state checks plus rubrics for completion and interaction quality, often with a simulated usertau-bench, tau2-bench
ResearchGroundedness, coverage and source quality checks, with rubrics calibrated against expertsBrowseComp
Computer useOutcome checks in a real or sandboxed environment, including backend stateWebArena, OSWorld

Handling non-determinism

Agent behavior varies between runs, so Anthropic recommends two metrics. pass@k is the chance of at least one correct solution in k attempts and rises as k grows. pass^k is the chance that all k trials succeed and falls as k grows. Anthropic works through a 75% per-trial success rate over three trials, which gives about 42% for all three passing.

The tau-bench authors introduced pass^k in 2024 and reported that agents which succeeded on under 50% of tasks were also inconsistent, with pass^8 under 25% in the retail domain. Anthropic points to pass^k for agents where consistency matters, and pass@k where one success is enough.

Capability and regression suites

Both Anthropic and LangChain separate two kinds of eval. Capability evals should start at a low pass rate and give teams something to climb. Regression evals should sit near 100% and flag backsliding. LangChain warns that without the split, teams either stop improving or ship regressions. Capability tasks that the agent passes reliably can graduate into the regression suite.

Anthropic also flags saturation: an eval at 100% catches regressions but no longer shows improvement, and SWE-bench Verified was nearing saturation above 80% in its January 2026 post.

Offline and online evaluation

LangChain explains that offline evals run on curated datasets with reference outputs, before every deployment and in CI, while online evals score sampled production traces without reference outputs and catch unexpected inputs and drift.

OfflineOnline
DataCurated datasets with reference outputsSampled production traces
Good forRegression, known edge casesDrift, safety, unexpected behavior
LimitCannot anticipate real user behaviorNo ground truth for correctness

To control LLM judge cost online, LangChain suggests a 10% sampling rate on production traces and triggering LLM checks when deterministic checks show quality drops. Production failures can then seed new offline datasets. LangChain’s own survey found that 89% of organizations have observability, but only 52% run offline evals and 37% run online evals.

Automated evals are one method among several. Anthropic lists production monitoring, user feedback, A/B testing, manual transcript review and systematic human evaluation as the others, and warns that automated evals can create false confidence if they do not match real usage.