AI agent evaluation: what to test before an agent ships

AI agent evaluation tests whether an agent finishes the job by a sound path, before it ships and after. Score the outcome, the trajectory and the failing component as separate results, because one blended number hides where the run broke. Use code checks where the right answer is exact. Use an LLM judge where the answer needs judgment, and check that judge against human labels before you trust it.

Most teams stop at an average score. I think the work that matters is deciding which failures block a release and keeping the grader honest.

What AI agent evaluation measures

AI agent evaluation measures a run at more than one depth. Kritin Vongthongsri at Confident AI names three depths in LLM Agent Evaluation Metrics in 2026: end-to-end, trajectory-level and component-level. He recommends reading them as a diagnostic stack, starting from the outcome and working down. End-to-end evaluation asks whether the user's goal was met. Trajectory review covers the plan, the tool calls, the retries and the handoffs. Component tests isolate a single retriever or sub-agent so the fix lands in the right place.

Fiddler's AI agent evaluation guide adds the setup work. Before running anything, Fiddler says to map the agent's input handling, reasoning paths, tool and API calls, and expected outputs. That map lets you say which step failed instead of which run failed.

Why a passing demo proves little about an agent

A single passing run proves almost nothing about an agent, because the same input can take a different path on the next run. Pratik Bhavsar at Galileo puts it plainly in The AI Agent Evaluation Blueprint: agents "operate on non-deterministic paths, capable of solving the same problem in multiple ways." A clean final answer can also hide a bad route. Kritin Vongthongsri at Confident AI writes in his agent evaluation guide that "An agent can look busy, reason intelligently, call the right-looking tools, and still fail to complete the task."

The practical rule I would use is to run every test case more than once and record the spread. A case that passes on some runs and fails on others is a failing case. Flaky passes turn into production incidents, because production is just more runs.

Match the metrics to how much the agent decides

Pick agent evaluation metrics by how much the agent decides on its own. Confident AI's guide to LLM agent evaluation sorts agents into four levels and changes the metric set at each one:

• Generator agent: answers from a prompt and retrieved context. Check answer relevancy, faithfulness and safety. Skip tool and plan metrics, since there is no tool trajectory.
• Tool-calling agent: decides when to call APIs, databases or search. Add tool correctness, argument correctness, step efficiency, task completion, latency and cost.
• Planning agent: sequences multi-step work. Add plan quality and plan adherence, because every tool call can look fine while the plan is wrong.
• Autonomous agent: runs across sessions. Add production sampling, human review and regression sets that grow as new failures show up.

Galileo's evaluation blueprint uses similar ideas under different names, such as tool selection quality and action completion. Fiddler's guide lists task success rate, response time, hallucination and toxicity. For a marketing agent that drafts and updates content, I would start with task completion and tool correctness, and leave plan metrics alone until the agent writes its own plans.

Weight test cases by what a failure costs

Mark every agent test case as blocking or non-blocking, and refuse to ship on any blocking failure no matter what the average says. The evaluation guides from Fiddler, Confident AI and Galileo focus on scores and metrics rather than release gates, and this gate is the decision rule I care about most. An average task completion score treats a slightly long blog intro the same as a page published with the wrong store hours. Those failures do not cost the same.

A case is blocking when the agent's action reaches the outside world or cannot be undone: sending, publishing, deleting or spending. Everything else, such as tone, length or formatting, is non-blocking and can move the average. A blocking case has to pass on every repeat run. One failure out of several runs on a blocking case means the agent is not ready for that action, even if the overall score went up.

Build the agent test set from reviewer edits

The best source of test cases for a marketing agent is the edits a human reviewer already makes. My workflows start with a real input, produce an output someone can inspect, and keep a human review step before it ships. Every rejection or correction in that review step is a labeled failure you did not have to invent.

Save each correction with the input, the agent's output, the corrected version and a one-line reason. Galileo's blueprint describes turning production examples into test datasets. The reason line is what makes a case gradeable. It becomes the rubric line for an LLM judge, or the assertion for a code check, and it stops the test set from filling with cases nobody can score.

Pin the LLM judge before you trust the score

An LLM judge is a model with its own failure modes, so record which judge model and judge prompt produced every score. Confident AI's guide recommends deterministic checks for exact things like tool correctness and LLM-as-a-judge for criteria that depend on the agent's output. That split is right. The cost few teams price in is grader drift.

Grader drift works like this. You upgrade the judge model, scores rise, and the agent gets credit for an improvement it did not make. The fix is a rule: when the judge changes, re-score the old agent version on the new judge first, then compare. Separately, check a sample of judged passes against human labels on a regular schedule.

Vendor claims about judge accuracy deserve the same check. Galileo's blueprint says teams can customize generic metrics "with as few as five annotated examples, improving accuracy by up to 30% and reducing metric development from weeks to minutes." Treat that 30% figure as Galileo's own claim about Galileo's product, and keep your own human-labeled sample regardless of which tool grades the runs.

A copyable eval case file for AI agents

An eval case file is a plain list of test cases with the input, the checks, the grader version and whether a failure blocks release. Copy this into evals/cases.yaml and replace the placeholders. The example agent updates location pages for a local business.

# evals/cases.yaml - one entry per test case
- id: holiday-hours-use-hours-sheet
  input: "Update the holiday hours on the location page for the Main Street store."
  source: reviewer_edit          # reviewer_edit | production_trace | written_by_hand
  reason: "Agent copied hours from last year's page instead of the hours sheet."
  blocking: true                 # output goes to a live page
  must_pass_every_repeat: true
  checks:
    - type: tool_called          # deterministic
      tool: read_hours_sheet
    - type: tool_not_called      # deterministic
      tool: publish_page
      unless: human_approved
    - type: llm_judge
      rubric: "Hours in the draft match the hours sheet exactly. No other copy changed."
  grader:
    judge_model: "<pinned model name>"
    judge_prompt: "evals/judges/hours.md@<commit>"
  added: "<date>"
  owner: "<reviewer name>"

- id: spring-guide-intro-tone
  input: "Draft an intro for the spring planting guide."
  source: reviewer_edit
  reason: "Draft opened with a rhetorical question and a generic claim about gardening."
  blocking: false                # draft only, a person edits before publishing
  must_pass_every_repeat: false
  checks:
    - type: llm_judge
      rubric: "Opens with a specific planting fact. No questions. Matches the brand tone file."
  grader:
    judge_model: "<pinned model name>"
    judge_prompt: "evals/judges/tone.md@<commit>"
  added: "<date>"
  owner: "<reviewer name>"

Run this release checklist against the file before each change ships:

• Every blocking case passes on every repeat run.
• Every case added from a reviewer edit since the last release passes.
• The judge model and judge prompt match the last run, or the old agent version was re-scored on the new judge first.
• A person spot-checked a sample of LLM-judged passes against the rubric.
• Steps and cost per task are no worse than the last release.

When an evaluated agent earns more autonomy

An agent earns more autonomy one step at a time, when that step's eval cases prove the simpler version holds. Galileo's blueprint points to Anthropic's advice to find the simplest solution and add complexity only when needed. In practice, a step moves from draft-for-review to auto-execute only after its blocking cases pass on every repeat and reviewers have stopped editing that kind of output.

Keep the human review step on anything blocking until both conditions hold. I wrote about why so many agents fail after launch in the AI agent production failure rate article. The quality and production kits hold plain-file workflows for review and release steps, if you want a starting structure.

Open your last batch of reviewer edits and turn the first one into a case in evals/cases.yaml today.

How this post was made: drafted with AI assistance (Claude Opus 5.5) from the sources linked above and checked by an automated quality gate that matched every figure and quotation to its source on 2026-10-27.