Agent evaluation frameworks: what they measure and how to pick one

An agent evaluation framework is a set of scenarios, traces, scores and review steps that tell you whether an agent finishes real tasks with the right tools at an acceptable cost. The tool you pick matters less than the spec behind it. My advice: write 20 to 50 scenarios from real failures first, then choose software that can score them and show you the trace.

What an agent evaluation framework has to cover

It has to score behavior across a whole run, not the text of one reply. InfoQ's article on evaluating AI agents says that single-turn accuracy metrics and classical NLP benchmarks like BLEU and ROUGE "don't capture how agents fail in practice." Its example is an order-triage agent that hits a refund API error, silently skips the refund and reports the case as resolved. A text score passes that. A trace check against the database fails it.

Confident AI's guide describes the same problem as a "ghost action": the transcript says done and nothing changed downstream. So every framework you consider should answer four questions:

• Did the task actually complete, checked against real side effects?
• Were the right tools called with the right arguments?
• What did the run cost in steps, tokens and time?
• Where does a person review the output?

Single-run scores hide unreliability

Run each important scenario several times before you trust a pass. Galileo's write-up cites peer-reviewed research on six agents across 300 enterprise tasks, where success fell from 60% to 25% when measured for consistency across eight runs. I have not reproduced that result, and Galileo sells an evaluation platform, so read it as a warning rather than a benchmark. The direction is plausible, though, and it matches Confident AI's advice to re-run critical scenarios because models are stochastic.

My rule: a scenario counts as passing only if it passes on every one of three runs. If that is too strict for your use, say so in the spec and pick a lower bar on purpose.

The metrics that recur across frameworks

Vendors name them differently, but the set is stable. Confident AI lists task completion, argument correctness, tool correctness, conversation completeness and turn relevancy. Galileo lists tool selection quality, action advancement, action completion and agent flow, plus efficiency. Map them to one table and you can compare tools without reading each feature page:

| What you need to know | Confident AI name | Galileo name | |---|---|---| | Did it finish the job | Task completion | Action completion | | Right tool, right inputs | Tool and argument correctness | Tool selection quality | | Did it make progress | Conversation completeness | Action advancement | | Did it follow the intended path | Trace inspection | Agent flow |

Cost and latency belong in the same traces. Confident AI makes the point that two runs can both pass task completion while one burns far more tokens.

Where humans stay in the loop

Keep a human rubric on a sample of traces. LLM judges drift, and a green metric can sit on top of an unhappy user. InfoQ puts it this way: "Human judgment captures what automation misses: tone, trust, and contextual appropriateness." It also recommends a separate judge model from the agent to reduce self-grading bias. I build my own workflows the same way: a real input, an output someone can inspect, and a review step before anything ships. I wrote up that approach in my AI agent evaluation article.

How to pick one

Choose by what you already run, not by metric count.

1. If you have engineers and a repo, an open-source library with CI hooks is enough to start. DeepEval, MLflow, TruLens, OpenAI Evals and Ragas all appear in the sources. Check each one's current docs, since InfoQ notes feature sets change quickly.
2. If non-engineers need to review outputs, weigh the annotation and trace views above the metric list.
3. If you cannot yet write ten scenarios from real failures, buy nothing. The gap is your spec.
4. If the agent can take irreversible actions, add permission and policy checks before you add quality scores. My agent safety checklist covers that part.

A starter eval spec you can copy

Save this as a plain file next to the agent. Fill one block per scenario.

scenario_id:
source: (support ticket, incident, or log line this came from)
input / conversation setup:
tools available:
done means: (a state change you can check, not a message)
expected tool calls and arguments:
budget: max steps ___, max tokens ___, max seconds ___
runs required: 3 (pass only if all pass)
automated checks: (assertions on side effects, judge metric if any)
human review: yes/no, reviewer, rubric
on failure: add a new scenario from the incident

Start with 20 scenarios. Galileo, citing Anthropic's guidance, suggests 20 to 50 tasks drawn from real production failures, and I agree with starting at the low end.

If you would rather start from a set of plain files than a blank page, the Growth Agent Starter Kit in my shop is one option. Paid kits there are one payment with no renewal.

Next step: pull your last 20 agent failures or support escalations and turn each into one block of that spec before you evaluate a single tool.

How this post was made: drafted with AI assistance (Claude Sonnet 5.5) from the sources linked above, then checked by Claude Opus 5.5 and an automated quality gate that matched every figure and quotation to its source on 2026-10-27.