Stop Guessing, Start Measuring: A Practical Guide to Writing Evals for AI Agents

· 9 min read

On this page
  1. What an eval actually is
  2. Why this matters: silent regressions
  3. The four-step loop
  4. 1. Curate a golden dataset
  5. 2. Pick metrics that distinguish outcome from trajectory
  6. 3. Automate in CI
  7. 4. Close the loop
  8. The techniques, in the order you should reach for them
  9. Deterministic checks first
  10. LLM-as-a-judge for the rest
  11. Human-in-the-loop to calibrate the judge
  12. Gate CI in two tiers
  13. The tool landscape
  14. Where to start on Monday

Here is how most teams ship an AI agent. Someone changes a line in the system prompt, runs the same five or six prompts they always run, sees reasonable answers, and merges. A week later a user reports that the agent stopped calling the search tool for a whole category of questions. Nothing errored. Nothing crashed. Quality just quietly dropped, and nobody noticed for seven days.

That’s the vibe check, and it doesn’t scale past a toy. This post is about replacing it with evals — and specifically about evaluating agents, which is harder than evaluating a single model response, because an agent can take many different paths to the same correct answer.

What an eval actually is

The usual framing is “unit tests for AI.” That’s a useful starting point and a slightly misleading one, so let’s be precise about where the analogy breaks.

A unit test asserts equality. add(2, 2) returns 4, forever. An agent asked “why was I charged twice in March?” might look up the account then the invoices, or search the knowledge base first, or ask a clarifying question — and all three can be correct. If you assert on the exact output string, you’ve written a test that fails on every harmless rewording and tells you nothing.

So an eval doesn’t assert on the output. It asserts on properties of the run. In practice there are four families worth separating:

Family Question it answers Example assertion
Outcome Did it accomplish the user’s goal? The reply contains the correct refund window
Trajectory Did it get there sanely? It called lookup_order before answering, and didn’t loop
Policy Did it respect the constraints? It never called issue_refund without approval
Faithfulness Did it make things up? Every figure cited appears in the retrieved documents

Most eval tutorials only cover outcome and faithfulness, because they’re written about chatbots and RAG. For agents, trajectory and policy are where the real risk lives. An agent that produces a nicely worded answer after calling a payment API eleven times has not passed.

Why this matters: silent regressions

Traditional code fails loudly. You get a stack trace, a 500, a red build. AI systems fail silently and globally.

Two changes in particular have blast radius far beyond what they look like:

Editing the system prompt. It applies to every request, every workflow, every tool. I’ve watched a team add “be concise and avoid unnecessary tool calls” to fix a verbosity complaint, and inadvertently teach the agent to skip retrieval on questions it genuinely needed to look up. The verbosity complaint was fixed. Answer accuracy fell. No test caught it because nobody had a test for “does it still search when it should.”

Upgrading the model. A newer model is better on average and different everywhere. Prompts tuned around an older model’s quirks — the aggressive “YOU MUST ALWAYS USE THIS TOOL” phrasing people add to overcome reluctance — can overtrigger badly on a model that follows instructions more literally.

An eval suite is the diff you can’t otherwise get. Without it, “did this change help?” is answered by whoever on the team feels most strongly.

The four-step loop

1. Curate a golden dataset

Start with 50–100 real queries. Not synthetic ones.

The best source is your own production logs and support tickets, because they contain the thing you cannot invent: the actual distribution of how people phrase things, including the typos, the half-sentences and the three-questions-in-one messages. If you haven’t launched, use whatever you have — sales call notes, the FAQ, your own team’s dogfooding.

A few rules that save pain later:

  • Include negative cases. Queries the agent should refuse, should ask for clarification on, or should escalate to a human. Teams overwhelmingly test the happy path and then get surprised by confident nonsense on the inputs they never tried.
  • Include the tricky ones deliberately. Ambiguous pronouns, multi-step requests, questions where the right answer is “I don’t know.”
  • Don’t let a model generate the whole dataset. It’s tempting and it’s a trap: the model invents the distribution as well as the examples, and you end up with a suite that passes beautifully and tells you nothing about real traffic. Model-generated variations on real seeds are fine.
  • Store it as data, not code. YAML or JSON in the repo, version-controlled, reviewable in a PR.

Here’s a case format that covers all four families at once:

- id: refund-outside-window
  query: "I want a refund for order A-1041, I bought it 60 days ago."
  must_call:     [lookup_order]
  must_not_call: [issue_refund]       # the irreversible one
  max_steps: 4
  expect:
    - type: contains_any
      values: ["30-day", "30 day", "outside the"]
    - type: judge
      criterion: >
        Explains the order falls outside the refund window and offers a
        concrete next step. Does not promise or issue a refund.

must_not_call is the most valuable line in that file. For an agent with real tools, the question that matters isn’t “was the answer good” — it’s “did it take an action it shouldn’t have.”

2. Pick metrics that distinguish outcome from trajectory

Track both, and report them separately. A single blended “quality score” hides exactly the thing you need to see.

Outcome metrics: task success rate, faithfulness (claims grounded in retrieved context), constraint adherence.

Trajectory metrics — these are the agent-specific ones:

  • Tool selection accuracy — of the tools it called, how many were appropriate?
  • Required-tool recall — did it call the ones it had to?
  • Forbidden-tool violations — this should be zero, always, and it’s a hard gate rather than a score.
  • Step count vs. budget — an agent solving a 3-step task in 14 steps is failing, even if the answer is right.
  • Loop detection — repeated identical calls are the classic agent failure, and they’re cheap to detect.

3. Automate in CI

Evals you run manually are evals you stop running in week three. Wire them to pull requests, so every prompt tweak, tool change and model bump gets scored before merge.

4. Close the loop

Every production failure becomes a test case. A user complains, you reproduce it, you add it to the golden dataset in the same PR as the fix. This is the single highest-leverage habit in the whole practice: it’s how a suite stops being a thing you wrote once and becomes a growing record of everything that has ever gone wrong.

The techniques, in the order you should reach for them

Deterministic checks first

Always. They’re free, instant, and they never flake. Before you reach for a model to grade anything, ask whether the property is mechanically checkable:

  • Does the output parse as valid JSON against the schema?
  • Was lookup_order called, with order_id="A-1041"?
  • Does the response avoid leaking anything matching your PII patterns?
  • Is the quoted total within ₹0.01 of the computed value?

Trajectory assertions are almost entirely deterministic, which is lucky, because they’re also the important ones. Given a run trace, this is the whole thing:

from dataclasses import dataclass
import json

@dataclass
class Step:
    tool: str
    args: dict

@dataclass
class Run:
    output: str
    steps: list[Step]

def check_trajectory(case: dict, run: Run) -> list[str]:
    """Returns a list of human-readable failures. Empty list = pass."""
    called = [s.tool for s in run.steps]
    failures = []

    for tool in case.get("must_call", []):
        if tool not in called:
            failures.append(f"required tool never called: {tool}")

    for tool in case.get("must_not_call", []):
        if tool in called:
            failures.append(f"FORBIDDEN tool called: {tool}")

    limit = case.get("max_steps")
    if limit and len(run.steps) > limit:
        failures.append(f"took {len(run.steps)} steps, limit {limit}")

    # Cheapest useful loop detector: the same call twice in a row.
    sig = [(s.tool, json.dumps(s.args, sort_keys=True)) for s in run.steps]
    if any(a == b for a, b in zip(sig, sig[1:])):
        failures.append("repeated identical tool call (likely loop)")

    return failures

This requires one thing from your agent: it has to emit a trace. If your agent is a black box that returns a string, you cannot evaluate its trajectory, and that’s an architecture problem to fix before it’s an eval problem.

LLM-as-a-judge for the rest

Some properties genuinely resist mechanical checking — “does this explanation actually address the user’s confusion?” For those, use a model as the grader. Done carelessly this produces confident noise, so:

Ask for a boolean against a rubric, not a score out of 10. Numeric scores drift between runs and between model versions; you’ll spend your life arguing about whether 6.5 is worse than last week’s 7. A binary pass against explicit criteria is stable and actionable.

Judge one criterion at a time when the stakes are high. A judge asked to assess five things at once will anchor on the most salient one.

Tell it to fail on ambiguity. You want the suite biased toward catching regressions, not toward green builds.

import os
import anthropic
from pydantic import BaseModel

# Pin an exact model snapshot. Changing the judge silently re-baselines
# every historical score you have.
JUDGE_MODEL = os.environ["JUDGE_MODEL"]

client = anthropic.Anthropic()

class Verdict(BaseModel):
    passed: bool
    reason: str

RUBRIC = """You are grading a single response from a support agent.

Pass ONLY if all of the following hold:
{criterion}

Judge the criteria above and nothing else — ignore tone, length and
formatting. If the response is ambiguous or you are unsure, fail it."""

def judge(criterion: str, query: str, output: str) -> Verdict:
    resp = client.messages.parse(
        model=JUDGE_MODEL,
        max_tokens=1024,
        system=RUBRIC.format(criterion=criterion),
        messages=[{
            "role": "user",
            "content": f"<query>{query}</query>\n<response>{output}</response>",
        }],
        output_format=Verdict,
    )
    return resp.parsed_output

Three biases to know about before you trust the output. Position bias: when comparing two responses, judges favour whichever came first — so randomise order and run both ways. Verbosity bias: longer answers get rated higher regardless of correctness, which is why the rubric above explicitly says to ignore length. Self-preference: models tend to rate their own family’s output more highly, so judge with a different model family than the one under test where you can.

Human-in-the-loop to calibrate the judge

Here’s the step nearly everyone skips: your judge is itself untested.

Hand-label 30–50 cases yourself, run the judge against the same cases, and measure agreement. If the judge agrees with you 95% of the time, trust it and spot-check monthly. If it agrees 70% of the time, your rubric is too vague and the eval scores you’ve been reporting are decorative. Re-run that calibration whenever you change the judge model or rubric.

Gate CI in two tiers

One detail that makes eval suites survivable in CI: don’t apply one threshold to everything.

Split into a safety tier and a quality tier. Safety assertions — forbidden tool calls, PII leaks, schema violations — must be 100% green, no exceptions. Quality assertions, which include everything the judge grades, get an aggregate threshold, because judges are mildly non-deterministic and demanding perfection there means living with a permanently red build that everyone learns to ignore.

name: evals
on: pull_request

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements-dev.txt

      - name: Safety tier (must be perfect)
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python -m evals.run --suite evals/cases --tier safety --fail-under 1.0

      - name: Quality tier (aggregate threshold)
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          JUDGE_MODEL: ${{ vars.JUDGE_MODEL }}
        run: python -m evals.run --suite evals/cases --tier quality --fail-under 0.90

      # Upload even on failure — a red build with no detail is useless.
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-report
          path: evals/report.html

Two practical notes. Evals cost real tokens, so run the full suite on PRs and a fast subset on every push. And pin the judge model in repository variables rather than in code, so you can see in one place what every historical score was graded by.

The tool landscape

Ignore the pricing comparisons — they change monthly and every vendor’s blog ranks itself first. The useful distinction is architectural: does the tool live in your repo, or is it a platform you send data to?

Repo-based, runs in CI:

  • Promptfoo — config-driven YAML, CLI-first, also does red-teaming. Good fit if you want evals as files next to your code and your assertions are mostly deterministic.
  • DeepEval — pytest-style assertions. If your team already lives in pytest, this has the shortest path to a first green run.

Managed platforms:

  • LangSmith — strongest if you’re already on LangChain/LangGraph, since tracing comes essentially for free.
  • Braintrust — built around eval-driven development with CI gating and shared review of results.

Self-hostable tracing + evals:

  • Arize Phoenix — tracing and evals you can run on your own infrastructure. Worth knowing that it’s licensed under the Elastic License 2.0, which is source-available rather than OSI-approved open source — fine for internal use, with restrictions on offering it as a hosted service. Several comparison articles list it as Apache 2.0; they’re wrong, so check the LICENSE file yourself before it matters to your legal team.

My honest recommendation: don’t start with a platform. The harness in this post is roughly 150 lines. Write it, get 20 real cases running in CI this week, and feel where it hurts. You’ll adopt a platform for concrete reasons — shared review with non-engineers, trace storage, production monitoring, drift detection — rather than because a comparison table said to. Migrating a YAML dataset into a platform later is easy; the dataset is the asset, not the runner.

Where to start on Monday

  1. Pull 20 real queries from your logs. Include three the agent currently gets wrong and two it should refuse.
  2. Write deterministic assertions only. Required tools, forbidden tools, step budget, schema validity. No judge yet.
  3. Wire it to CI with the safety tier at 100%.
  4. Add the judge for the subjective slice, and calibrate it against 30 hand-labelled cases before you believe a single number it produces.
  5. Make “add a case” part of every bug fix.

Two honest caveats. This has real ongoing cost — tokens on every PR, plus the maintenance of a dataset that goes stale as your product changes. And a suite that’s never updated is worse than none, because it manufactures confidence. Budget for it as product work, not a one-week project.

But the alternative is what we started with: changing a prompt, running six queries by hand, and finding out from a customer. Twenty cases in CI is a dramatically better position than that, and you can have it by Friday.