Evals for analytics agents: grading answers against the ledger

TL;DR

• An agent that is right 85% of the time is useless in finance unless you know which 15% it gets wrong. Accuracy has to be a number with a date on it.
• Build the golden set from real requests, compute ground truth from certified models rather than the agent's own semantic layer, freeze each case at a known_at date, and include questions that should be refused.
• Score numbers and tables deterministically with per-metric tolerances. Use a model judge only for narrative, never for the number inside it.
• Tag every failure from a fixed taxonomy. The distribution tells you where to fix: metric descriptions, semantic-layer joins, tool schemas or the prompt.
• Run the suite in CI on every relevant change and nightly, three repeats per case, gating on overall and per-tag drops plus refusal recall and precision.

An analytics agent that is right 85% of the time is worse than no agent in finance, unless you know which 15% it gets wrong. A controller can work with a tool that is reliably right about balances and reliably refuses to answer about margins. She cannot work with a tool that is right most of the time about everything, because then every number needs checking and the tool has saved nobody anything.

So the question I care about is not “is the agent good?” It is “for which questions, by how much, and did yesterday’s change make it worse?” That is a testing problem, and we already know how to solve testing problems. This post is how I build and run evals for an agent that answers questions against finance data.

Why vibes are not enough

The usual way teams judge an analytics agent is a demo plus a week of people trying it. That process has two defects. It over-samples the questions people find interesting, which are rarely the ones that break. And it has no memory: when someone changes the system prompt in March, nobody re-asks the forty questions that worked in January.

The things that change an agent’s answers are also more numerous than people expect. The prompt. The tool descriptions. The model version. The semantic layer definitions, which in my setup are the agent’s only path to data (I argued for that in the semantic layer is the API for AI). And the data itself, because a restatement changes the right answer to an old question. Any of these can move accuracy by several points in either direction. If you are not measuring, you find out from a stakeholder.

The golden set

A golden set is a list of questions with known correct answers, frozen at a point in time. Mine has about 140 cases. The rules I follow for building it:

Harvest, don’t invent. Every question comes from a real request: a ticket, a Slack thread, a month-end email. Invented questions are too clean. Real ones say “AUM end of last month for the institutional book excluding the test accounts”, and that phrasing is what the agent has to handle.

Ground truth comes from certified models only. The expected answer is computed by a query a human wrote and reviewed, against models that have passed certification. Never against the same semantic layer the agent uses, or a bug in a definition becomes a bug in both the answer and the key, and the test passes.

Freeze the clock. Each case has a known_at date. Expected values are stored, not recomputed on every run. Finance data restates, and if the key moves silently the eval is measuring the data pipeline instead of the agent. The bitemporal approach in as-of vs. as-was is what makes this practical: the ground-truth query filters on knowledge time, so it returns the same answer next month.

Include questions that should be refused. Roughly 15% of my cases have no correct number: an undefined metric, a period that has not closed, a dimension the person is not entitled to. The right answer is a refusal with a reason.

Here is one answerable case and one refusal, in YAML:

- id: aum-eom-by-segment-2025-12
  question: "What was AUM at the end of December by client segment? Exclude test accounts."
  source: request_queue          # harvested, not invented
  known_at: 2026-01-15           # knowledge date (end of day, New York) the values were frozen at
  metric: assets_under_custody
  grain: [client_segment]
  period: { kind: month_end, value: "2025-12" }
  answer_type: table
  key_columns: [client_segment]
  value_columns: [aum_usd]
  tolerance: { rel: 0.0, abs: 0.005 }   # money: half a cent
  difficulty: medium
  tags: [semi_additive, exclusion_filter]
  ground_truth_sql: |
    select client_segment, sum(market_value_usd) as aum_usd
    from certified.fct_custody_balance_bitemporal
    where balance_date = date '2025-12-31'
      and coalesce(is_test_account, false) = false
      -- half-open: everything known before midnight New York on Jan 16
      and recorded_at < timestamp '2026-01-16 00:00:00 America/New_York'
      and (superseded_at is null
           or superseded_at >= timestamp '2026-01-16 00:00:00 America/New_York')
    group by client_segment
  expected:
    - { client_segment: institutional, aum_usd: 812450120.37 }
    - { client_segment: retail,        aum_usd:  96210004.12 }

- id: refuse-customer-ltv
  question: "What's our customer lifetime value this year?"
  source: request_queue
  answer_type: refusal
  refusal_reason: no_governed_metric
  difficulty: easy
  tags: [refusal]

The tags field is what turns a single accuracy number into something you can act on. More on that below.

Scoring

Different answer types need different graders, and I resist the temptation to use a model as the judge for anything that has a number in it.

Answer type	Grader	Notes
Single number	Numeric comparison with per-metric tolerance	Counts exact; money to half a cent; rates to 1e-4 relative
Table	Result-set equivalence	Normalize column names, ignore row order, compare values with tolerance
Refusal	Did it refuse, and was the reason category right?	Answering a should-refuse question is the worst failure
Explanation	Rubric graded by a model, 10% spot-checked by a human	Only for narrative; never for the number inside it

Tolerances belong to the metric, not to the case. A dollar amount reported in the general ledger has a precision; an FX-converted figure has another; a fee rate in basis points has a third. If each case author picks a tolerance, someone will eventually pick 1% to make a flaky case pass.

The scorer itself is deliberately boring. The harness asks the agent to return its final answer as structured JSON with the case’s column names, which removes most of the parsing problem. What remains is type normalization, row matching and tolerance:

import math
from dataclasses import dataclass


@dataclass
class Result:
    passed: bool
    reason: str = ""


def to_number(v):
    """Coerce int, float, Decimal or a numeric string like '$1,234.50'. None stays None."""
    if v is None:
        return None
    if isinstance(v, bool):
        raise TypeError("boolean is not a number")
    if isinstance(v, str):
        v = v.replace(",", "").replace("$", "").strip()
        if v == "":
            return None
    return float(v)


def numbers_match(expected, actual, rel_tol=0.0, abs_tol=0.0):
    e, a = to_number(expected), to_number(actual)
    if e is None or a is None:
        return e is None and a is None   # NULL matches only NULL
    return math.isclose(e, a, rel_tol=rel_tol, abs_tol=abs_tol)


def _key_part(v):
    # (is_null, text) so a NULL key never collides with an empty string
    return (v is None, "" if v is None else str(v).strip().lower())


def _index(rows, key_cols):
    out = {}
    for row in rows:
        row = {str(k).strip().lower(): v for k, v in row.items()}
        key = tuple(_key_part(row.get(c)) for c in key_cols)
        if key in out:
            return None, key          # duplicate key: almost always a fan-out
        out[key] = row
    return out, None


def score_table(expected, actual, key_cols, value_cols, rel_tol=0.0, abs_tol=0.0):
    exp, dup = _index(expected, key_cols)
    assert exp is not None, f"golden set has a duplicate key: {dup}"
    act, dup = _index(actual, key_cols)
    if act is None:
        return Result(False, f"duplicate key in answer: {dup}")
    if exp.keys() != act.keys():
        missing, extra = len(exp.keys() - act.keys()), len(act.keys() - exp.keys())
        return Result(False, f"row sets differ: {missing} missing, {extra} extra")
    for key, erow in exp.items():
        arow = act[key]
        for col in value_cols:
            if col not in arow:
                return Result(False, f"missing column {col}")
            try:
                ok = numbers_match(erow.get(col), arow[col], rel_tol, abs_tol)
            except (TypeError, ValueError):
                ok = False
            if not ok:
                return Result(False, f"{key} {col}: expected {erow.get(col)}, got {arow[col]}")
    return Result(True)


def score_case(case, answer):
    if case["answer_type"] == "refusal":
        if not answer.get("refused"):
            return Result(False, "answered a question that should be refused")
        ok = answer.get("refusal_reason") == case["refusal_reason"]
        return Result(ok, "" if ok else "refused for the wrong reason")
    if answer.get("refused"):
        return Result(False, "refused an answerable question")
    tol = case.get("tolerance", {})
    return score_table(case["expected"], answer.get("rows", []),
                       case["key_columns"], case["value_columns"],
                       tol.get("rel", 0.0), tol.get("abs", 0.0))

Two details earn their keep. Treating a duplicate key in the agent’s answer as an immediate failure catches fan-out before any value comparison, and the reason string says so. And math.isclose with both a relative and an absolute tolerance handles the zero case correctly: a relative tolerance alone would demand an exact match whenever the expected value is 0.

Failure taxonomy

A failing case gets a tag from a fixed list. I tag by hand for the first few weeks, then have a model propose the tag from the trace and a human confirm it. The distribution is the most useful output of the whole system, because each category points at a different fix.

Failure	Typical symptom	Usually fixed in
Wrong metric	Chose net_revenue when gross was meant	Metric descriptions
Wrong grain / fan-out	Totals inflated, often by an exact multiple	Semantic layer joins
Summed a semi-additive measure	Balance totals about N times too large, N = days in the period	Measure type in the layer
Wrong period or time zone	Off by one day’s activity	Tool defaults, fiscal calendar
Missed a filter	Test accounts or intercompany included	Prompt, entity model
Hallucinated field	Tool call rejected, then improvised	Tool schema enums
Stale data	Answered from an unclosed period	known_at default
Should have refused	Invented a metric from parts	Prompt, refusal examples
Right number, misleading narrative	Correct table, wrong “up 12%”	Explanation rubric

When I started, a third of failures were grain or semi-additive errors, which told me the agent still had a path to raw tables. After closing it, those went to nearly zero and “wrong metric” became the largest bucket. That is a much better problem: it is fixed by writing clearer definitions, which also helps the humans.

Running it in CI

The suite runs on every pull request that touches anything that can change an answer, and nightly against the current model endpoint, because a model alias can start pointing at a new version without notice.

name: agent-evals
on:
  pull_request:
    paths: ["agent/prompts/**", "agent/tools/**", "semantic/**", "agent/config.yaml", "evals/**"]
  schedule:
    - cron: "17 6 * * *"
jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e ".[evals]"
      - run: python -m evals.run --suite golden --repeats 3 --out results.json
      - run: >
          python -m evals.gate --results results.json --baseline evals/baseline.json
          --max-drop-overall 0.02 --max-drop-per-tag 0.05
          --min-refusal-recall 0.95 --min-refusal-precision 0.95

Each case runs three times. For anything with a dollar figure I require all three runs to pass, because a question that is right two times out of three is wrong in production one time in three. The gate fails on an overall drop of more than two points or a five-point drop in any tag. It also fails if refusals slip in either direction: recall (the share of should-refuse questions it actually refused, the worst failure) or precision (the share of its refusals that were warranted). The per-tag rule matters: I have seen a prompt change lift overall accuracy from 81% to 84% while the semi-additive bucket fell from 92% to 75%. The aggregate looked like progress.

A full run of 140 cases times three is roughly 420 agent sessions. With a mid-sized model that is about 25 minutes in parallel and low tens of dollars, which is cheap next to one wrong number in a board pack.

What goes wrong in practice

Ground-truth drift. Because the ground-truth query filters on knowledge time, a restatement alone does not move the key. Code does: someone fixes a bug in a certified model, and the same query now returns a different answer for the same known_at. That is the third clock from as-of vs. as-was. I re-run every ground-truth query weekly and open a ticket when it disagrees with the stored value. A human decides whether the old key or the new code is wrong, and the change is reviewed like code.

The golden set goes stale. Questions harvested in a quiet quarter do not cover what people ask at year-end. I add ten to fifteen cases a month from the request queue and retire ones that test metrics we have deprecated.

Overfitting the prompt. If the same person writes prompt changes and golden cases, the prompt slowly accretes special cases for the eval. Keep a held-out slice of about 20% that prompt authors do not see.

Grading narratives with numbers in them. A model judge will pass “AUM rose to $908M” when the truth is $908.7M and the agent said up 4% when it was down. Extract numbers and score them deterministically; let the judge handle only tone and completeness.

Cost creep. Tracking accuracy without tokens and latency per case lets a change that doubles cost for a one-point gain slip through. Put cost on the scorecard.

The evals are the spec

The scorecard goes to the people who rely on the agent, broken down by tag: here is what it is good at, here is what it refuses, here is what it still gets wrong. That publication changes the conversation. Finance stops asking “can we trust the AI?” and starts asking “can we trust it for month-end balances?”, which has an answer.

Without evals, “the agent is pretty good now” is an opinion. With them, it is a number with a date on it, and you can tell when it changes.