Certifying data models with an agent in the loop

TL;DR

A certified label should be the output of a repeatable, logged process, not a senior person having looked at the model once.
Deterministic gates run first: naming, owner, column docs, enforced contracts, required tests and ticket-traceable commits. No tokens are spent on what a rule can check.
Tests run against production data inside a BigQuery temp-table script with a hard byte cap, so nothing persists and nothing can run away.
A coding agent is used only to triage failures into a strict JSON schema. It never applies the label, and its proposed patches are never applied automatically.
Certification went from one to two days in queue plus an hour of review to about five minutes and $1.50 to $3 per run, so the label now tracks the code.

A dbt model passes through static gates, a sandboxed test run and a traceability check; a failure is handed to a triage agent before the certified label is applied.

For a long time “certified” in our data catalog meant that someone senior had looked at the model once, probably in a meeting, possibly while it was being built. The badge stayed green through forty subsequent commits. Nobody could tell you what had been checked, against which data, or when.

In finance that is not good enough. A certified label is a promise to accountants and auditors that a number can be used in a report without re-deriving it. A promise like that should be the output of a repeatable process with a log, not a social one. Over the past few months I built that process as a Claude Code plugin for my team. This post describes its shape: what the deterministic gates check, how tests run against production data without touching it, what the agent is and is not allowed to do, and what it costs.

The problem with “someone looked at it”

Manual certification fails in predictable ways. Reviewers check what they remember to check, so coverage depends on who is on rotation. The review happens once, so it certifies a commit rather than a model. And the queue is slow: a certification request typically waited a day or two for a reviewer and took 45 to 60 minutes of their attention, which meant people stopped asking and uncertified models crept into reports.

The obvious response in 2026 is “have an agent review it”. I think that is half right. An agent is very good at reading a failing test, a diff and a query result and saying what kind of problem it is looking at. It is a poor substitute for a rule that can be checked mechanically, and it should never be the thing that applies the label.

Architecture

The /certify run. Rules check everything they can; the agent only reads failures; the label is applied by a pipeline.

The plugin packages four things:

a /certify command that orchestrates the run;
a skill containing my team’s data standards in a form the agent can follow;
Python scripts that implement the gates;
a triage subagent with a narrow prompt and a structured output schema.

The scripts do the checking. The agent does the reading.

The gates

Static gates run against the dbt manifest (the compiled metadata for every model in the project) and are the same code that CI runs on every pull request. There is no private path to green. The model must follow naming conventions for its layer, have an owner in its metadata and document every column. It must declare an enforced contract: column names and types that fail the build if they change. And it must carry the tests the standards require: unique and not-null on the declared grain, relationships to the dimensions it references, accepted values on status columns. Incremental models must declare a unique_key and a partition filter. Hard-coded table names instead of ref() fail. Measures must declare whether they are additive or semi-additive, which is the same metadata the semantic layer uses downstream. If any static gate fails, the run stops and lists the violations. No tokens are spent explaining a missing description.

The gates are configured in YAML so the standards are reviewable in one place:

certification:
  applies_to: ["models/marts/finance/**"]
  static:
    require_owner: true
    require_column_docs: 1.0          # fraction of columns documented
    require_contract_enforced: true
    required_tests:
      grain: [unique_combination_of_columns, not_null]
      foreign_keys: relationships
    incremental:
      require_unique_key: true
      require_partition_filter: true
    forbid: [hardcoded_relations, select_star_in_final_cte]
  sandbox:
    max_bytes_billed: 2000000000      # 2 GB hard cap per job; dry run must estimate below it
    partitions_to_test: 7             # most recent N partitions for incremental models
  traceability:
    ticket_pattern: "[A-Z]{2,6}-\\d+"
    since: last_certification
  triage:
    allow_patch_paths: ["models/**"]  # never tests/, never contracts
    max_patch_lines: 40

The sandboxed run is the interesting part. Tests in CI run against a small fixture dataset, which is good for logic and useless for finding out whether production data violates the grain. So the plugin compiles the model with every ref() resolved to the production tables, wraps it in a BigQuery multi-statement script, and builds it into a temp table. Every test then runs against that temp table inside the same script. When the script ends, the temp table is gone.

-- Generated per run. The service account can read prod marts and create temp tables, nothing else.
create temp table candidate as
select * from (
  -- compiled body of fct_fee_accrual_daily (abridged), refs resolved to prod;
  -- grain is client × schedule × day
  select b.client_id, b.d as accrual_date, b.schedule_id, b.basis_usd,
         ... as fee_usd                -- tier pricing logic
  from `prod.finance_marts.fct_fee_basis_daily` as b
  -- ... joins to schedules and tiers ...
  where b.d >= date_sub(current_date('America/New_York'), interval 7 day)
);

select 'grain_not_null' as test,
       countif(client_id is null or accrual_date is null or schedule_id is null) as failures
from candidate
union all
select 'grain_unique', count(*)
from (
  select client_id, accrual_date, schedule_id
  from candidate
  group by client_id, accrual_date, schedule_id
  having count(*) > 1
)
union all
select 'fee_non_negative', countif(coalesce(fee_usd, 0) < 0)
from candidate
union all
select 'schedule_exists', count(*)
from candidate c
left join `prod.finance_marts.dim_pricing_schedule` s
  on s.schedule_id = c.schedule_id
where c.schedule_id is not null and s.schedule_id is null;

Before execution the plugin does a dry run and refuses to proceed if the estimate exceeds the cap; the job is then submitted with maximum_bytes_billed set, so a bad estimate still cannot run away. Most finance marts are small and partitioned, so a typical run scans kilobytes to a few megabytes, though on-demand pricing still bills BigQuery’s 10 MB minimum per table referenced. The largest model we certify scans under 2 GB because only the last seven partitions are built. Every job ID and its billed bytes go into the run report.

Traceability is a short script: list every commit that touched the model’s files since its last certification and check each message for a ticket reference. A model with an untraced change cannot be certified until someone links the change to a ticket, which in practice takes thirty seconds and has twice surfaced changes nobody on the team remembered making.

Where the agent comes in

Only when something fails. The triage subagent receives the failing gate output, the model’s diff since last certification, the relevant rows from the test queries (capped and redacted), and the team’s standards skill. It must return JSON that validates against this schema:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["category", "summary", "evidence", "confidence", "proposed_patch", "escalate_to"],
  "properties": {
    "category": {
      "type": "string",
      "enum": ["model_logic_bug", "upstream_data_issue", "test_definition_wrong",
               "upstream_schema_change", "standards_violation", "transient"]
    },
    "summary": { "type": "string", "maxLength": 400 },
    "evidence": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["source", "ref", "excerpt"],
        "properties": {
          "source": { "type": "string", "enum": ["test_result", "diff", "query_rows", "manifest", "log"] },
          "ref": { "type": "string" },
          "excerpt": { "type": "string", "maxLength": 1000 }
        }
      }
    },
    "confidence": { "type": "string", "enum": ["high", "medium", "low"] },
    "proposed_patch": { "type": ["string", "null"], "description": "Unified diff, or null." },
    "escalate_to": { "type": ["string", "null"], "description": "Owner from model metadata, or null." }
  }
}

The plugin enforces two rules the schema cannot express: exactly one of proposed_patch and escalate_to is non-null, and any patch must apply cleanly, touch only paths allowed in the config, and stay under the line limit. A proposed patch is never applied automatically. It is attached to the report, and if a developer applies it the whole certification runs again from gate 1.

The schema is what makes triage testable. I keep a regression set of about 60 past failures with the category a human assigned, and every change to the triage prompt runs against it, the same approach as in evals for analytics agents. Prompts are config: versioned, reviewed, and blocked from merging if category agreement drops.

Numbers

These are from roughly 130 runs over the first three months, across about 40 certified models.

Measure	Before	With the plugin
Wall-clock per certification	1–2 days in queue, 45–60 min review	≈5–6 min per run
Reviewer time per model	45–60 min	~10 min reading the report
Model cost per run	n/a	≈$1.50–3 (higher when triage runs)
BigQuery scan per run	n/a	KB-scale for most models (billed at the 10 MB minimum); 2 GB at most
Triage category agreed with human	n/a	~85%

The cost line surprised people. A full run with triage costs less than the coffee during the meeting it replaced. The time line matters more: because certification is now cheap, we re-certify on every merge to a certified model instead of once, and the label actually tracks the code.

What goes wrong in practice

The agent patches the test instead of the model. The first version of triage, faced with a uniqueness failure, proposed relaxing the test to a warning. Technically that made the run green. That is why patches to tests/ and contracts are rejected by path before a human ever sees them.

Upstream lateness looks like a bug. About half of early failures were late-arriving source data that made a relationships test fail for the most recent partition. Triage correctly called them upstream_data_issue but people still lost time on them. We now exclude the current partition from relationship tests for sources with a known landing lag.

Gates can only check what you wrote down. Two models passed every gate and were later found to sum a balance over days in a downstream report. Neither gate nor test could see it, because the misuse happened outside the model. The fix was in the standards: semi-additive measures must declare it, and the semantic layer enforces it. Certification proves a model meets the standards, not that the standards are complete.

Confident triage on thin evidence. When the failing rows were redacted heavily, the agent still returned high confidence. I added a rule that confidence cannot be high unless evidence includes at least one query_rows or test_result item, and low-confidence results escalate by default.

Permissions drift. The sandbox’s safety depends on the service account being able to read production and create only temp tables. Someone granted it dataset write access for an unrelated job. The plugin now checks its own effective permissions at startup and refuses to run if it can write anywhere.

Rules first, the agent for the residue

The pattern I would take to any “is this artifact fit for use?” decision is the order of operations. Check everything a rule can check, with the same rules for humans and agents. Run the real test against real data in a sandbox that cannot persist anything and cannot spend much. Require provenance for every change, the same instinct as the manifests in turning regulatory PDFs into agent packs. Then give the agent the residue: the judgment calls about what a failure means, returned in a structure you can test and audit.

The label itself is applied by a pipeline reading a report. That is the part that turns “certified” from a statement about someone’s confidence into a statement about a process, which is what an auditor wanted all along.