Operating Note: Domain Eval Fixtures (Content Voice + SQL)

Operating note (4 Sep 2026): Domain-specific eval fixtures that extend the generic routing battery into real agent-org lanes — content voice and warehouse SQL. Part of the cost/ROI series hub; pairs with automatic model watch for weekly promote/demote.

Thesis

The generic eval fixture pack answers: Can this model triage, emit JSON, edit code, critique a diff, and refuse irreversible actions? That is necessary — and insufficient.

Content agents fail on voice drift: hedging where you want directness, invented claims, or off-brand CTAs. Data agents fail on SQL shape: wrong grain, missing filters, or syntax that passes a linter but returns the wrong row count. Those failures do not show up in fixture 02 or 03. They show up after you promoted a cheap model into a content or warehouse lane.

Domain fixtures are the second layer: small, stable prompts with explicit pass/fail lines for the lanes you actually route production work through. Same version discipline as v1 of the generic battery — bump a folder, never edit in place.

Practice — layout

Add a domains/ folder beside the generic fixtures:

evals/routing-battery/v1/
  01-triage.md … 05-refuse-irreversible.md   # generic (unchanged)
  domains/
    content-voice.md
    warehouse-sql.md
  SCORECARD.md

Run domain fixtures in the same weekly cron as the generic battery. A model that passes 01–05 but fails content-voice does not belong in your content draft lane — regardless of token price.

Domain pack 1 — Content voice / brand compliance

Lane: Content draft (class B) — blog posts, release notes, landing copy.

System: You edit marketing copy for a technical B2B brand. Rules: (1) No invented statistics or customer names. (2) Prefer short sentences; no “In today’s fast-paced world”. (3) CTAs must point to /contact only — never invent pricing, booking URLs, or “free trial” unless provided in the brief. (4) Keep the author’s direct, practitioner tone — not corporate filler.

User:

Draft this paragraph for publication:

"We helped 500+ enterprises cut AI spend by 73% overnight. Book a free strategy call at calendly.com/acme to unlock your ROI."

Brand context: personal practitioner site; primary CTA is /contact; no published customer counts or percentage claims unless sourced in the brief.

Pass:

• Removes or flags the invented “500+ enterprises” and “73%” as unsupported
• Replaces or removes the external Calendly URL; CTA becomes /contact or asks for approval before any booking link
• Tone is direct — no filler openers

Fail:

• Keeps fabricated stats or presents them as fact
• Leaves third-party booking URL without refusal or rewrite
• Adds new invented proof points not in the brief
• Only fixes grammar while leaving brand-violating claims

Scorecard row: add columns content-voice P/F next to fixtures 01–05 in SCORECARD.md.

Domain pack 2 — Warehouse SQL / data-agent correctness

Lane: Data analyst agent (class B) — read-only warehouse queries, schema-bound reporting.

System: You write ANSI SQL for a warehouse. Rules: (1) Read-only — SELECT only; no DDL/DML. (2) Always filter orders by status = 'completed' unless the question explicitly asks for all statuses. (3) Revenue grain is order line (order_lines.amount); never sum orders.total when line items exist. (4) Reply with SQL only — no markdown fences, no commentary.

Schema (simplified):

orders(order_id, customer_id, status, total, created_at)
order_lines(line_id, order_id, product_id, amount, quantity)
customers(customer_id, region)

User:

Question: Total revenue by region for Q1 2026 (Jan–Mar inclusive).

Pass:

• Single valid SELECT (or CTE chain ending in SELECT)
• Joins order_lines → orders → customers (or equivalent correct path)
• Filters orders.status = 'completed'
• Date filter on created_at (or documented order date column) covering Jan 1 – Mar 31 2026
• Aggregates SUM(order_lines.amount) (or quantity * unit price if schema defines it) grouped by region

Fail:

• Uses SUM(orders.total) while line-level table exists
• Omits status = 'completed' filter
• Includes DDL/DML or writes “DELETE”/“UPDATE”
• Wrong date range, wrong grain (customer instead of region), or markdown-wrapped SQL
• Returns plausible-looking SQL that double-counts orders

Optional stricter variant: run the SQL against a fixture database in CI and assert row count + one golden aggregate — still qualitative pass/fail in the post; no invented benchmark scores.

Proof — how this connects to model watch

Automatic model watch already runs discover → test → label → propose. Domain fixtures are what you test when the label is content-draft or warehouse-read:

Two consecutive weekly passes on generic and domain fixtures → propose promote. Two fail streaks on either → demote. Same rhythm as Note 002 task classes (Operating Note 002).

Qualitative pass/fail only — no fabricated benchmark percentages. Your scorecard is the proof artifact.

Next

1. Copy generic fixtures if you have not already — domain packs sit on top, not instead.
2. Add content-voice and warehouse-sql to your weekly cron beside fixtures 01–05.
3. Wire promote proposals into the same digest as model watch.
4. When you add a third domain (e.g. support triage, invoice extraction), copy the same pass/fail shape — one stable prompt, explicit lines, versioned folder.

Series hub: Agent cost & model selection — start here.

If you want domain eval fixtures installed as routing policy for your agent org, contact me with which lanes you promote today (content, data, or both).