Automatic Model Watch: Know What New Models Are Good At Before You Route Traffic
New models appear weekly. Marketing pages claim they are “best for coding” or “best for reasoning.” If you run agents in production, that is not enough.
You need a model watch: an automatic loop that answers what is this model actually good at for my workloads? before it touches real traffic.
This sits in the same niche as Operating Note 002 — cost/ROI through better selection — but it does not require a specific gateway. You can feed results into OpenRouter routes, Cursor rules, LiteLLM, or a handwritten allowlist.
Why manual “try it once” fails
- You remember the win, forget the silent failure
- Ranking shifts overnight; your anecdote is stale
- Agents amplify a weak model across hundreds of runs
A watch turns anecdotes into a roster with labels.
The minimal loop
discover → smoke-test → label → propose → (human) promote/demote → log1. Discover
Sources you already have access to:
- Provider changelogs / model lists (API or RSS)
- Aggregator catalogs if you use a gateway
- Your own “candidates” list from peers and release notes
Output: candidate_id, date seen, source URL.
2. Smoke-test (fixed battery)
Keep the battery small and stable — otherwise you cannot compare week to week.
Example battery (adjust to your agents):
| ID | Task class | What you measure |
|---|---|---|
explain-fn |
A | Correctness + latency |
extract-json |
A | Schema validity |
edit-with-tests |
B | Tests green / patch applies |
critique-diff |
C | Catches planted bug? |
refuse-irreversible |
D | Refuses or asks for approval? |
Run the same fixtures every time. Score pass/fail + latency + token estimate. Store raw outputs for a week.
3. Label
Translate scores into capability labels, not star ratings:
good: triage, json-extractweak: multi-file-refactorunsafe-alone: external-actionscost-tier: free | cheap | mid | frontier
Labels map onto the A–D classes in Note 002.
4. Propose, don’t auto-promote
Automation should propose roster changes:
Promote
model-xto lane A (triage). Demotemodel-yfrom lane B after 3 fail streaks onedit-with-tests.
A human (or a high-bar policy) accepts. Auto-promote into class D is how you get surprising publishes.
5. Log
Keep a changelog: date, model, lane change, evidence links. That log is content gold later — and auditability for yourself.
Where to run it
Patterns that work on a lean stack:
- Cron + script on the same VPS as the agents
- n8n schedule → HTTP to model APIs → write results to git/Notion/Supabase
- Weekly digest (email or Tech Pulse bullet): “3 candidates tested, 1 proposed for lane A”
I already use a weekly review habit for Cursor routing (OpenRouter writeup). Model watch is that habit made mechanical and agent-wide.
Failure modes to design for
| Failure | Mitigation |
|---|---|
| Battery drifts (you keep editing prompts) | Version the battery; treat edits as a new experiment |
| Provider rate limits | Queue candidates; test off-peak |
| Good demo, bad agent tools | Include at least one tool-using fixture |
| Cost of testing exceeds savings | Cap candidates/week; prioritize models you would actually route |
How this positions the work
Cost optimization is not only “pick a cheaper API.” It is knowing the frontier of capability for your tasks so you never overpay for A-work or underpower D-work.
Next in this series: when “cheap” models create expensive failures — and how to price that into ROI.
Related: Note 002 · €35 content org · /hire