When Cheap Models Get Expensive: Pricing Failure Into Agent ROI
“We switched to a free model and cut the bill 90%.”
Sometimes that is true. Sometimes you moved the bill into retries, incidents, and your calendar.
This note is part of the cost/ROI series (Note 002, model watch). No vendor required — only honest accounting.
The trap
Token dashboards optimize what is easy to measure. Agents fail in ways dashboards ignore:
- Wrong JSON → downstream job no-ops (“success”)
- Weak critique → bad code merges, expensive later
- Rate-limit thrash → five retries, same spend, no outcome
- Human cleanup → the real scarce resource
If you only watch $/1M tokens, you will crown the wrong winner.
A simple ROI sketch
For a workflow W over a week:
ROI_proxy = accepted_outcomes / (model_spend + infra_share + human_hours × hourly_rate)Definitions:
- accepted_outcomes — units that passed your quality bar (published post, green PR, closed ticket)
- model_spend — API bill attributable to W
- infra_share — portion of the ~€35/mo (or whatever) stack
- human_hours — approval + cleanup + incident time
You do not need perfect finance. You need the same formula every week so promote/demote decisions are comparable.
Worked micro-example (illustrative)
| Setup | Outcomes | Model $ | Human h | Proxy cost* | Outcomes per € |
|---|---|---|---|---|---|
| Free model | 40 | 0 | 6 | 600 | 0.07 |
| Mid-tier | 48 | 12 | 1 | 112 | 0.43 |
*Assuming €100/h fully loaded for illustration — use your own number.
The free lane “won” the token chart and lost the week.
Design rules that follow
- Never promote on token price alone — require a pass rate on your model watch battery.
- Price silent failure higher than loud failure — loud failures page you; silent ones corrupt state.
- Separate producer and critic models — same-model self-review shares blind spots (stack notes).
- Keep class D expensive on purpose — irreversible actions should not optimize for pennies.
Where gateway routing still helps
A router (OpenRouter or otherwise) makes model_spend attributable per lane. It does not replace failure accounting. Use both: routing for control, ROI_proxy for judgment.
Practice this week
Pick one agent workflow. Log for seven days: outcomes, model spend, human minutes, silent fails. Compute the proxy once. Decide one promote and one demote.
That single loop teaches more than another vendor comparison table.
Series: Note 002 · Model watch · OpenRouter Cursor example · /hire