Route by completion cost, not token cost — the math that cuts agent expenses 60%

Nvidia just open-sourced their AI cost router, and the math on inference pricing just changed completely. Not because of the router itself — because of what it reveals about how routing should actually work.

Most people are building routers wrong. They route by price: "Use Haiku for cheap tasks, Sonnet for hard ones." But that's not how costs actually work in production.

Here's what we learned after routing 50,000+ agent requests through different models:

Route by failure cost, not token cost.

A $0.02 Haiku call that gives you the wrong answer costs more than a $0.30 Sonnet call that gets it right the first time. Because you'll make that Haiku call three more times, then escalate to Sonnet anyway, then spend 20 minutes debugging why your agent "succeeded" but nothing actually worked.

The Nvidia router does something smart: it measures task completion probability across models, not just speed and price. A coding task that Sonnet completes 95% of the time but Haiku only completes 60% of the time should always route to Sonnet, even if Haiku is 10x cheaper per token.

Here's the routing logic that cut our actual costs 60%:

def route_by_completion_cost(task_type, complexity_score):
    models = {
        'haiku': {'cost_per_token': 0.0001, 'completion_rate': get_completion_rate('haiku', task_type)},
        'sonnet': {'cost_per_token': 0.0015, 'completion_rate': get_completion_rate('sonnet', task_type)}
    }
    
    for model, stats in models.items():
        expected_attempts = 1 / stats['completion_rate']
        true_cost = stats['cost_per_token'] * expected_attempts * estimate_tokens(complexity_score)
        models[model]['true_cost'] = true_cost
    
    return min(models.items(), key=lambda x: x[1]['true_cost'])[0]

The key insight: completion rates vary dramatically by task type. Haiku is actually more expensive for code reviews because it misses edge cases 40% of the time. Sonnet is overkill for data extraction because Haiku gets it right 98% of the time.

We track completion rates by measuring whether the agent escalates, retries, or asks for clarification within 5 minutes. If any of those happen, we count it as a routing failure.

The pattern that changed everything: measure true cost (tokens × attempts), not advertised cost (tokens × price).

Your agent architecture needs this now, not later. Because when everyone else is routing by sticker price, you'll be routing by actual results — and your agents will complete tasks in one shot while theirs loop through three models and still get it wrong.

The math just shifted from "use the cheapest model that works" to "use the most reliable model for each task type." Most agents are still optimizing for the wrong variable.