Lessons from routing 100M tokens

Last week we crossed 100M routed tokens. That's not a lot in the grand scheme of inference, but it's enough volume that the routing engine has met most of its real-world failure modes, and we've gotten to make corrective changes to it under load. Here are five lessons from the production logs — each one a thing we genuinely didn't predict, and each one of which changed how the engine works today.

1. Mixed-model JSONL is the norm, not the exception

When we designed the splitter, we assumed mixed-model JSONL would be a minority case — maybe 5-10% of incoming jobs. We sized the per-lane control-plane fee around that assumption and gave splitting a deliberately spartan UI in the dashboard. Then we started looking at the data.

The pattern is consistent across customer segments. Document pipelines run a cheap classifier (Haiku, gpt-4o-mini) against every row, then send the ones that scored above some threshold to a stronger model for extraction. Enrichment jobs use a small model for the long tail and a frontier model for the hard cases. Eval runs mix a reference model and a candidate. Once you actually let customers send mixed JSONL, they will, because it's how production work was always shaped — they just couldn't express it that way before.

What we changed: we made splitting a first-class path, not a fallback. The quote response now line-items the per-lane breakdown by default, the dashboard renders split jobs as a single batch with sub-lane status, and the webhook payload includes a lanes[] array so a customer's reconciliation script doesn't have to infer the split from the line items.

2. Latency flips routing even when prices are identical

For models available on multiple lanes — Claude on Anthropic direct vs. Claude on Bedrock, for instance — the published prices are often pennies apart. Our first version of the scorer treated them as effectively tied and tie-broke on a static lane preference. That was wrong, and the bug was visible in the SLA reports before it was visible to us.

{
  "model": "anthropic.claude-3-5-sonnet",
  "lanes": [
    {
      "lane_id": "anthropic-direct",
      "provider_rate_per_mtok": 3.00,
      "effective_rate_per_mtok": 2.70,
      "p95_dispatch_ms": 480,
      "p95_completion_minutes": 47,
      "recent_error_rate_24h": 0.003
    },
    {
      "lane_id": "anthropic-bedrock-us-east-1",
      "provider_rate_per_mtok": 3.00,
      "effective_rate_per_mtok": 2.70,
      "p95_dispatch_ms": 720,
      "p95_completion_minutes": 112,
      "recent_error_rate_24h": 0.011
    }
  ]
}

The Bedrock lane was 2.3x slower at p95 completion, mostly because of regional queue depth, and had a 4x higher error rate during one provider incident. At identical price, the engine should clearly route to the direct lane. The old scorer didn't, because it only consulted latency as a tiebreaker below a price threshold of $0.0001 per row — and identical published rates plus a small public-provider discount put both lanes just barely above that threshold for some token counts and below it for others. Routing was effectively non-deterministic in the tied case.

What we changed: latency and recent error rate are now part of the primary score, not a tiebreaker. The score formula is (cost × 1.0) + (p95_completion_minutes × cost_per_minute_penalty) + (error_rate × penalty). The penalty coefficients are tuned per workflow — a workflow that cares about completion time pays more attention to latency, a workflow that cares about absolute floor cost weights cost more. The default profile prefers the direct lane in the example above by a margin large enough that no token count flips it.

3. Webhook redelivery dominates the post-completion cost

We deliver a webhook when a batch lands. For a $0.30 batch that's a thirty-cent operation; for a $40 batch, it's still one delivery. The variable cost we hadn't sized correctly was the redelivery path. A customer endpoint that returns a 503 once, or that doesn't respond inside our 10-second timeout, triggers an exponential-backoff redelivery sequence: 30s, 2m, 10m, 1h, 6h, 24h. On low-cost batches with intermittently slow customer endpoints, the redelivery work — Worker time, R2 reads of the cached payload, DO state updates — was approaching the cost of the original dispatch.

What we changed: we batch low-cost completion webhooks into a single batches.completed event delivered every 30 seconds, instead of one event per batch. Customers who care about immediate per-batch notifications can opt back into per-batch delivery. We also expose a webhook.delivery.attempts metric in the dashboard so customers can see when their endpoint is causing retries. The economics line up better — and as a side effect, customers fix their own slow endpoints faster when the metric is staring at them.

4. Idempotency tokens are mandatory, not optional

OpenAI's batch format uses custom_id to identify a row. We accept the same field. Early on, we treated duplicate custom_ids in a single JSONL as a warning, not an error — we'd dispatch the batch, the provider would do what it does, and the customer would discover the duplication when they tried to merge results back into their database by custom_id. The first time this actually mattered, a customer had a 4,200-row job where two rows shared an id by accident. The downstream consumer overwrote a row of clinical-trial data with the wrong response.

That incident was three lines of code on our side. The customer wrote a frank post-mortem and we made the corresponding change in the engine.

What we changed: duplicate custom_ids are now a hard validation failure at quote time. The error response includes every duplicated id and the row indices where each duplicate appeared. We also extended idempotency to retries: when a row fails on the primary lane and gets re-dispatched to an alternate lane, we attach an idempotency token so the alternate provider can't accidentally double-bill us, and we attach a re_dispatch_count to the row so the customer's reconciliation can detect it.

5. Mid-batch model deprecations actually happen

A model that's listed in our catalog at the moment a customer quotes their job is not necessarily still listed by the time the provider dispatches it twenty minutes later. This is rare — twice in 100M tokens — but it has a long tail of unpleasant consequences. The job dispatches, the provider returns a model-not-found error, the retry policy fires, the alternate lane is also using the deprecated model id, and now we're spending real time figuring out what to do with rows that nobody can serve.

What we changed: we snapshot the model id pin at quote time. The quote record stores the exact model_id plus the provider's model_snapshot (where available — Anthropic exposes these, OpenAI exposes them for some models). On dispatch, we send the snapshot, not the slug. If the snapshot is deprecated between quote and dispatch — which has happened exactly twice — we surface an explicit model.deprecated_post_quote error to the customer, refund the credit reservation, and propose an alternate model in the response. The customer re-quotes; we don't try to be clever about substituting.

What we got right

In the spirit of balance — five things we'd build the same way again if we started today.

• One customer batch id across splits. Even when we run three internal lanes, the customer sees one batch_id and gets one merged results file. It's the single biggest reduction in customer-side complexity, and it's the thing most existing batch APIs don't do.
• Usage-based receipts in micro-dollars. Holding the math in micro-USD and only rounding at display means a customer with 50,000 small jobs doesn't accumulate a rounding drift. The first time a finance team reconciles us against a provider invoice and the numbers tie to the cent, you can hear the sigh through the support thread.
• Sub-cent precision in credit reservations. Reserving in micro-dollars and settling against actual usage means customers never overpay for headroom. On a few-large-jobs profile this looks like rounding noise. On a many-small-jobs profile it compounds to ~3-5% over a quarter.
• Quote synchronously, dispatch asynchronously. Keeping the quote path read-mostly and synchronous means agents and humans get the same fast feedback loop. The expensive work hides behind accept, where it belongs.
• Snapshot lane health, don't poll providers on the hot path. A 30-second SLA snapshot in a Durable Object is cheap, available, and good enough. Quoting never depends on a provider being reachable.

More to come on dispatch, retries, and the cost of stitching split results — those are bigger posts. In the meantime, /docs/webhooks covers the redelivery semantics we landed on, and /docs/api/batches has the current custom_id validation rules and the model_snapshot pinning behavior.

100M tokens isn't a finish line. It's enough volume to be honest about what worked, and to write down what we'd tell a past version of the team to do differently. The list will be different at a billion.