Why we built Batchrouter

Every major model provider sells the same inference at two prices. The sync price — what you pay when you call /v1/chat/completions and wait for tokens. And the batch price — what you pay when you're willing to wait up to 24 hours. The batch price is usually 50% lower. Sometimes more. This isn't a secret tier, it isn't gated, and it isn't hard to find — it's right there in OpenAI's and Anthropic's pricing pages.

And yet, when we talk to teams burning real money on LLMs — classification queues, nightly enrichment, eval runs, document pipelines, agent backfills — almost none of them are using batch. They know the discount exists. They've usually tried it once. They went back to sync. We started Batchrouter because that gap is the most expensive habit in modern AI infra, and the reason it persists isn't laziness. It's that the 50% you save almost never survives contact with the operational work it takes to actually claim it.

The hidden tax of async

A sync call is one line of code. A batch job is a project. You have to assemble JSONL in a format the provider accepts, upload it to their storage, poll a job endpoint for state transitions, handle the partial-failure case where 12 of 4218 lines errored, download results, parse them, reconcile them with your original input by custom_id, and only then do you get to use the output. Multiply that by two providers if you want to route across OpenAI and Anthropic — because their batch APIs aren't the same shape, the error semantics differ, and the file formats diverge in small but expensive ways.

Then there's billing. Each provider has its own credit balance, its own minimum top-up, its own invoice cycle. Reconciling a sub-cent batch line against a provider invoice that lands a week later is genuinely awful. We've watched competent teams give up on batch not because the savings weren't real, but because finance couldn't tell them, with any confidence, what a given batch had actually cost.

That's the operational tax. And in our experience, it eats the entire 50% discount for any team smaller than a dedicated platform group. Most teams quietly stay on sync, pay double, and tell themselves it's worth it for the simplicity. They're not wrong — given those two options.

Routing as the missing layer

We think there's a third option: a router. Not a wrapper, not a proxy — a routing layer that owns lane selection, retries, settlement, and reconciliation, and exposes the OpenAI Batch API shape on top. You PUT a JSONL file. You GET a batch back. Same custom_id semantics, same status enum, same results format. Underneath, we pick the cheapest eligible lane that meets your SLA and privacy constraints, dispatch to the provider, retry on a different lane if the first one errors, and settle the cost back to your credit balance in micro-dollars.

The reason this is interesting is that lane selection is genuinely a decision problem, not a lookup. Eligible depends on the model. Cheapest depends on the current provider rate plus our 5.9% service fee — and native batch lanes (OpenAI Batch, Anthropic Message Batch) carry an automatic 10% discount on the provider rate, which usually beats whatever the dynamic option would be. Capacity matters because OpenAI Batch has a 200MB-per-file ceiling and rate limits per org. Privacy matters because some workloads must not leave a specific region. Solving this once, well, in a dedicated control plane is dramatically less work than every customer reinventing it inside a queue worker.

One customer batch ID, many internal lanes

Real production JSONL is rarely a single model. A nightly enrichment job might use gpt-4o-mini for the cheap rows and claude-3-5-sonnet for the hard ones. A document pipeline might mix a Haiku classifier with an Opus extractor. Most batch APIs reject mixed-model files outright; you end up running two or three separate jobs and stitching their results back together client-side. That's more state, more polling, more failure surfaces.

Batchrouter accepts mixed-model JSONL and splits it on the routing side. You get back one customer-facing batch_id. Internally, we run one lane per distinct model — each with its own dispatch, retries, and settlement — and merge results back into a single output file, ordered by your original custom_id. The control-plane fee for managing those splits is $0.02 per internal lane, which is real but tiny against the savings. Your idempotency story stays simple: one ID in, one results file out, regardless of how many providers we actually touched.

Usage-based receipts

When you submit a batch, we estimate cost from your input tokens plus a generous output-token reserve, and we hold that amount against your credit balance. When the job lands, we settle to actual usage and release the rest. The receipt you get is precise to sub-cent, because we hold the math in micro-dollars and only round at display time.

The implication is small but important: you stop paying for the headroom you didn't use. On a single batch this looks like rounding. Over a fiscal year of nightly jobs it compounds into a number your finance team actually notices. We'll go deeper on the settlement model in a follow-up post.

Where we're going

Three directions, in priority order.

Workflow products. Most batch jobs we see are one of five shapes — document classifier, structured extractor, rerank scorer, summariser, translate. We're shipping these as outcome contracts at /workflows: you describe the shape (input schema, output schema, target quality), we pick the lane, cache what's safe to cache, and bill 11.8% all-in. Often net cheaper than DIY routing, always less code.

Agent-native onboarding. Increasingly the buyer is an agent, not a human. We expose UCP 0.1 at /.well-known/ucp, ACP 1.0 at /.well-known/acp, an MCP server card, and an RFC 9727 API catalog — so a sufficiently autonomous agent can discover Batchrouter, register, top up credits, submit a job, and reconcile a receipt without ever loading a dashboard. We're scored 100/100 at isitagentready.com.

Open commerce discovery. The same agent-native surface is open standards, not proprietary. We want a world where any router — ours or someone else's — speaks the same discovery language, and an agent can comparison-shop between routing platforms the same way it currently comparison-shops between flights. We're early on this. We'll write more.

If you ship batch jobs, or you're paying sync prices for work that could wait an hour, you're who we built this for. Start with the calculator — drop in a typical job, see what it would cost on the cheapest eligible lane. Then /docs/quickstart walks you through your first batch in about five minutes. The OpenAI Batch shape means you almost certainly don't need to change your code.

The 50% discount has always been there. We just want to make it survivable.