Routing a JSONL batch in under 50ms

A JSONL file lands at POST /v1/quotes/model. About 50ms later, the client has an estimate, a per-lane cost breakdown, and a small explanation of why we picked the lane we picked. The customer thinks of this as a single request. Internally it's a tiny pipeline — parse, validate, shortlist, score, split, persist, serialize — and the whole thing has to run inside one Cloudflare Worker invocation without ever touching the dispatch queue. This post walks the hot path step by step.

We're focused on the quoting path here, not the dispatch path. Quoting is read-mostly and synchronous. The expensive work — calling the provider's Batch API, polling for status, downloading the artifact, stitching results — happens later, after batch creation with quote_id. Keeping quoting fast is what lets an agent loop fire 50 quotes a minute against /v1/quotes/model without spending its budget on round trips.

The hot path

From the worker's perspective, the request handler is a six-step pipeline. The shape is roughly this:

// src/index.ts — POST /v1/quotes/model handler (paraphrased)
export async function handleQuote(req: Request, env: Env): Promise<Response> {
  const t0 = performance.now();

  // 1. Parse JSONL (streaming, line-by-line, fail-fast on bad rows)
  const parsed = await parseJsonl(req.body, { maxBytes: 10_485_760 });

  // 2. Validate + estimate tokens per row
  const validated = validateRows(parsed.rows); // throws on schema or dup custom_id

  // 3. Shortlist eligible lanes per distinct model
  const shortlist = await shortlistLanes(env, validated.models);

  // 4. Score every (model, lane) pair
  const scored = scoreLanes(validated, shortlist, env.SLA_SNAPSHOT);

  // 5. Split mixed-model JSONL into one internal lane per model
  const lanes = splitIntoLanes(validated, scored);

  // 6. Persist quote, return estimate + explanation
  const quote = await persistQuote(env.DB, lanes);
  return jsonResponse(quote, { ms: performance.now() - t0 });
}

Each step is small on purpose. The parser is a streaming reader, not a JSON.parse over a buffered string — at 5,000 rows we don't want to allocate a 10MB string in V8 just to throw it away. Validation runs inline on each parsed row so we can fail fast on the first bad custom_id instead of finishing the parse and then discovering trouble. Shortlisting is a Map lookup. Scoring is a tight inner loop over a list that's almost never longer than six lanes per model. Splitting is a group-by. Persistence is a single D1 prepared statement.

Where the time actually goes

We instrument every step with performance.now() deltas and ship them to a Durable Object that holds a rolling 60-second histogram. The p50 breakdown for a typical 1,200-row mixed-model JSONL looks like this:

Two honest caveats. The p99 number isn't a hot-path regression — it's almost entirely D1 write tail latency, which we've measured spike to ~120ms during a maintenance window once a week. We've considered moving the quote write off the request path by returning a quote_id before the write completes, but the next public hop is POST /v1/batches with that returned quote_id. Batch creation depends on the persisted quote snapshot when it reserves credits and dispatches, so the quote write stays on the quote request path. The other caveat: cold starts add 40-80ms the first time a Worker isolate sees a request, but on Cloudflare's network we get warm isolates for the vast majority of traffic.

Decisions made up front

A lot of what makes the hot path fast is what isn't on the hot path. Three things in particular.

The model catalog is an in-memory map, populated at module init from a versioned KV namespace. It's about 8KB and covers ~140 model slugs. The lookup is a Map.get — single-digit microseconds. The cost is paid once per isolate, on first invocation, as part of cold start. We rebuild the catalog on a webhook from our admin tool whenever a lane goes into or out of rotation, and we ship the new isolate via a normal Cloudflare deploy when the schema changes.

Token estimation is a heuristic, not a tokenizer call. Calling a real BPE tokenizer per row would be 0.5-2ms each, which on a 5,000-row batch turns into multi-second quote latency. Instead we estimate from the request shape: count the JSON string characters in the prompt and message fields, apply a per-model coefficient we calibrated against real provider receipts, and clip to a sensible band. The estimate is wrong by 3-7% on average and we know it; the actual settlement at receipt time corrects it, and we hold a small headroom in the credit reservation precisely so the customer never gets surprised.

Lane health comes from a snapshot, not from polling providers at quote time. A Durable Object (PlatformEventHub) maintains an SLA snapshot updated every 30 seconds — recent error rate per lane, recent p95 latency, recent capacity headroom. The scorer reads this snapshot from the DO via a single internal RPC. Reading per-quote means we never depend on a provider being reachable to *quote* a job; we only depend on it being reachable to dispatch one.

The splitter

About 30% of incoming JSONL references more than one model. The splitter groups rows by model, produces one internal lane per group, and assigns each lane its own dispatch envelope. The customer-facing batch_id stays stable across the split — when results come back later, we re-stitch them into a single output file by custom_id. The customer sees one ID in, one results file out.

// Splitter — group rows by model, score lanes per group.
function splitIntoLanes(
  rows: ValidatedRow[],
  scored: Map<string, ScoredLane[]>,
): InternalLane[] {
  const byModel = new Map<string, ValidatedRow[]>();
  for (const r of rows) {
    const bucket = byModel.get(r.model) ?? [];
    bucket.push(r);
    byModel.set(r.model, bucket);
  }

  return [...byModel.entries()].map(([model, modelRows]) => {
    const best = scored.get(model)![0]; // already sorted asc by score
    return {
      laneId: best.laneId,
      model,
      rowCount: modelRows.length,
      tokenEstimate: modelRows.reduce((s, r) => s + r.tokenEst, 0),
      providerCost: best.providerCost,
      controlPlaneFee: 0.02, // per internal lane, flat
    };
  });
}

The $0.02 control-plane fee per internal lane is what covers the additional dispatch, poll, and stitch overhead. On a typical job that splits into two lanes, this is $0.04 against a savings of $4-8 versus running sync. We line-item it on the receipt so customers can see exactly what splitting cost them.

When the hot path goes cold

Three things push us out of the sub-50ms band, and we handle each one differently.

• Large JSONL files (10MB+). We stream the parse and apply backpressure on the input reader. Worst case at 10MB is ~140ms parse time. Files above 10MB are rejected with a 413; the customer is told to use the input_file_id flow, which references a pre-uploaded R2 artifact and lets us defer the parse to a background job.
• Unknown models. A row referencing a model slug that isn't in the catalog is a hard fail at validate. We return a 400 with the offending custom_id and the row index. No estimate is produced. This is intentional — mid-batch model deprecations are real and we'd rather the customer see an error at quote time than a partial result a day later.
• Capacity squeezes. If the chosen lane is within 10% of its in-flight token ceiling, the splitter may fan out across two lanes for the same model. This is rare — it triggers on maybe 0.4% of quotes — and is configurable per workflow. When it triggers, the per-row cost stays the same; the customer pays one extra $0.02 control-plane fee.

Why this doesn't use a queue

Cloudflare Queues are a great primitive and we use them heavily for dispatch, retry, and webhook fan-out. But quoting deliberately stays on the request path. The reason is the agent loop: an agent that's deciding whether to submit a batch is asking us should I, not please do. If the answer requires a queued background job and a status poll, the agent's loop length triples. Synchronous quoting keeps the agent's reasoning tight: ask, get a number, decide, accept.

There's also a simpler argument. Quoting is read-mostly. We read the model catalog (in-memory), the SLA snapshot (DO RPC), and the customer's wallet balance (D1 read, cached for 5s). The only write is the quote record itself, which is small and indexed by quote id. If we ever push beyond what a single Worker invocation can do, the right move isn't a queue — it's to push the parser and validator into a smaller, dedicated WASM module. We haven't needed to yet.

We'll go deeper on dispatch and retry in a follow-up — that's where the queues, the Durable Objects, and the provider-specific quirks actually live. Quoting is the easy half. The fact that it has to be fast, predictable, and synchronously honest is what makes it interesting.