Batch Inference Pricing vs Real-Time API Pricing for Async Workloads

What the 50% discount looks like across major providers, and where it breaks

Batch inference and real-time inference run the exact same model and return the exact same output. The only thing that changes is how fast you need the answer back, and providers charge roughly double for skipping the wait. That gap is the center of one of the most avoidable cost problems in production AI: teams paying real-time prices for work nobody was actually waiting on.

The discount comes from a mechanism: a GPU cluster running at sustained 90% utilization costs a provider a fraction of what the same hardware costs at 35% utilization, something like a 2.5x difference on identical machines. A GPU cluster running at sustained 90% utilization costs a provider a fraction of what the same hardware costs at 35% utilization, something like a 2.5x difference on identical machines. Real-time inference forces GPUs to sit ready around the clock, spiking and idling based on whenever a user hits send. Batch work gets scheduled into the gaps, the off-peak stretches where those same GPUs would otherwise sit half-used. The 50% off is payment for letting the provider pick the timing instead of you. It's payment for letting the provider pick the timing instead of you.

So the question engineers should be asking isn't which API costs less. It's whether the workload in front of them needs the latency guarantee it's paying for. A lot of async work doesn't, and teams that never ask end up funding idle GPU time they didn't need to fund.

The 50% figure isn't a marketing number with fine print attached. Google, OpenAI, and Anthropic each run an async batch lane priced at exactly half the sync rate. Three competitors who don't coordinate pricing landing on the same number says something about how uniform the underlying GPU economics actually are.

The numbers make the gap concrete. GPT-5.4 drops from $2.50/$5.00 per million tokens (input/output) on the standard API to $1.25 input in batch. Claude Opus 5 drops to $2.50/$12.50 in batch, a price that turns Opus from a model you'd save for your hardest problems into one you can point at a high-volume async queue without flinching at the invoice. OpenAI applies the flat 50% cut across its lineup, covering models including GPT-5.4, Mini, Nano, and the embeddings models. Anthropic's version, the Message Batches API, promises results within 24 hours at half the standard token price, and Anthropic states that batch carries no quality difference from sync.

The pricing holds up fine. What breaks is the moment you submit the job. That's where the real operational homework starts, and most teams skip it.

The mechanics of submitting and retrieving a batch job

Batch submission doesn't look like a normal API call. Instead of firing off one request and waiting on one response, you upload a single file, one batch, containing up to 50,000 individual prompts, and the provider works through the whole thing as one job.

That job runs during the provider's off-peak windows, and results land back within 24 hours. Treat that number as the outer boundary. Most batches actually come back in something like 1 to 6 hours, depending on job size and how busy the provider's infrastructure happens to be. A pipeline built assuming 24 hours is typical breaks the first time a batch actually takes that long.

The providers don't handle this identically either. Providers differ in how strictly they enforce that window, and the consequences of missing it vary. A hard expiration means a job that misses the deadline just fails outright, and if a downstream pipeline assumed results would show up, somebody has to notice the gap and patch it by hand.

Workload characteristics that qualify for batch, and the ones that do not

Most teams get this backwards: they ask whether a job can tolerate a delay, when the real question is whether anything downstream is actually waiting on the response the second it's generated. If nothing is, it belongs in batch. If something is, batch is the wrong lane no matter how much money it would save.

Strong candidates share that one trait. Document pipelines where the output feeds a later stage that hasn't started yet. Data enrichment work, tagging, classifying, annotating large datasets, where the value comes from volume rather than speed. Nightly analytics runs with a defined window. Offline evals and regression suites. Content queues, product descriptions, drafts, summaries, where publication is scheduled anyway and a few extra hours changes nothing.

Poor candidates are just as obvious once named. Anything user-facing, where latency is the product, a chatbot, a voice interface, a streaming response, doesn't belong in a 24-hour lane. CI/CD eval gates need an answer in seconds, not hours; a batch job that returns after the next deployment already shipped makes the gate pointless. Workloads where one failed row is expensive or time-sensitive to recover from don't fit either. And there's a volume floor: run a job with a few dozen rows, and the overhead of polling, handling failures, and auditing for drift eats more time than the 50% discount saves.

Quality drift deserves its own callout, because for some jobs it's a real disqualifier, not a footnote. The same model, called under the same alias, can score 2 to 5 points lower on quality rubrics through the batch path than through sync. JSON-mode strictness and schema conformance can drift across a single large job too, which matters if a downstream system expects clean, parseable output every time.

The setup that actually survives contact with production runs both lanes side by side, not one instead of the other. Canary requests and the recovery path go through sync, where problems get caught immediately. The bulk of the volume goes to batch. The two lanes aren't competing for the same job; they're doing different jobs, and treating them as complementary is what makes the discount usable instead of theoretical.

How to build the eval loop that batch inference requires

The eval loop most teams already have doesn't survive contact with batch. Submit a prompt, score the response, block the release if the score drops: that pattern assumes an answer comes back in seconds. Batch breaks the assumption outright, since a response might not land for hours.

The fix runs two loops instead of one, and keeps them separate. Trying to force one loop to do both jobs is where teams get stuck.

The inner loop is the actual release gate. Pull a small canary sample from the batch input, run it through the sync API instead, score it against the usual rubrics, and gate the batch submission on that score. Don't let the full batch job gate its own release: by the time it finishes, the decision it was supposed to inform has already been made, or missed.

The outer loop runs after the fact. Once the result file lands, score every row in it. That score isn't a gate on the job that just ran, it's the gate for the next one.

A drift audit underlies both loops: a paired comparison of sync versus batch responses on identical prompts. Anything showing more than a 2 to 5 point gap on a given rubric earns an alarm. Per-row span attribution matters just as much. Without it, a quality dip somewhere inside a large batch leaves no trail to follow. Whoever's on call knows a problem exists but has no way to find which rows failed or what those rows had in common, so nothing gets fixed.

Routing decisions: model tier, open-weight providers, and the self-hosted question

Batch pricing has other levers, and some of them beat the 50% discount outright before it's even in play. Dropping a model tier is usually the bigger win. GPT-4.1 mini, Claude Haiku 4.x, and Gemini Flash 3.1 typically run 8 to 12 times cheaper than their flagship siblings, so for plenty of workloads, a smaller model at real-time pricing costs less than a flagship model in batch. Chasing the batch discount on a flagship model when a mini model at sync pricing would do the job is a common way teams leave money on the table.

Flagship pricing spreads out more than people assume, too. Claude Opus 4.6 runs $5.00 input and $25.00 output per million tokens. GPT-5.2 runs $1.75 input and $14.00 output, 65% cheaper than Opus on input alone while holding its own on plenty of benchmarks. Gemini 3.1 Pro runs $2.00 input and $12.00 output, the price leader among current flagships. Claude Haiku 4.x runs roughly $0.80 input and $4.00 output.

Open-weight providers add a separate axis worth taking seriously rather than treating as a fallback. DeepSeek V3.2, run through a provider like Together AI, prices around $0.14/$0.28 per million tokens net, delivering reasoning close to frontier quality at a fraction of frontier cost.

Speed is its own lever, distinct from cost per token. Groq's LPU hardware runs Llama 4 Scout at $0.11/$0.34, and the draw isn't the per-token price, it's raw throughput. Sometimes tokens-per-second is the number that actually matters, not dollars-per-million-tokens, and mixing the two up leads teams to pick the wrong provider for the job.

What routing this correctly requires at the infrastructure level

Most engineering teams aren't picking one provider and living with it. Most large organizations run three or more model families at once, and that multiplies every batch routing decision by however many providers sit in the mix. Each provider has its own submission format, its own polling mechanics, its own way of handing back results.

Writing that logic by hand, provider by provider, inside application code, is fighting the battle at the wrong layer. It works, until a provider changes its API, and then somebody's stuck maintaining bespoke integration code that has nothing to do with the actual product.

An LLM gateway pulls that logic out of the application. The app just declares what it needs: how much latency the workload can tolerate, roughly what volume is coming through. The gateway decides which provider, and which lane, batch or sync, actually handles the request. That's a cleaner split of responsibility, and it's why more and more enterprises are choosing to run a gateway instead of hand-rolling the routing logic themselves.

Spend visibility across batch and real-time lanes

Most enterprises have watched AI spend blow past budget, and the root cause usually isn't mysterious: nobody ran the cost math before the workload hit production.

Batch spend hides this problem even more effectively than real-time spend does. A single batch job might carry one job ID covering tens of thousands of rows, and until that result file gets pulled apart, there's no way to see which team, which project, or which model actually drove the cost. One number stands in for what should be ten.

The real bill is never just the sticker price per token, either. Input and output lengths rarely match what testing estimated. Caching and batch discounts only apply if requests are actually structured to qualify for them; they don't kick in automatically just because a request hit the batch endpoint. Agent workflows with retries and tool calls quietly multiply the request count in ways that stay invisible until the invoice reveals the total.

Routing at least half of eligible workload through the batch API tends to drop total monthly spend somewhere between 35% and 48%. That number comes with a condition attached: it assumes the routing actually happens the way it's supposed to. Workloads that should hit batch but quietly keep hitting the sync endpoint instead don't show up as a line item anywhere. They sit there as leakage nobody's watching for, because by the time the monthly bill lands, the decision that caused it was made weeks earlier, by someone who's long since moved on to the next sprint. Waiting for the invoice to catch this is already too late. The fix has to sit upstream, at the point where the routing decision gets made, not downstream where someone's stuck explaining it.