Multimodal Token Accounting for Vision and Audio Inputs

Text tokenization has a rough rule of thumb: about three-quarters of a word per token. That rule falls apart the moment an image or an audio clip enters the prompt. Vision and audio inputs get converted into tokens through entirely different mechanics, and those mechanics decide both how much context a request eats and how much it costs. Any team shipping multimodal products needs to understand that pipeline, because the billing meter runs on logic that has nothing to do with words.

Every multimodal model follows roughly the same blueprint. A domain-specific encoder processes the non-text input (an image encoder for pictures, an audio encoder for sound), then a projection step maps that encoder's output into the same token space the language model already understands. The LLM itself stays text-only. It just reads a mixed sequence: some tokens came from the prompt, some from a photo, some from thirty seconds of speech, and once they're all sitting in context, it can't tell the difference.

That means every image and every audio clip turns into a run of tokens tacked onto the prompt, metered and billed the same way word-tokens are, but produced by counting rules that vary by encoder, by provider, and by configuration. Text-plus-image is already the default combination in production systems, text-plus-audio is close behind, and text-plus-video is catching up fast. Most teams priced their multimodal features before anyone worked out what that combination actually costs.

How images become patch tokens: the vision encoder pipeline

Vision encoders are almost always Vision Transformers, or some close variant. The mechanism is simple once you see it: the model chops the image into a grid of small patches, usually 14x14 or 16x16 pixels, and turns each patch into one embedding vector.

Do the math on a standard case. A 224x224 image cut into 14x14 patches produces exactly 256 patch tokens. Token count equals patch count, full stop. Resolution is a direct lever on context usage and, downstream, on the invoice. It's a direct lever on context usage and, downstream, on the invoice.

A handful of encoder families dominate current architectures. CLIP ViT-L/14 runs 24 transformer layers on 224px or 336px inputs and shows up across the earlier LLaVA family, from LLaVA through LLaVA-NeXT, though newer builds like LLaVA-OneVision have moved to SigLIP instead. SigLIP follows the same basic idea as CLIP but swaps in sigmoid loss and gets more out of the same training data. InternViT-6B-448px powers NVLM 1.0 and InternVL. Pixtral-ViT, from Mistral, was built specifically to handle variable-resolution images without forcing everything into one fixed size.

Where things really diverge is in how patch embeddings get handed off to the LLM, and that handoff is a token-count decision just as much as an architecture choice. Most newer open-weight models default to the token-hungry option, and that default deserves more scrutiny than it gets:

Projection-based connectors (LLaVA, NVLM, Pixtral) use a small MLP to map each patch embedding straight to one LLM token. Token count equals patch count, with no compression happening anywhere in the pipeline. It's simple to build. That simplicity has made it the default, even though it's the most expensive option on the list by a wide margin.

Q-Former (BLIP-2, InstructBLIP) uses a learnable transformer to squeeze N patch embeddings down into K query tokens, where K is much smaller than N. BLIP-2 fixes K at a small number regardless of input size. This saves a lot of context, but it costs an extra training phase to teach the compressor what to keep.

Cross-attention adapters (Flamingo, OpenFlamingo) slot new cross-attention layers between the LLM's existing layers, while the LLM itself stays frozen. It adds parameters to the model but avoids touching the base weights.

Three strategies, three very different token bills for the same input image. None of them is the correct one in some absolute sense. Each trades context cost against training complexity and how much visual detail survives the squeeze, and a team that picks projection-based connectors purely because it's the popular default is choosing the most expensive path without ever making that choice on purpose.

How audio frames are projected into the token space

Audio skips patches. There's no 2D grid to slice up. Instead, the audio encoder samples the waveform at a fixed rate and produces a sequence of frame embeddings, which then get projected into the LLM's token space the same way patch embeddings do for images.

The billing consequence follows directly from that mechanism: audio token count scales with duration, and it does so linearly. Longer clips mean more tokens, in a relationship that's predictable in a way vision tokenization often isn't, since resolution and fusion strategy don't factor in the same way.

Concrete numbers help here. Gemini's Live API tokenizes audio input at 32 tokens and audio output at 25 tokens. Separately, audio input through Google's Agent Platform runs at 25 tokens, a rate that shifts depending on platform configuration. These figures are provider-specific, not universal constants. A rate quoted for one API doesn't carry over to another product from the same company, and it definitely doesn't carry over to a competitor.

What the same listed price costs across providers: how tokenization methodology dominates the bill

Three vision models, all listed at what looks like comparable pricing, were run against the same 4K screenshot. The resulting costs came out to $2.00, $9.57, and $19.58. Same image, same task, and nearly a 9:1 spread in what it actually cost to process.

That spread has nothing to do with which model produced better output or which one ran faster. Each provider's token count for that image depends on the encoding methodology, the patch size, and the fusion strategy, all of it baked into a number the customer never sees until the invoice arrives. Picking a vision model on sticker price alone creates a real chance of paying nine times more than the alternative for the exact same input.

Take GPT-4o as a reference point. Vision pricing runs $0.005 to $0.015 per unit, and a single 4K image processed at high detail comes out to roughly 2,550 tokens, working out to about $0.01275 per image. Running that across a batch of 1,000 images brings the total to about $12.75. Small per-image numbers add up fast at production scale, so counting methodology affects the total cost far more than the sticker price does.

GPT-4.1 replaced GPT-4o as OpenAI's recommended production model in 2025. It costs less on both ends ($2.00/$8.00 versus GPT-4o, quoted elsewhere at $2.50/$10.00 depending on tier), and it supports a 1M token context window against GPT-4o's 128K. It does not accept image inputs, which limits its use for vision-heavy workloads but makes it a strong option for text-only or text-plus-code pipelines where the lower cost and larger context window are the priority.

Provider-specific discounts that change the token math for multimodal workloads

Raw pricing never tells the whole story. Every major provider stacks discounts on top of the base rate, and those discounts change the calculus for anyone running multimodal pipelines at scale.

GPT-4 family models offer up to a 90% discount on cached input. That matters for a common pattern: a fixed system prompt paired with a changing image, which describes most document-processing and product-image pipelines. Pay full price once, then pay a tenth of that on every repeat.

Gemini runs a similar scheme through context caching. Cached input costs 90% less than fresh input, dropping from $0.75 to $0.075 on current Flash models. On top of that, Gemini offers batch processing options for workloads that can tolerate being queued instead of run in real time.

OpenAI offers a comparable batch discount on GPT-4o, bringing costs down close to 70% versus standard rates. For any team running large offline jobs, image tagging, transcription cleanup, bulk document review, batch APIs aren't a nice-to-have. Skipping them means leaving most of the available savings on the table.

Difficulty of measuring token counts from multimodal inputs accurately in production

Text token counts are easy. The API hands them back in the response, and logging them means reading a field off a JSON payload. Multimodal token counts don't work that way. Estimating them ahead of a call means knowing the provider's specific encoding methodology, resolution, patch size, fusion strategy, and applying all of it correctly, or just trusting whatever number shows up after the fact.

That gap creates a real measurement problem. Given the near 9:1 cost spread across providers for the same image, dumping "input tokens" from three or four different APIs into one dashboard, without normalizing for how each provider actually counts, produces a number that looks precise and means almost nothing.

Modality mixing makes attribution even harder to trace. A single request with a system prompt, a user question, and three attached images comes back with one blended "input token" figure. Text and vision costs get mashed together in that number, and without breaking it apart by modality, there's no way to tell whether the bill is being driven by verbose prompts or by high-resolution images.

Audio compounds it further, since duration adds a whole extra dimension. A pipeline handling variable-length voice recordings, three seconds one call, three minutes the next, sees token counts swing wildly from request to request. Without logging duration alongside token count for every single call, those swings are just noise, and there's no way to explain them after the fact.

What token compression research reveals about where multimodal costs are generated

Recent research suggests a lot of what gets billed as necessary tokens simply isn't. A study from the University of Surrey (arXiv:2606.10147) found that audio-visual and other non-text token types can be discarded once their information has passed into the LLM, with minimal effect on prediction accuracy and, in some cases, a slight improvement. That result held across multiple models and model sizes, including Qwen2.5-Omni and Video-SALMONN2 Plus.

The ReMo paper (arXiv:2607.21179) goes further. In Omni-LLM inputs, visual tokens make up the large majority of the total input sequence, and most of them turn out to be redundant. ReMo strips out 54% of input tokens on Qwen2.5-Omni without losing accuracy, actually landing at 101.2% and 101.3% of the full-token model's average accuracy across two model sizes. Cutting more than half the tokens and coming out slightly ahead is a strong signal that the original count was inflated well past what the model ever needed.

OmniDelta (arXiv:2607.25669) pushes the same idea to an extreme. At just 25% token retention on Qwen2.5-Omni-7B, it cuts GPU memory use by a meaningful margin and delivers a real end-to-end speedup over full-token inference, without degrading output quality.

Putting those three findings together, the implication is hard to avoid: most visual tokens carry redundant information. The "input tokens" figure on an invoice is a weak proxy for how much information the model actually used. That compression research hasn't made its way into production billing yet. Until it does, every team pays for the full, uncompressed sequence, redundancy included, because the providers doing the billing have no incentive to shrink the number the meter runs on.

Infrastructure requirements for tractable multimodal token accounting

The infrastructure sitting between an application and a model provider has four jobs to do here, and skipping any one of them is how teams end up with multimodal spend nobody can explain.

Pre-call token estimation has to be broken out by modality. Applying the provider's own encoding methodology to an image or audio clip before the request fires turns cost into something planned, rather than something discovered a month later on an invoice.

Per-modality attribution belongs in every log line. Text, image, and audio token counts need to be split apart on each request, then rolled up by team, project, model, and provider close to real time. Skipping this step means nobody can answer the basic question of which modality is driving the bill.

Normalization across providers isn't optional either. A raw token count from Gemini and a raw token count from OpenAI are not the same unit of measurement, not when their encoders, patch sizes, and fusion strategies differ this much. Any system routing the same workload across multiple providers has to account for that difference before comparing numbers, or the comparison means nothing.

Fallback and routing logic needs to account for modality support directly. The GPT-4o to GPT-4.1 transition makes the case cleanly: not every model handles every input type, and pricing, context window, and vision capability all shifted together in that one move. Routing decisions built without that awareness will misroute requests, or send image inputs to a model that was never built to take them.