From API Keys to Cost Centers — Routing LLM Traffic Through a Proxy to Get Team and Model Spend in Real Time

Real-time AI spend tracking by team and project requires routing model requests through a proxy that records organizational identity, model, provider, tokens, and cost together. This architecture turns shared API consumption into attributable spending across providers. The proxy also makes budgets and model-access policies enforceable, giving teams the infrastructure to replace reconstruction from monthly invoices with management of AI spend as a cost center.

Why API keys are a broken unit of account for LLM spend

An API key establishes who can authenticate. A single credential may serve several teams, environments, projects, and models, while the provider’s invoice reports total consumption against that credential. Finance needs a different unit of account. It needs to know which team spent what, on which model, for which project, and the key alone does not encode those relationships.

Teams may also keep their own provider keys, write their own retry logic, and use different formats to report costs. That fragmentation makes comparisons harder even when each team has some local visibility. A total can be accurate without being attributable.

Multiple providers add another boundary to reconciliation. A team using OpenAI directly, Anthropic directly, and AWS Bedrock can have spending spread across three billing surfaces. As of 2026, both OpenAI and Anthropic models can instead be consolidated onto a single Bedrock account and bill, and AWS Cost Explorer provides per-model and per-provider breakdowns for Bedrock spend. That consolidation improves the provider view. It does not, by itself, identify the internal team or project responsible for each request.

How production AI traffic grows into a cost-attribution problem no dashboard can fix retroactively

At small scale, a key per team can provide a workable approximation of ownership. Production traffic exposes the limits of that arrangement as teams share workflows, environments, and model access. More keys multiply the security surface and make rotation harder. They still do not explain which model consumed the tokens or which feature caused the call without additional request-level context.

By the time a cost spike appears in an end-of-month review, its traffic ran days or weeks earlier, and the code responsible may already serve another production workflow. A dashboard cannot show context that was never recorded. Joining logs afterward works only if they contain the organizational identity and usage details needed to link a charge to its owner.

Shadow usage creates a more fundamental gap. Engineers can call provider APIs directly from laptops or personal accounts, leaving no trace in the organization’s key-based dashboard. The cost may arrive only as a corporate card line coded to “software.” The architectural requirement is therefore a shared point through which organizational traffic passes, where ownership is captured before it is lost and where direct access can be controlled.

What a proxy actually does to LLM traffic and why that creates a natural attribution layer

A proxy sits between the application code and the model providers. Instead of calling each provider through its own SDK, the application calls the proxy’s single endpoint. The proxy authenticates the caller with an internal virtual key, adds routing metadata, and forwards the request to the selected provider. It records the transaction as the response returns. From there, the proxy can tie the request identity to the model target, token count, and price.

The proxy issues keys scoped to a team or project and maps them to provider credentials held centrally, rather than distributing those credentials to applications. Each call is recorded under its assigned scope. Because the proxy controls the outbound connection, it can attach team, project, environment, or user context to the usage record before storage.

That timing separates attribution from ordinary log aggregation. The proxy does not have to match timestamps across independent systems to infer who caused a charge. Ownership and token usage belong to the same record. Portkey’s gateway buyer’s guide describes this centralized proxy and control layer as a unified interface for routing, credentials, policies, and model observability across providers. The same request path also gives policies a place to run, so budget checks, model-access rules, and sensitive-data redaction need not be duplicated in every application.

How can teams track AI API costs by team, project, model, and provider in real time?

A spend-tracking proxy produces at least four dimensions for each request, including the caller’s team or virtual key, the model, the provider, and token usage with its cost. Together, those fields connect consumption to responsibility. Per-team attribution shows engineering managers which squad is driving growth, while the provider dimension keeps that view coherent when a team uses more than one service.

A model breakdown shows where volume lands across a mixed-model architecture, including whether premium models are serving tasks that do not require them. Project or key scope connects spending to a product surface or customer segment. That relationship supports product-level margin accounting or customer-level chargeback, rather than leaving finance to allocate a shared total without evidence of its origin.

Because the proxy writes usage at request time, spend totals can be current to the minute, letting teams watch a deployment’s token rate before it becomes a budget problem. Showback shows those costs to the teams that caused them without moving money. Chargeback books them against team budgets in the accounting system. Most teams start with showback while they build trust in the tagging, but the same request records work for either accounting choice.

What the proxy must enforce at request time to keep the spend data trustworthy

The proxy’s record is complete only if relevant traffic passes through it. An application that can call a provider directly can bypass both attribution and budget policy. Credential lockdown closes that route for centrally managed access. Provider keys live only in the proxy, while applications receive virtual keys that have no value outside it. Credentials obtained separately and use through personal accounts still need governance because a dashboard cannot find them on its own.

A proxy can hold a request or return an error when a team’s or project’s token budget is exhausted. These are configurable controls. They show excess consumption before it grows, while role-based access control at the gateway sets which teams may use which models. Guessing an expensive model’s identifier should not bypass its access policy.

An immutable audit log attributes each request to a virtual key and preserves its model and token count, giving finance and security a line-by-line account that the spending team cannot edit. PII and PHI redaction before forwarding, coupled with logging that excludes raw personal data, keeps the audit record reviewable without turning it into another store of sensitive content. Trustworthy spend data depends on these controls because missing traffic, editable ownership, or unsafe logs weaken its usefulness.

Provider fallback and routing decisions that affect cost, and why they belong in the proxy

The choice of model sets the cost before any usage record is written. Sending a routine task to a premium model is a spending decision, even if it appears in application code as a technical default. When each team makes that choice independently, centralized reporting can only describe the result. Routing logic gives the proxy a way to act. It can distribute requests by task complexity, cost tier, or latency target, reserving premium models for requests that justify their expense.

When a primary provider returns a rate-limit error, a proxy can route the request to a secondary provider while keeping the resulting usage in the same attribution structure. The alternative may be a hard failure followed by more aggressive application retries at higher cost. A fallback therefore belongs in the spending policy. Its destination changes which provider and model receive the work.

Caching changes whether a provider call is needed at all. With semantic caching, identical or semantically similar queries can receive a cached response with zero token cost, and the ledger records the cache hit with zero new provider consumption. Central routing also makes model-cost experiments easier to isolate. Redirecting a share of traffic becomes a configuration change rather than an application deployment, while the unchanged team and project dimensions show the experiment’s cost impact in real time.

Managed versus self-hosted proxies, and which setup gives teams spend visibility without adding operational burden

Self-hosting gives teams full data-residency control and the most freedom to make changes, but they must also deploy, scale, and keep the proxy running. That work belongs in the deployment decision. LiteLLM Proxy requires PostgreSQL in production deployments and Redis when more than one proxy instance runs. Its SSO and audit-log features sit behind a paid enterprise license, so a prototype’s requirements differ from those of a production deployment with compliance obligations.

A fully managed proxy transfers gateway hosting and maintenance to the provider. Teams change a base URL instead of running the service themselves, and the gateway supplies spend tracking, fallbacks, and governance. That removes a specific infrastructure burden. It does not remove the need to choose organizational scopes, budgets, or access policies. For teams focused on shipping product features, this division of responsibility can make production visibility achievable without building a dedicated gateway operation.

Concentrate AI offers a fully managed gateway with a wide model catalog, automatic fallbacks, and real-time spend tracking by team, project, key, model, and provider. It uses one base URL and has no markup or platform fee. There is no proxy for the customer to maintain. For AI-native startups and scale-ups, it is a concrete alternative to self-hosting LiteLLM or connecting separate provider SDKs.

Compliance can determine which deployment is acceptable. Workloads that require prompts to remain within a controlled network boundary call for an in-VPC deployment, while a managed gateway with zero data retention and a BAA is relevant where the organization’s requirements permit that arrangement. Concentrate AI offers a HIPAA-compliant configuration with PHI redaction, zero data retention enforcement, and audit logging, with a BAA available on its paid plan. The evaluation should also establish whether SSO, RBAC, audit logs, and PII redaction are available from the outset, since gating necessary controls behind a later contract can encourage workarounds and shadow spend.

Treating AI spend as a cost center once the proxy is in place

A cost center needs a complete spending record, attribution to the unit that caused it, and a way to hold that unit accountable. When traffic is required to pass through the proxy, it provides those foundations for each token. The organizational work can then begin. Spending by team, model, and project can then be reviewed each week alongside cloud infrastructure costs, with live usage checked next to deployment frequency and error rates.

Budget ownership becomes practical because teams can see consumption while they still have choices to make. A team can receive a token budget for a sprint, monitor its remaining allocation, and use that information in model selection. The feedback arrives during the work itself. Finance can access model-level breakdowns, while security can inspect request audit records from the same data source, without asking engineering to reconstruct each view or change application code.

A budget set after the invoice arrives cannot change requests that have already run, and adding a team label to a spreadsheet cannot enforce model access. The proxy supplies the missing operational boundary. Its purpose is not another dashboard, but cost-center infrastructure that binds each request to an owner and places spending controls on the path where consumption happens.