Semantic cache and token budget calculator

Two gateway-layer controls, priced: answers served from cache instead of the model, and budgets that stop runaway spend before it happens.

An AI gateway sits between every consumer and every model, which makes it the one place two economic controls can live: a semantic cache that answers repeated questions without touching a GPU, and token budgets that cap what any team, tenant, or agent can burn. This calculator prices both.

01: The traffic

02: The cache

Semantic caching matches meaning, not exact strings—'reset my password' and 'how do I change my password' can share one answer.

A hundred requests at the gateway

Green requests are answered from the semantic cache. Blue ones go through to a model.

  • Served from cache
  • Sent to a model

More Info

A gateway is the only component that sees every request before a model does, which is what makes both of these controls possible. Semantic caching compares the meaning of a request against recent answers and returns a match without generating; token budgets meter consumption per team, tenant, or agent and enforce the limit in the request path rather than on next month's invoice. Neither is possible downstream of the model, and neither is reliable when implemented per application.

Frequently asked questions

Where this leads

From AI factory math to AI factory control

The same position enables the rest of the gateway pattern: routing across models and providers, failover when an endpoint degrades, and one audit trail for every consumer—all of it upstream of the first GPU cycle.