Your Token Bill Is a Design Problem, Not a Pricing Problem

The first serious AI invoice arrives and the response is almost always procurement. Someone asks whether there is a committed-use discount. Someone else prices the same workload at a competing provider and finds it is 20 percent cheaper on input. A third person suggests capping usage per customer. All three are answers to the question "how much are we paying per token," and none of them is the question that matters.

The bill is a function of two things: the rate you pay per token, and the number of tokens you send and receive. The rate is a vendor's decision and it moves in single-digit percentages. The token count is your decision, it is made in your code, and in most systems that have never been tuned it is inflated by a factor of three to ten. You are not overpaying for tokens. You are buying tokens you do not need, on every single call, and then negotiating the price of the ones you should not have bought.

Start by finding out which half of the bill you are arguing about

Before touching anything, split the spend. Input tokens and output tokens are priced differently, usually with output several times more expensive per token, and teams reason about cost as if output dominates because output is the part they can see. In a retrieval-heavy or agentic system it almost never does.

Here is an illustrative worked example. It uses round numbers so you can redo it with your own; nothing in it is a measured Abisam result.

Take an assistant handling 50,000 requests a day. Each request sends a 12,000-token prompt (system instructions, tool definitions, retrieved documents, conversation history) and returns a 600-token answer. Assume a frontier model at 3 dollars per million input tokens and 15 dollars per million output tokens.

input  = 50,000 x 12,000 = 600,000,000 tokens/day
       = 600 M x $3 / 1M  = $1,800 / day

output = 50,000 x    600 =  30,000,000 tokens/day
       =  30 M x $15 / 1M =   $450 / day

total  = $2,250 / day, about $67,500 over 30 days

Input is 80 percent of that bill, and output is the part everyone was trying to shorten. The prompt engineering effort went into "be concise" instructions that address a fifth of the problem while the other four fifths are decided by whatever your context assembly step happened to stuff in.

Do this split first. Every lever below is worth a different amount depending on which side of it you are standing on, and you cannot prioritize levers you have not sized.

Context you did not choose is context you still pay for

Ask what is actually in those 12,000 tokens. In systems that have not been audited, the answer is usually some version of this:

• A system prompt that has been appended to for a year and never pruned, including instructions for features that shipped and were removed.
• Full JSON schemas for every tool the agent could theoretically call, including the twelve it never calls in this code path.
• The top eight retrieved chunks, because eight was the default in the tutorial, at whatever chunk size the ingestion pipeline used.
• The entire conversation history verbatim, because truncation felt risky.
• A few thousand tokens of few-shot examples added during a quality push in month two, kept because nobody wanted to be the person who removed them.

None of these is unreasonable on its own. Together they are the bill. The discipline is to make each one a measured decision rather than an accumulated default.

Concretely: retrieve eight chunks and rerank down to three, rather than sending eight and hoping. Send only the tool definitions reachable from the current state. Summarize conversation turns older than the last few instead of resending them verbatim. Cut few-shot examples one at a time and watch your eval pass rate rather than assuming they earn their tokens.

Suppose that work takes the prompt from 12,000 tokens to 5,000.

input  = 50,000 x 5,000 = 250,000,000 tokens/day
       = 250 M x $3 / 1M = $750 / day

total  = $750 + $450 = $1,200 / day, about $36,000 over 30 days

That is a 47 percent reduction in total spend from one lever, before a single conversation with a vendor. Compare it to the 20 percent input discount someone was chasing, which on the original prompt would have saved 360 dollars a day, and note that the two are not exclusive. The point is the order of magnitude, and which one you should spend your quarter on.

The check on this lever is not vibes. Cut context against a frozen eval set and read the pass rate. If quality holds, the tokens you removed were never doing work. If it drops, you have learned something specific about what the model actually needed, which is more useful than the tokens you saved.

Every stable prefix you resend uncached is a discount you declined

Most providers offer prompt caching: send the same prefix again within a short window and the cached portion bills at a steep discount, commonly a small fraction of the normal input rate. The mechanism is prefix-based, which is the part teams miss. It matches from the start of the prompt forward, and it stops matching at the first byte that differs.

So the ordering of your prompt is a cost decision. Put the stable material first - system instructions, tool definitions, policy text, long-lived few-shot examples - and the volatile material last: the retrieved chunks, the user's turn, the timestamp. Teams routinely destroy their own cache hit rate by interpolating a current date, a request id, or a per-user greeting into the top of the system prompt. One variable token at position 40 makes the next 4,000 tokens uncacheable.

Continue the example. Of the trimmed 5,000-token prompt, say 4,000 is a stable prefix and 1,000 varies per request. Assume cached input reads at 0.30 dollars per million, one tenth of the standard rate.

cached prefix  = 50,000 x 4,000 = 200 M x $0.30 / 1M = $60 / day
variable tail  = 50,000 x 1,000 =  50 M x $3.00 / 1M = $150 / day
input          = $210 / day   (was $750)

total          = $210 + $450 = $660 / day, about $19,800 over 30 days

Two design changes, no model change, no vendor negotiation, and the monthly figure has moved from about 67,500 dollars to about 19,800. Use your provider's actual cache rate, minimum cacheable prefix length, and cache lifetime when you redo this; they differ, and some providers charge a small premium to write the cache entry, which matters if your traffic is too sparse to get reads out of it.

Instrument the hit rate. A cache you believe in but never measured is a common way to find out in month three that a single dynamic field has been costing you the entire discount.

One model for every call is a routing decision you made by not making it

Almost every AI product starts by pointing everything at the strongest available model, which is correct: you are trying to find out whether the product works at all, and a capability question should not be confounded by a cost optimization. The mistake is leaving it there once the traffic mix is known.

Look at what the calls actually are. In a typical assistant, a large share of the volume is not the hard reasoning task the product is named after. It is classification (is this question in scope), extraction (pull the order number out of this sentence), routing (which tool applies), short factual lookup, and summarization of retrieved text. These are tasks where a small model, given clean input, is at or near parity - and a small model is commonly an order of magnitude cheaper per token.

The arithmetic, again illustratively. Say 70 percent of requests are these simple calls and a small model prices at 0.30 dollars per million input and 1.50 per million output, one tenth of the frontier rates. Take the post-caching per-day figure of 660 dollars and split the traffic:

30% on the frontier model: 0.30 x $660 = $198 / day
70% on the small model:    0.70 x $660 x 0.10 = $46 / day

total = $244 / day, about $7,300 over 30 days

The honest caveat: routing is the lever most likely to cost you quality if you do it by feel. It needs a classifier you trust, an eval set per route, and an escalation path where a low-confidence small-model answer gets retried on the strong model. That retry has a cost too, and if your escalation rate is high enough the routing saves nothing. Measure the escalation rate as a first-class metric. It is also the lever that tells you the most about your product: a route that escalates constantly is a task you have misunderstood.

Retrieval that runs every turn re-buys documents you already have

Two related patterns, both common, both invisible in a demo.

The first is re-retrieval within a conversation. A user asks about a policy document, you retrieve four chunks, the model answers. The user asks a follow-up. Naive implementations run retrieval again from scratch, get largely the same chunks, and pay for them a second time. Over an eight-turn conversation about a single topic, you may have paid for the same passages eight times. Pin retrieved context for the topic and re-retrieve only when the query moves, detectable by an embedding-distance check against the previous query rather than a fresh search on every turn.

The second is quadratic history growth. If you resend the whole conversation and each turn averages 500 tokens, turn N carries all N-1 prior turns, so the tokens processed across a conversation of N turns is:

sum over t of (t x 500) = 500 x N x (N + 1) / 2

N = 10  ->    27,500 token-reads
N = 100 -> 2,525,000 token-reads

Ten times the conversation, about ninety times the cost. Your most engaged users are not slightly more expensive than casual ones; they are two orders of magnitude more expensive, and they are exactly the users you want more of. The fix is the same one that fixes memory generally: keep the recent turns verbatim, summarize the rest, and keep long-lived facts in a store you retrieve from rather than a transcript you resend.

An agent loop pays for the whole transcript once per step

The most expensive pattern of all is an agent that reasons in a loop, because it combines every problem above and multiplies them by the step count.

Each step of an agent loop is a full model call whose input is the system prompt, the tool definitions, the original task, and every previous step's reasoning and tool output. Step 1 sends the base context. Step 6 sends the base context plus five rounds of accumulated transcript, including the raw tool results, which for a search or a database query can be large. A single user request that resolves in six steps is not six times the cost of a one-shot call. It is closer to fifteen or twenty times, and if the agent gets stuck in a retry loop there is no natural ceiling at all.

What controls it:

• A hard step budget per request, enforced in code, that fails visibly rather than looping. Unbounded is not a design.
• Tool outputs summarized or truncated before they enter the transcript. A 40,000-token API response does not need to be re-read on every subsequent step.
• Loop detection: if the last two steps proposed the same tool call with the same arguments, stop. It is not going to work the third time.
• The state-machine question. For a workflow whose steps you can enumerate in advance, an agent is a nondeterministic and expensive way to run code you could have written directly. Autonomy earns its cost when the path genuinely varies; when it does not, it is a bill with extra steps.

Set a cost-per-request alert, not just a monthly budget alert. A monthly budget tells you about the loop bug three weeks after it shipped.

Work the levers in order of magnitude, not in order of effort

Everything above, in the order a team should attack it:

Lever	Typical magnitude	Risk to quality
Measure the input / output split	None; it sizes everything else	None
Trim assembled context	Large, often 2x or more	Real; gate on an eval set
Prompt caching	Large on stable prefixes	Low; it is the same tokens
Model routing	Large on simple-heavy traffic	Highest; needs eval per route
Fix re-retrieval and history	Large on multi-turn traffic	Low to moderate
Bound agent loops	Removes the tail risk	Low; it caps failure modes
Negotiate the rate card	Percentages	None

Rate negotiation is last on the list, not because it is worthless, but because it is the only item on it that does not compound with the others. A 20 percent discount applied to a system that sends three times the context it needs is a 20 percent discount on waste.

None of the multipliers in this article are your numbers. They are arithmetic you can rerun with your own request volume, prompt size, traffic mix, and provider rates, and the exercise takes an afternoon. The reason to do it before the next optimization sprint is that it will almost certainly tell you the sprint is aimed at the wrong lever.

Cost is an output of the architecture, so treat it as a design review

The reframe is simple. Token spend is not a line item that gets managed by finance after the fact. It is an emergent property of decisions made in context assembly, prompt layout, model selection, retrieval policy, and control flow - all of which are engineering decisions, all of which are reversible, and none of which appear on an invoice in a form you can read.

So put cost in the design review. When someone proposes adding a retrieval step, ask what it adds to the prompt on every call and whether the result is cached. When someone adds a tool, ask which code paths now carry its schema. When someone proposes an agent, ask for the step budget. Track cost per successful request as a product metric next to latency and pass rate, and watch it per release rather than per invoice.

Teams that do this stop having the quarterly conversation about whether AI is too expensive to run. The systems that are too expensive to run are usually not expensive because of the model. They are expensive because nobody ever decided what to send it.