Why Your AI Bill Keeps Climbing

The first serious AI invoice arrives and the response is almost always procurement. Someone asks whether there is a committed-use discount. Someone else prices the same workload at a competing provider and finds it is 20 percent cheaper on input. A third person suggests capping usage per customer. All three are answers to the question "how much are we paying per token," and none of them is the question that matters.

The bill is a function of two things: the rate you pay per token, and the number of tokens you send and receive. Rates and discounts matter, but the amount of context sent on each request is also under your control. Before negotiating a price, check whether the system repeatedly sends instructions, documents, or conversation history that do not improve the answer.

Start by finding out which half of the bill you are arguing about

Before touching anything, split the spend. Input tokens and output tokens are priced differently, usually with output several times more expensive per token, and teams reason about cost as if output dominates because output is the part they can see. In some retrieval-heavy or agent workflows, input becomes the larger share even when output costs more per token.

Here is an illustrative worked example. It uses round numbers so you can redo it with your own; nothing in it is a measured Abisam result.

Take an assistant handling 50,000 requests a day. Each request sends a 12,000-token prompt (system instructions, tool definitions, retrieved documents, conversation history) and returns a 600-token answer. Assume a frontier model at 3 dollars per million input tokens and 15 dollars per million output tokens.

input  = 50,000 x 12,000 = 600,000,000 tokens/day
       = 600 M x $3 / 1M  = $1,800 / day

output = 50,000 x    600 =  30,000,000 tokens/day
       =  30 M x $15 / 1M =   $450 / day

total  = $2,250 / day, about $67,500 over 30 days

Input is 80 percent of that bill, and output is the part everyone was trying to shorten. In this example, shortening answers addresses only a fifth of the bill. The larger opportunity is checking what went into the prompt.

Do this split first. Every lever below is worth a different amount depending on which side of it you are standing on, and you cannot prioritize levers you have not sized.

Context you did not choose is context you still pay for

Ask what is actually in those 12,000 tokens. In systems that have not been audited, the answer is usually some version of this:

A system prompt that has been appended to for a year and never pruned, including instructions for features that shipped and were removed.
Full JSON schemas for every tool the agent could theoretically call, including the twelve it never calls in this code path.
The top eight retrieved chunks, because eight was the default in the tutorial, at whatever chunk size the ingestion pipeline used.
The entire conversation history verbatim, because truncation felt risky.
A few thousand tokens of few-shot examples added during a quality push in month two, kept because nobody wanted to be the person who removed them.

None of these is unreasonable on its own. Together they are the bill. The discipline is to make each one a measured decision rather than an accumulated default.

Concretely: retrieve eight chunks and rerank down to three, rather than sending eight and hoping. Send only the tool definitions reachable from the current state. Summarize conversation turns older than the last few instead of resending them verbatim. Cut few-shot examples one at a time and watch your eval pass rate rather than assuming they earn their tokens.

Suppose that work takes the prompt from 12,000 tokens to 5,000.

input  = 50,000 x 5,000 = 250,000,000 tokens/day
       = 250 M x $3 / 1M = $750 / day

total  = $750 + $450 = $1,200 / day, about $36,000 over 30 days

That is a 47 percent reduction in total spend from one lever, before a single conversation with a vendor. Compare it to the 20 percent input discount someone was chasing, which on the original prompt would have saved 360 dollars a day, and note that the two are not exclusive. The point is the order of magnitude, and which one you should spend your quarter on.

Check the quality before keeping the change. Cut context against a frozen eval set and read the pass rate. If quality holds on representative cases, those tokens may not have been needed for that workload. If it drops, you have learned something specific about what the model actually needed, which is more useful than the tokens you saved.

Every stable prefix you resend uncached is a discount you declined

Some providers offer lower prices for reused prompt content. The rules, minimum size, and discount vary, so check your provider's terms before planning around a cache. The mechanism is prefix-based, which is the part teams miss. In prefix-based systems, early changes can reduce how much of the prompt is reused.

So the ordering of your prompt is a cost decision. Put the stable material first - system instructions, tool definitions, policy text, long-lived few-shot examples - and the volatile material last: the retrieved chunks, the user's turn, the timestamp. Teams routinely destroy their own cache hit rate by interpolating a current date, a request id, or a per-user greeting into the top of the system prompt. One variable token at position 40 makes the next 4,000 tokens uncacheable.

Continue the example. Of the trimmed 5,000-token prompt, say 4,000 is a stable prefix and 1,000 varies per request. Assume cached input reads at 0.30 dollars per million, one tenth of the standard rate.

cached prefix  = 50,000 x 4,000 = 200 M x $0.30 / 1M = $60 / day
variable tail  = 50,000 x 1,000 =  50 M x $3.00 / 1M = $150 / day
input          = $210 / day   (was $750)

total          = $210 + $450 = $660 / day, about $19,800 over 30 days

Two design changes, no model change, no vendor negotiation, and the monthly figure has moved from about 67,500 dollars to about 19,800. Use your provider's actual cache rate, minimum cacheable prefix length, and cache lifetime when you redo this; they differ, and some providers charge a small premium to write the cache entry, which matters if your traffic is too sparse to get reads out of it.

Instrument the hit rate. Measure cache hits so you can tell whether the change actually reduces spend.

One model for every call is a routing decision you made by not making it

Almost every AI product starts by pointing everything at the strongest available model, which is correct: you are trying to find out whether the product works at all, and a capability question should not be confounded by a cost optimization. The mistake is leaving it there once the traffic mix is known.

Look at what the calls actually are. In a typical assistant, a large share of the volume is not the hard reasoning task the product is named after. It is classification (is this question in scope), extraction (pull the order number out of this sentence), routing (which tool applies), short factual lookup, and summarization of retrieved text. A smaller model may handle some of these tasks well at a lower cost. Test each route against your quality standard before changing traffic.

The arithmetic, again illustratively. Say 70 percent of requests are these simple calls and a small model prices at 0.30 dollars per million input and 1.50 per million output, one tenth of the frontier rates. Take the post-caching per-day figure of 660 dollars and split the traffic:

30% on the frontier model: 0.30 x $660 = $198 / day
70% on the small model:    0.70 x $660 x 0.10 = $46 / day

total = $244 / day, about $7,300 over 30 days

The honest caveat: routing is the lever most likely to cost you quality if you do it by feel. It needs a classifier you trust, an eval set per route, and an escalation path where a low-confidence small-model answer gets retried on the strong model. That retry has a cost too, and if your escalation rate is high enough the routing saves nothing. Measure the escalation rate as a first-class metric. It is also the lever that tells you the most about your product: a route that escalates constantly is a task you have misunderstood.

Retrieval that runs every turn re-buys documents you already have

Two related patterns, both common, both invisible in a demo.

The first is re-retrieval within a conversation. A user asks about a policy document, you retrieve four chunks, the model answers. The user asks a follow-up. Some implementations search again, find largely the same chunks, and pay to send them again. Over an eight-turn conversation on one topic, those passages may be repeated many times. Consider reusing relevant context for follow-ups, and test when fresh retrieval is needed so the answer does not rely on stale information.

The second is quadratic history growth. If you resend the whole conversation and each turn averages 500 tokens, turn N carries all N-1 prior turns, so the tokens processed across a conversation of N turns is:

sum over t of (t x 500) = 500 x N x (N + 1) / 2

N = 10  ->    27,500 token-reads
N = 100 -> 2,525,000 token-reads

Ten times the conversation, about ninety times the cost. Your most engaged users are not slightly more expensive than casual ones; they are two orders of magnitude more expensive, and they are exactly the users you want more of. The fix is the same one that fixes memory generally: keep the recent turns verbatim, summarize the rest, and keep long-lived facts in a store you retrieve from rather than a transcript you resend.

An agent loop pays for the whole transcript once per step

Agent loops can multiply cost because each step adds another model call, often with more context than the last.

Each step of an agent loop is a full model call whose input is the system prompt, the tool definitions, the original task, and every previous step's reasoning and tool output. Step 1 sends the base context. Step 6 sends the base context plus five rounds of accumulated transcript, including the raw tool results, which for a search or a database query can be large. A six-step request can cost considerably more than six one-shot calls if each step resends accumulated context. A retry loop can push that further without a step limit.

What controls it:

A hard step budget per request, enforced in code, that fails visibly rather than looping. Unbounded is not a design.
Tool outputs summarized or truncated before they enter the transcript. A 40,000-token API response does not need to be re-read on every subsequent step.
Loop detection: if the last two steps proposed the same tool call with the same arguments, stop. It is not going to work the third time.
The state-machine question. For a workflow whose steps you can enumerate in advance, an agent is a nondeterministic and expensive way to run code you could have written directly. Autonomy earns its cost when the path genuinely varies; when it does not, it is a bill with extra steps.

Set a cost-per-request alert, not just a monthly budget alert. A monthly budget tells you about the loop bug three weeks after it shipped.

Work the levers in order of magnitude, not in order of effort

Everything above, in the order a team should attack it:

Lever	Typical magnitude	Risk to quality
Measure the input / output split	None; it sizes everything else	None
Trim assembled context	Large, often 2x or more	Real; gate on an eval set
Prompt caching	Large on stable prefixes	Low; it is the same tokens
Model routing	Large on simple-heavy traffic	Highest; needs eval per route
Fix re-retrieval and history	Large on multi-turn traffic	Low to moderate
Bound agent loops	Removes the tail risk	Low; it caps failure modes
Negotiate the rate card	Percentages	None

Rate negotiation is last on the list, not because it is worthless, but because it is the only item on it that does not compound with the others. A 20 percent discount applied to a system that sends three times the context it needs is a 20 percent discount on waste.

None of the multipliers in this article are your numbers. They are arithmetic you can rerun with your own request volume, prompt size, traffic mix, and provider rates, and the exercise takes an afternoon. The reason to do it before the next optimization sprint is that it may change which improvement you tackle first.

Cost is an output of the architecture, so treat it as a design review

The reframe is simple. Token spend is not a line item that gets managed by finance after the fact. It is an emergent property of decisions made in context assembly, prompt layout, model selection, retrieval policy, and control flow - all of which are engineering decisions, all of which are reversible, and none of which appear on an invoice in a form you can read.

So put cost in the design review. When someone proposes adding a retrieval step, ask what it adds to the prompt on every call and whether the result is cached. When someone adds a tool, ask which code paths now carry its schema. When someone proposes an agent, ask for the step budget. Track cost per successful request as a product metric next to latency and pass rate, and watch it per release rather than per invoice.

Teams that do this stop having the quarterly conversation about whether AI is too expensive to run. For some systems the largest saving is a vendor or model change. For others it is a simpler request that sends only what the model needs. Measure both before deciding.