Short answer: your AI bill usually grows faster than your usage because every request carries more than you think: long system prompts, full conversation histories, retrieved documents, multi-step agent loops, retries, and the most capable model doing even the simplest steps. LLM cost optimization comes down to sending less, sending it to a cheaper model when you can, reusing what you've already paid for, and measuring cost per completed outcome rather than cost per call.

This post explains where model spend actually comes from, the techniques that reduce it (model routing, prompt caching, context trimming, batching, output limits), the order to apply them in, and the cuts that save money on paper but cost you more in errors.

Where does the money in an AI bill actually go?

Most model APIs charge per token, a chunk of text roughly the size of a short word. You pay for tokens going in (your instructions, the conversation, any documents) and tokens coming out (the response). Output tokens usually cost more per token than input tokens, and larger models cost more than smaller ones, often by a wide margin.

The trap is that input dominates in most business systems, and it's invisible. A customer types one sentence. Behind it, your system sends:

  • A system prompt with role, rules, tone and examples
  • The full conversation so far
  • Several retrieved policy or product passages
  • Tool definitions describing every tool the agent can use
  • Results from earlier tool calls

Then an agent may make several model calls to finish one task: plan, call a tool, read the result, call another, draft, check. Each call resends most of that context. A task that looks like "one question, one answer" can quietly be several large requests.

Why does the bill grow faster than usage?

Usage growth explains some of it. These explain the rest:

Conversation length compounds. If each turn resends the whole history, a long conversation costs far more than several short ones, because later turns carry everything before them.

Prompts accrete. Every time someone patches a failure, the system prompt gets another paragraph. Six months later, it's several times its original length, and every call pays for it.

Retrieval is generous. Pulling ten document chunks "to be safe" when two would do multiplies input on every request.

Agents loop. An agent that retries a failing tool, or keeps "thinking" without a step limit, can burn through a surprising amount on a single stuck task.

One model does everything. The largest model classifies emails, extracts dates and writes greetings, work a much smaller model could do.

Nobody owns the number. Spend shows up as one monthly invoice, with no breakdown by feature, workflow or customer, so nobody can see what changed.

Which LLM cost optimization techniques work best?

Here are the main levers, roughly in the order we'd apply them.

TechniqueWhat it doesTypical effortWatch out for
Measure cost per taskTags every call by workflow and outcomeLowWithout it, you're guessing
Context trimmingSends only what the step needs: summarize history, fewer chunks, fewer toolsLow to mediumCutting context the model actually needed
Prompt cachingReuses a stable prompt prefix at a discounted rate where providers support itLowOnly helps if the start of the prompt stays identical
Output limitsCaps response length; asks for structured, concise outputLowTruncated answers if limits are too tight
Model routingSends simple steps to small, cheap models; hard ones to larger modelsMediumNeeds evaluation to prove the small model is good enough
BatchingRuns non-urgent work through discounted asynchronous batch endpointsLow to mediumResults arrive later; only for work that can wait
Response cachingStores answers to repeated identical questionsMediumStale answers if underlying data changes
Agent budgetsHard limits on steps, tokens and retries per taskLowEscalate to a human when hit, don't fail silently
Fine-tuning a smaller modelTrains a small model for a narrow, high-volume taskHighOnly worth it at high, steady volume on a stable task

Let's take the most effective ones in turn.

Model routing: stop using your biggest model for small jobs

Most agent workflows are a mix of easy and hard steps. Classifying an email as "invoice, query or spam", extracting a date, checking whether a reply mentions a refund: these are jobs for a small, fast model. Drafting a careful reply to an angry customer or reasoning over a messy contract may need a larger one.

Routing can be static (this step always uses the small model) or dynamic (a cheap model attempts the task and escalates to a larger one when its confidence is low or validation fails). Static routing is simpler and usually captures most of the savings. Our post on small language models for business looks at what smaller models handle well.

The rule: never downgrade a step without running your test set against the cheaper model first. A model that's cheaper per call but wrong more often costs more per correct outcome.

Prompt caching: pay less for the parts that never change

Several providers offer prompt caching: if the beginning of your prompt is identical across requests, the cached portion is billed at a reduced rate and often processed faster. Structure prompts so stable content comes first (system instructions, tool definitions, reference material) and variable content comes last (the user's message, the specific record). Avoid putting timestamps or user names at the top, which break the cache on every call.

Context trimming: send less

  • Summarize older conversation turns instead of resending them verbatim.
  • Retrieve fewer, better chunks. Improving search quality often lets you send less context and get better answers.
  • Only include the tool definitions relevant to the current step.
  • Strip boilerplate from documents before they reach the model: email signatures, legal footers, repeated headers.

Batching: let non-urgent work wait

Overnight report generation, bulk document classification, back-catalog product descriptions and weekly summaries don't need instant answers. Several providers offer discounted batch processing for requests that can complete within hours. Moving that work to batch is one of the simplest savings available.

What does a cost review look like in practice?

For illustration, picture a support agent that handles order queries for an online retailer. The monthly bill has crept up and nobody knows why. A review might go like this:

  1. Tag and measure. Add workflow and outcome tags to every call. After a week, cost per resolved conversation is visible, along with the most expensive conversations.
  2. Read the expensive ones. The costliest runs turn out to be long conversations where the agent resent the full history each turn, plus a handful of loops where an order-lookup tool kept timing out and the agent kept retrying.
  3. Fix the loops first. Add a retry limit and a step budget, with escalation to a human when hit.
  4. Trim context. Summarize history after a few turns; cut retrieved passages from ten to three after testing that answers don't get worse.
  5. Restructure for caching. Move the long policy instructions and tool definitions to a stable prefix.
  6. Route. Use a small model for intent classification and order-status replies; keep the larger model for complaints and complex returns.
  7. Re-measure. Compare cost per resolved conversation and resolution quality before and after.

The order matters. Fix waste and loops before switching models, because waste scales with whatever model you use.

Why is cost per outcome the metric that matters?

Cost per token and cost per call are easy to measure and easy to misread. A cheaper call that fails more often leads to retries, escalations and human rework, all of which cost money too.

Measure cost per completed task: total model spend for a workflow divided by tasks successfully completed. Better still, add the human time spent reviewing and fixing, so you see the full cost per outcome. That number lets you compare a cheap model with a high escalation rate against a pricier one that gets it right first time. It also ties AI spend to business value, which is what your finance team actually wants to know. See how to measure AI ROI and what an AI system really costs after it's built for the wider picture.

Which cost cuts backfire?

  • Downgrading models without testing. Quality drops quietly, escalations rise, and customers notice before your dashboard does.
  • Trimming context that mattered. The agent stops seeing the return policy clause it needed and starts improvising.
  • Caching answers to questions about live data. A cached "your order has shipped" is worse than no answer.
  • Cutting human review to save time. Review is where errors are caught cheaply. Removing it moves the cost to customer complaints.
  • Self-hosting purely to save money. At modest volumes, running your own models often costs more once infrastructure and engineering time are counted. That's a separate decision, covered in open-source LLMs vs API models.

How do you start getting your AI costs under control?

This week: tag every model call with the workflow it belongs to, set a monthly budget alert with your provider, and add step and retry limits to every agent. Next, pull the twenty most expensive runs and read them. The causes are usually obvious once you look, and the first fixes, loops and bloated context, rarely need any model change at all.

Frequently Asked Questions

Why is my OpenAI or Claude API bill so high?

Usually because each request carries far more input than the user's message: long system prompts, full chat histories, retrieved documents and tool definitions, repeated across several calls per task. Agent loops and retries add more. Tag calls by workflow and inspect the most expensive runs to find the cause.

Does prompt caching reduce LLM costs?

Yes, where your provider supports it and your prompts share a stable prefix. Put fixed instructions, tool definitions and reference material at the start of the prompt and variable content at the end. Savings depend on how much of each request is repeated.

Are smaller AI models good enough for business tasks?

For narrow, well-defined steps like classification, extraction and short templated replies, often yes. For complex reasoning or sensitive customer conversations, larger models may still be needed. Test on your own examples before switching, and compare cost per correct outcome rather than cost per call.

How do I set a budget for AI usage?

Set spend alerts and hard limits with your provider, add per-task limits on steps and tokens in your agent code, and track spend by workflow so you can see which feature is driving costs. Review the numbers monthly alongside the business outcomes they deliver.

Is it cheaper to run open-source models ourselves?

Sometimes, at high and steady volume with a team that can operate the infrastructure. At low or spiky volume, API models are usually cheaper once you count GPUs, engineering time and maintenance. Model the full cost, not just per-token price.

Want a second look at your AI spend?

If your AI bill is climbing and you can't see why, send us a description of the workflows behind it. We'll help you find where the spend goes and which fixes are worth making first. Here's how we approach agentic AI development.