Reduce LLM costs: caching, model choice and limits (2026)

Hephaestus Technology7 min read

To reduce LLM costs, cut the tokens you send and the price you pay per token: cache the parts of the prompt that repeat, use the smallest model that passes your evaluation, send non-urgent work through batch APIs at half price, and put hard limits on usage. On the price lists published on 8 November 2026, cached input costs a small fraction of normal input and batch processing is 50% off at Anthropic, OpenAI and Google. The seven steps below are ordered by how much they typically save for the effort involved.

Key takeaways

  • Measure cost per task first; you cannot reduce what you have not attributed to a feature, customer or step.
  • Prompt caching rewards stable prompts: put instructions, tool definitions and reference material first and anything that changes last.
  • Model choice is the largest lever, with roughly a 100x spread in per-token prices between the smallest and largest models on current price lists.
  • Batch APIs are 50% cheaper at the three major providers for work that does not need an instant answer.
  • Hard limits on spend, tokens, requests and agent steps turn unpredictable bills into predictable ones and protect against abuse.

Where does LLM spend actually come from?

LLM spend is input tokens, output tokens and tool fees, multiplied by volume. Output tokens usually cost around five times as much as input tokens on current price lists; for example, both Anthropic and OpenAI list mid-tier models at $2 per million input tokens and $10 per million output tokens. But input volume is often larger, because every request resends instructions, history, retrieved documents and tool definitions. Agents multiply this, since each step is another request. Our breakdown of the cost to build an AI agent includes a worked example.

Step 1: How do you measure cost per task?

Log the token counts the API returns for every request, tagged by feature, customer and step. All major APIs report input, output and cached tokens in the response. From that you can calculate cost per conversation, per document or per customer, and spot the steps worth optimising. Without this, teams tend to optimise the wrong thing.

Step 2: How does prompt caching cut costs?

Prompt caching lets the provider reuse the processed start of a prompt that it has seen recently, billing those tokens at a heavily reduced rate. Prices and mechanics differ by provider:

Provider How it works Cached input price Notes
Anthropic Automatic or explicit cache_control breakpoints 0.1x base input on most models; 0.05x on Opus 5.5 and Sonnet 5.5 Five-minute writes cost 1.25x, one-hour writes 2x
OpenAI Enabled by default for supported models Discounted up to 95% Minimum 1,024 tokens for GPT-5.6 and later
Google Context caching Listed per model on the pricing page Some models add an hourly storage charge

Anthropic notes that with five-minute caching, caching pays off after one cache read. Its minimum cacheable length is 512 tokens on its newest models and up to 4,096 on some older ones, and shorter prompts are processed without caching with no error returned.

The rule that makes caching work is ordering. Both Anthropic and OpenAI advise putting stable content first: OpenAI says to put stable developer instructions and shared reference material first, and Anthropic warns that a cache hit requires 100 percent identical prompt segments up to the breakpoint. A timestamp or user name at the top of your system prompt silently breaks caching for every request.

Step 3: How do you choose the cheapest model that works?

Test smaller models on your evaluation set and use the smallest one that meets your thresholds. On 8 November 2026, Anthropic's Claude Haiku 5.5 and OpenAI's gpt-6-luna were both listed at $0.10 per million input tokens and $0.50 output, against $10 and $50 for OpenAI's gpt-6-astra. That spread only matters if the small model is accurate enough, which is why this step depends on a good LLM evaluation set.

Two common patterns:

  • Routing: a small model handles classification, extraction and simple answers; only hard cases go to a larger model.
  • Splitting steps: in an agent or workflow, use a small model for routine steps (summarising tool output, formatting) and a larger one for planning.

Compare cost per task, not per token. Tokenizers differ: Anthropic states that its newer models produce approximately 30% more tokens for the same text than its older tokenizer.

Step 4: When should you use batch processing?

Use batch APIs for any work that can wait, such as nightly summaries, document backfills, evaluation runs and bulk classification. Anthropic offers a 50 percent discount on both input and output tokens through its Batch API, Google describes a 50% cost reduction, and OpenAI's batch prices are half its standard prices on the same pricing page. Anthropic also notes that caching and batch discounts can be combined.

Step 5: How do you send fewer tokens?

Trim what goes into each request:

  • Retrieve less, better. Send the three most relevant passages, not the top twenty.
  • Summarise long histories instead of resending a full conversation every turn.
  • Cap output length with a sensible maximum and ask for concise formats, such as JSON fields instead of prose.
  • Load tools selectively. Every tool definition is input tokens on every call; expose only the tools a step needs.
  • Watch prompt-length price tiers. Some models price long prompts higher; Anthropic's Haiku 5.5, for instance, is listed at $0.50 instead of $0.10 per million input tokens for prompts over 100,000 tokens.

Step 6: What limits should you set?

Set limits at every level, so a bug, a loop or an abusive user cannot run up a large bill. The OWASP Top 10 for LLM applications lists unbounded consumption as a risk for exactly this reason.

  1. Monthly spend limits and alerts in each provider's console.
  2. Requests per user per minute and per day, tied to their plan.
  3. Maximum input and output tokens per request.
  4. For agents, a maximum number of steps per task; Anthropic recommends stopping conditions such as a maximum number of iterations.
  5. A per-customer monthly budget that triggers a downgrade or a warning, which also helps when designing SaaS pricing models for AI features.

Step 7: How do you keep costs down over time?

Review cost per task monthly and price lists quarterly. Providers change prices and retire models regularly: on 8 November 2026, Google's pricing page listed scheduled increases for gemini-4-argon on 1 December 2026 and gemini-3.8-flash on 1 January 2027. New, cheaper models also appear often, so rerun your evaluation against them and switch when one passes.

What is the checklist?

  • [ ] Token usage logged per feature, customer and step
  • [ ] Stable content first in every prompt, dynamic content last
  • [ ] Cache hit rate checked in API usage fields
  • [ ] Smallest passing model chosen per step, verified on your evaluation set
  • [ ] Non-urgent jobs moved to batch APIs
  • [ ] Retrieval, history and tool lists trimmed
  • [ ] Output length capped
  • [ ] Spend, rate, token and step limits enforced
  • [ ] Price list and model retirement review in the calendar

Frequently asked questions

How much can prompt caching save?

It depends on how much of each prompt repeats. Cached input is billed at 10% of the normal input price on most Anthropic models and is discounted up to 95% at OpenAI, so an agent that resends long instructions on every step can save more than half its input bill. Check the cache fields in API responses to confirm hits.

Is it worth switching to a cheaper model?

Only if it passes your evaluation set. Smaller models can be one twentieth of the per-token price of mid-tier models, but more errors, retries or escalations to people can wipe out the saving. Measure cost per successful task.

Does the Batch API work for chat features?

No. Batch processing is asynchronous, so it suits background jobs such as summaries, classification and evaluation runs. Interactive features need standard requests, where caching and model choice are the main levers.

Should I self-host an open model to save money?

Sometimes, at high and steady volume, but compare the full cost: GPU hosting, engineering time, monitoring and lower utilisation at quiet times. For most early-stage products, API pricing with caching and small models is cheaper and simpler.

How do I pass AI costs on to customers?

Measure cost per customer first, then choose between including a usage allowance in each plan, charging per use, or offering an AI add-on. Per-customer limits protect your margin whichever model you pick.

If your LLM bill is growing faster than your revenue, book a free 30-minute call and we will look at it with you.

Cover photo by Yan Krukau on Pexels.

Hephaestus Technology

We help founders and businesses validate, build and launch products across software, AI and hardware, from offices in Ho Chi Minh City, Dublin and Delaware. See how we work.

Tell us what
you're building.

A free 30-minute call. You'll get an honest view of the idea, the risks we see and what we'd do first, whether or not we end up working together.