LLM Cost Levers, Ranked by Typical Impact

The formula

cost = sum over models of (input tokens x input price) + (output tokens x output price)

Every lever shrinks one of those four terms. Output tokens typically cost 3-5x input tokens, so output-side cuts pay disproportionately.

The levers, ranked

Rank	Lever	Typical savings	Applies to	Prerequisite	Main risk
1	Model routing	30-60% of model spend	Anyone on premium models	Eval gate per task type	Silent quality regression
2	Prompt and semantic caching	20-50% of input spend	Repeated prefixes, RAG, multi-turn	Cache-aware prompt layout	Stale or leaked cached context
3	Prompt discipline	10-30% of total spend	Everyone	Nothing	Output truncation bugs
4	Quantization	40-60% of serving infra	Self-hosted only	Quality eval on your tasks	Accuracy loss on hard tasks
5	Spot capacity	60-90% on eligible batch	Self-hosted batch only	Checkpointing and retries	Mid-job interruptions

Ranges are synthesized from published sources and assessment work; your traffic distribution decides where you land. Order by effort-adjusted payback: 3 and 2 this quarter, 1 next quarter, 4 and 5 when self-hosting is already justified.

Prompt discipline (do this first)

• Cap max_tokens per feature. An output cap is often worth more than any input optimization.
• Trim chat history to the turns that matter; stop resending resolved context.
• Compress system prompts. Every line costs on every call.
• Cache reference documents instead of inlining them per request.
• Measure p50 and p95 output length per feature before and after. Typical outcome: 10-30% of total spend for a week of work.

Caching facts (as of mid-2026; verify against current provider docs)

Provider	Mechanism	Pricing shape
Anthropic	Explicit cache control on prompt blocks	Cache reads around 10% of base input price; writes carry a premium; default TTL 5 minutes
OpenAI	Automatic prefix caching	Cached input tokens discounted roughly 50-90% depending on model; best-effort, so measure hit rate before forecasting

Layout rule: stable content first (system prompt, tool definitions, few-shot examples), variable content last. Caches match prefixes; a volatile timestamp at the top of the prompt zeroes the lever.

Routing table template

Task type	Quality bar	Tier	Why
Classification, extraction	Deterministic eval	Small/cheap	10-15x cheaper per token, clears eval
Summarization, FAQ	Eval plus spot checks	Mid	Quality insensitive at this length
Client-facing drafting, analysis	Human rubric, sampled	Frontier	Quality is the product
Regulated data (any task)	Policy	Approved tiers only	Routing is a data-governance artifact

Rules: routing lives in the gateway, not in app code; every tier change passes an eval gate; the fallback chain is configured and has carried traffic at least once.

When fine-tuning beats prompting on cost

Breakeven: fine-tune when monthly prompt-overhead cost exceeds monthly fine-tune cost.

monthly prompt overhead = extra few-shot tokens per call x calls per month x token price

monthly fine-tune cost = training amortized over model lifetime + any serving premium

Fine-tuning tends to win when all of these hold:

• One stable task shape (classification, extraction to a fixed schema, a house style).
• High, steady call volume on that shape.
• A few-shot prompt long enough that its tokens dominate the call.
• A maintainable training set in the hundreds to low thousands of good examples.

Prompting wins when the task changes weekly, volume is low, or the dataset cannot be maintained. The end state is a fine-tuned small model behind the same gateway interface: the routing rule changes, the app code does not.

Quick checks

• Cost per successful request by feature: measured, not estimated.
• p50 and p95 output length per feature; caps set.
• Cache hit rate measured before savings are claimed.
• Eval gate runs before any routing change ships.
• The bill reconciles with attribution data weekly.