A FinOps Loop for AI Workloads That Survives Past the First Cleanup

Most AI cost programs are one-time cleanup projects. Someone gets paged by the invoice, a team spends two weeks deleting idle GPUs and capping output tokens, the bill drops 30%, and two quarters later it is back. Costs drift because the system that generates them changes every week: new features, new models, new prompts, new retries. What works is not a project but a loop. This is the one I run: where the money actually goes, the levers in order, anomaly detection on billing data, and budgets as code. I built the automated half of it as CostSentinel, a demo you can poke at (finops.arielsofi.com).

Where the money actually goes

Decompose the bill before touching anything. For API-based workloads:

cost = sum over models of (input tokens x input price) + (output tokens x output price)

Output tokens typically run 3-5x the input price, so anything that shortens responses pays disproportionately. For self-hosted models the shape changes: GPU hours times utilization, and utilization is where the bodies are buried. A 70B model serving four requests a minute on two 80GB GPUs is a very expensive space heater.

Then the spend nobody forecasts: dev and staging environments that mirror prod's model calls, eval suites that re-run on every merge, retry storms during provider incidents, embedding backfills, and abandoned fine-tuning endpoints still billed by the hour. In every review I have done, at least one of these was material.

The levers, ranked

Honest ranges, assuming you actually measure. Your traffic decides where you land.

Lever	Typical savings	Effort	Main risk
Prompt discipline	10-30% of total spend	Days	Output truncation bugs
Prompt and semantic caching	20-50% of input spend	Days to weeks	Stale cached context
Model routing	30-60% of model spend	Weeks, needs evals	Silent quality regression
Quantization	40-60% of serving infra	Weeks, self-hosted only	Accuracy loss on hard tasks
Spot capacity	60-90% on eligible batch	Weeks, self-hosted only	Mid-job interruptions

Ranked by effort-adjusted payback: prompt discipline and caching first, routing next with an eval gate, quantization and spot only when the bill or sovereignty requirements justify self-hosting. The full version with sizing math lives in the LLM cost levers cheatsheet.

Two cautions. Routing without a per-task eval gate is how you learn about quality regressions from customer tickets. And below some bill size, optimization does not cover the engineering: at $5K a month, a heroic 40% saving is $2K. Do the breakeven before the work.

The weekly loop

Thirty to sixty minutes, every week, same time.

1. Reconcile. The billing export lands daily; the first job is confirming it arrived, parsed, and matches the provider console. A FinOps loop running on stale data is worse than none, because it looks informed.
2. Attribute. Cost per model, per feature, per environment, per tenant where relevant. This needs cost-per-request telemetry in the gateway, not just the invoice. If you cannot state cost per successful request by feature today, that is the only project this month.
3. Detect anomalies. Run detection over per-service daily cost series (next section).
4. Act. Every finding becomes a ticket with an owner and an expected saving. No owner, no finding; it goes in the notebook as context, not into the tracker as noise.
5. Verify. Next week's numbers confirm the fix moved the bill or they do not. Optimization work that is never re-measured is a rumor.

Anomaly detection on billing data

Billing data is a hostile time series: strong weekly seasonality, step changes from deploys, heavy tails from batch jobs. Fancy forecasting models miss the point. The job is to answer one question: did anything move more than its history says it should, and where.

What works: per-service daily series, a robust baseline (rolling median with median absolute deviation, or an IQR fence), computed separately for weekdays and weekends. Flag a service-day when cost lands outside the fence. Robust statistics matter because one embedding backfill should not permanently widen your thresholds, which is what mean-and-standard-deviation methods do.

Then the LLM does what it is good at. For each flagged anomaly, the system hands the model the anomaly record, recent deploy events, and that service's usage metrics, and gets back a short diagnosis: what probably changed, what to check first. The detection is deterministic; the LLM never sees the raw billing dump. This keeps both cost and hallucination down, and it means a wrong narrative never changes which anomalies get flagged.

This is the CostSentinel pipeline: a Temporal-scheduled daily workflow pulls billing and usage data, runs the detector, fans out LLM analysis for the top anomalies, and posts to Slack with links into a small dashboard. Temporal matters here because the pipeline is a dozen flaky API calls strung together; durable retries and a schedule the server owns remove the two classic ways this kind of job silently dies. The project writeup has the architecture.

Budgets as code

Budgets that live in a console get edited by whoever is logged in during the incident. Budgets in Terraform get reviewed.

Per account or project, define budgets with alerts at 50%, 80%, 100%, and 120% of the monthly figure, wired to the channel the team actually reads. The interesting part is deriving the figure: expected volume times measured unit cost, per feature. When a feature's cost per successful request is known, its budget is arithmetic, and a breach has an interpretation (volume grew, unit cost grew, or both) instead of just a red bar.

Commit the budgets next to the infrastructure they cover. When a PR doubles expected request volume, the budget diff shows up in the same review.

What does not work

Monthly invoice review: too slow, thirty days of drift before anyone looks. Account-level alerting: a 40% spike in one service hides inside a quiet account. Optimizing before measuring: the levers above are useless without attribution, because you cannot verify them. And trusting provider dashboards for attribution: they answer "what did we spend," never "which feature spent it."

The loop is deliberately boring. That is the point. Cost control that depends on heroics fails the first busy quarter; cost control that runs on a schedule just runs.