CostSentinel

The problem

Cloud and AI bills move silently. A retry loop against a degraded provider, an embedding backfill someone forgot to scope, a new feature whose output tokens run 3x the estimate: each is invisible until the invoice, weeks later. Most teams have budgets and alerts at the account level, which is exactly the wrong resolution. A 40% spike in one service hides inside a quiet account.

Constraints

• Read-only against billing APIs. The system must never become a way to mutate infrastructure.
• No agents installed in workloads. Billing exports and usage APIs only.
• The pipeline itself must be cheap; a cost tool with a cost problem is a parody.
• It must survive its own deploys. A daily job that silently dies during a rollout is worse than no job.

Architecture

A Temporal Schedule fires a daily workflow per connected account. The workflow has four stages, each a set of activities with their own retry policies:

1. Collect. Pull the billing export and provider usage APIs (token counts per model from the LLM providers). Land raw data in Postgres.
2. Detect. Build per-service daily cost series and run robust anomaly detection: rolling median with median-absolute-deviation fences, computed separately for weekdays and weekends. Robust statistics on purpose: one backfill should not permanently widen the thresholds, which is what mean-and-standard-deviation detectors do.
3. Diagnose. For each flagged anomaly, an LLM receives the anomaly record, recent deploy events, and that service's usage metrics, and returns a short diagnosis: what likely changed, what to check first. The LLM never sees the raw billing dump. Detection is deterministic, so a wrong narrative can never change what gets flagged.
4. Deliver. One Slack message per anomaly, linking into a small Next.js dashboard: the series, the baseline, nearby deploys, and the LLM's note.

Deploy is a standard Kubernetes setup managed by ArgoCD: one worker deployment per task queue, schedules defined in the repo, the dashboard as a separate service.

Decisions

Temporal, not a CronJob. The pipeline is a dozen flaky API calls strung together. Durable retries, per-activity timeouts, and a schedule the server owns remove the two classic ways this kind of job dies: transient API failures and missed windows during deploys. The workflow code reads like the pipeline diagram, which made review easy.

Deterministic detection before LLM analysis. The expensive, unreliable component gets the smallest possible job. Detection costs nothing and never hallucinates; the LLM adds narrative to facts that already exist. This also caps the tool's own token spend at a handful of calls per day.

Chaos-tested recovery. I killed worker pods mid-workflow during load tests and let Temporal replay them. The interesting bug: the deliver stage was not idempotent at first, so a replay after a successful Slack post produced duplicates. Fixed with an idempotency key derived from workflow ID and anomaly ID. Recovery from a mid-collect kill now completes without duplicate alerts or partial data.

Outcome

The demo at finops.arielsofi.com runs against synthetic billing data seeded with the anomaly classes the detector was built for: retry storms, scope-creep features, forgotten batch jobs, weekday seasonality shifts. It catches those reliably. Honest limits: the detector needs two to three weeks of history before its baselines mean anything, and a step change from a real launch will always need a human to say "expected." The point was to prove the loop (collect, detect, diagnose, deliver, verify) runs itself. That part holds.