What I Look for When Reviewing an AI Platform Architecture

An AI platform review is not a code review and it is not a vendor evaluation. It is two days of asking where things live, who can reach them, and what happens when they fail. Six areas decide whether a platform survives contact with production: tenancy, egress control, model routing, observability, kill-switches, and IaC coverage. For each: the questions I ask, what good looks like, and what I usually find instead.

1. Tenancy

Questions: how is one tenant isolated from the next: namespace, account, VPC, or a row filter? Who shares GPU capacity, and what happens when one tenant's batch job lands during another's latency-sensitive window? Can a tenant's plugin or tool read another tenant's data through a shared cache or a shared vector index?

Good: isolation boundaries are infrastructure, not application logic. Namespaces or accounts per tenant tier, network policy default-deny between them, per-tenant encryption keys for anything regulated, and an explicit answer for noisy neighbors on shared accelerators.

Usually found: a shared cluster with per-row tenant IDs and a sincere belief that the ORM enforces them. The vector database is shared, namespaced by a metadata field, which means one malformed filter is a cross-tenant data leak.

2. Egress control

Questions: where can a workload send data, physically? Is model traffic forced through a single gateway, or can any pod with the API key call the provider directly? What does the DNS layer resolve from the AI subnets? If an agent gets prompt-injected tomorrow, which channels can it use to exfiltrate?

Good: the AI subnets have no public route, model APIs resolve to private endpoints, DNS is allowlisted, and the only permitted destination is a gateway that authenticates workloads, redacts, logs, and calls providers over pinned paths. The control is a property of the network, not a promise in a policy document.

Usually found: an acceptable-use policy and a NAT gateway. The routing table disagrees with the policy, and the routing table wins.

3. Model routing

Questions: where does the decision of which model serves a request live? Is there a routing table, and does it encode data sensitivity as well as task difficulty? What is the fallback when a provider degrades? Who changed the routing last, and how would you know?

Good: routing lives in the gateway, not in application code. The table maps task types to model tiers with an eval gate behind each tier change. Data classification is a routing input: which model may see which data is encoded in the router, not in a wiki page.

Usually found: model names hardcoded in application code, three services each calling the provider SDK directly, and a fourth service nobody remembers deploying.

4. Observability

Questions: can you state cost per successful request by feature? Do traces connect the user action to the model call to the provider response? Where are prompts and completions logged, and who can read them? What evals run continuously, and what happens when they regress?

Good: every model call is logged at the gateway with workload identity, token counts, cost, and latency. Traces carry the feature and tenant. Eval suites run on a schedule and gate the routing table. Prompt logs have their own access policy, because they are the most sensitive data in the system.

Usually found: request logs without token counts, so cost attribution is archaeology. No evals in production at all. Prompts in a general-purpose log bucket readable by half the engineering org.

5. Kill-switches

Questions: can you disable one model, one feature, or one tenant in minutes, without a deploy? When did someone last actually do it? If a provider has an incident at 2am, what routes around it, and has that path ever been exercised?

Good: flags at the gateway for model, feature, and tenant, each tested within the last quarter. Provider failover is configured and has carried traffic at least once, even in a drill. A kill-switch that has never been pulled is a hypothesis.

Usually found: disabling a model means a code change and a pipeline run. The failover path exists in a diagram.

6. IaC coverage

Questions: what percentage of this platform could you rebuild in a fresh account from the repo? Which resources were created in a console? Where is drift detected? Are the IAM policies and the endpoint policies in version control, or only the compute?

Good: everything except a documented break-glass list is in Terraform or Pulumi, drift detection runs nightly, and the answer to "rebuild it" is measured in hours. Policies, the things a reviewer actually asks about, are code.

Usually found: compute is in Terraform; IAM, endpoints, DNS, and budgets are console archaeology. The platform can be redeployed but cannot be re-secured.

How I score it

Each area gets a rung, not a grade: what is true today, with evidence. The platform's effective maturity is capped by its weakest critical area, and for regulated industries the critical ones are egress control and observability, because those are what an examiner or a breach investigation will ask about first. A platform with excellent MLOps and open egress is not mature. It is fast and exposed.

Sequencing the fixes matters more than the scores. Kill-switches and egress first, because both are days of work and they bound the blast radius of everything else. Routing and observability next, because they turn cost and quality from mysteries into measurements. Tenancy work is usually the most expensive and should be planned, not panicked into. IaC coverage is continuous: every fix lands as code or it did not land.

The tell of a healthy platform is not any single answer. It is how fast the team can produce evidence. If every question takes a day of archaeology, the platform runs on memory, and memory does not survive the next reorg.