GitOps AI Platform

The problem

An internal AI platform serving several product teams had grown by accretion: clusters click-opsed into existence, pipelines in three CI systems, datasets discoverable only by asking the person who made them. Every new team onboarding was a bespoke project, and every audit question started an archaeology dig.

Constraints

• Multiple teams with different data sensitivity levels on shared clusters; isolation had to be infrastructure, not convention.
• Everything reproducible. If the platform could not be rebuilt from the repo, it did not count as managed.
• Teams keep autonomy over their workloads; the platform provides rails, not a ticket queue.
• Pipelines range from minute-long eval runs to day-long training jobs; one workflow engine had to serve both.

Architecture

Kubernetes clusters per environment, all state reconciled by ArgoCD from a GitOps repo using an app-of-apps layout. The root app deploys platform components in sync waves: CRDs and namespaces first, then the secret store, cert-manager, and external-dns, then platform services, then team workloads.

Per-tenant structure: each team gets a namespace set (dev, staging, prod) provisioned from a template ApplicationSet, with default-deny network policies, resource quotas, and a workload identity binding. Teams self-serve by adding their Application manifest to the repo; the platform review is a PR review.

Pipelines run on Argo Workflows: eval suites, embedding jobs, fine-tuning runs, and data refreshes are all workflow templates in the repo, triggered by CI or by schedules. Metadata flows into DataHub, which ingests from workflow runs, the feature store, and the warehouse, so dataset lineage from source to model input is queryable instead of remembered.

DNS is external-dns against the internal zone, records from annotations, upsert-only until the audit passed, sync after. Secrets via External Secrets against the cloud secret manager, rotation tested in the quarterly drill.

Decisions

App-of-apps over one giant Application. Independent sync and rollback per workload beat the operational simplicity of a single app the first time one team's broken manifest would have frozen everyone's deploys.

Argo Workflows over a managed pipeline service. The teams needed containers as first-class steps and schedules they could read in a diff. The cost is operating another controller; the payoff is that every pipeline is reviewable code living next to the thing it builds.

DataHub as the catalog, wired to the workflow engine. Catalogs die when documentation is a separate chore. Ingesting metadata from workflow runs means the catalog updates because work happened, not because someone remembered.

Outcome

New team onboarding went from weeks of bespoke setup to a PR against the ApplicationSet template plus a day of working sessions. The quarterly rollback and recovery drill runs in an afternoon, including the database-migration case. Audit questions about where a dataset came from or who can reach a model now end with a query and a link. The platform still needs two people who understand it deeply; the difference is that the bus factor is documented rather than fatal.