When people hear that cluster provisioning went from weeks to hours, they usually ask which tool did it. At Lucid Motors, the honest answer is none of them on its own. We used Cluster API, Argo CD, Terraform and Atlantis, but teams use those tools and still wait weeks. What changed was that we built one paved path out of them and took the ticket queue out of the critical path.

The Old Path

A new cluster started as a ticket. Someone created the account or found one, set up credentials for the deployment tooling, created the cluster with Terraform, registered it with Argo CD, delegated DNS, installed the platform add-ons and set cost tags. Each step had its own owner and its own queue. Most of the elapsed time went to waiting for the next person to pick up the ticket, not to the work itself.

The result was a set of snowflakes. Two clusters built a month apart by different people had different add-on versions, different node settings and different exceptions, and nobody had a complete record of which differences were intentional.

The Paved Path

We replaced the sequence of tickets with a sequence of merge requests.

One file per cluster. Each cluster was described by a values file of about twenty keys: account, cloud, region, environment, API endpoint, DNS zones, cost tags and a list of components the cluster opts out of. Everything else came from that file: the Argo CD Applications, the IAM role names, the Cluster API identity, DNS delegation and cost allocation. Onboarding a cluster meant writing that file and opening a merge request.

Cluster API for lifecycle. Clusters were declared in Git as Cluster API objects and created by a controller on the hub, which assumed a role in the target account instead of using stored credentials.

Terraform and Atlantis for reviewable changes. Account-level resources went through Terraform, planned and applied from merge request comments through Atlantis. The plan was posted on the merge request, so the reviewer saw exactly what would change before approving it.

Argo CD for delivery. One hub delivered the full platform baseline to every cluster. New components reached every cluster by default, and any exclusion was recorded in that cluster’s file and reviewed with it.

On top of that, a self-service portal turned common requests into the same commits an engineer would have written. A request to initialize an account drove Terraform through Atlantis. A request for a cluster committed values to Git and let the GitOps controllers take over.

Getting a new cluster end to end went from weeks to hours. The narrower part we measured formally, from merging a cluster’s registration to every Application on it reconciling, fell from a mean of 4.0 days to 1.7 hours across 11 onboardings.

Where the Savings Came From

The same program saved more than $100,000 a month. People expect that kind of number to come from a better pricing deal. Ours came from standardization, which made cost work possible everywhere at once.

Once every cluster ran the same baseline, a cost control written once applied to all of them. Non-production node groups were scaled down, and non-production databases were stopped on a schedule. Node provisioning moved to an autoscaler that picks instance sizes to fit the pods actually waiting. Right-sizing recommendations were available for every workload. A sweeper deleted unattached volumes, and an exporter reported idle resources on the second cloud. Cost tags were derived from the same cluster file, so spend could be attributed to a team without anyone tagging by hand.

I won’t claim to know how the savings split across those levers. They reinforced each other, and none of them would have been practical on 34 hand-built clusters.

What Still Needed Humans

Three things stayed outside the paved path, and they should.

Exceptions. Many clusters predated the platform and were created with Terraform. We adopted them in place, and the platform only took ownership of resources that were explicitly imported.

Regions. Data residency requirements decided where some accounts lived. Our second cloud had a different identity model, so the same configuration produced different security properties there, and someone had to understand the difference.

Regulated workloads. Government and in-country requirements came with reviews that no automation should skip.

The platform didn’t remove judgment from these cases. It made them visible as exceptions, instead of mixing them in with routine work.

Advice for Teams Mid-Journey

Automate the review path before the build path. Teams usually start by automating cluster creation, and then the automated builds sit in a review queue because nobody can tell what a change will do.

We rendered the full deployment graph for every cluster in CI on each merge request and posted the manifest difference as a comment. A reviewer could see what a template change would do to every account before approving it. That’s what made fast approval safe, and fast approval is what turned weeks into hours.

Nitish Mane is a Software Engineer at Philo. At Lucid Motors, he led the design of a GitOps control plane spanning 22 AWS accounts and an OCI tenancy.

SHARE THIS STORY