From 48 Hours to 2: What Cluster Automation Actually Taught Me

When we started the VMware-to-OpenShift migration at EA, provisioning a production OpenShift cluster took 48 hours and a very patient engineer working through a runbook that had grown to cover three datacenters: EU1, IAD1, and IAD1-ISO. Today it takes under two hours at 85% automation coverage. Everyone wants to talk about the number. The number is not the interesting part.

What the runbook actually was

A 48-hour runbook is not 48 hours of work. It is 48 hours of calendar time: waiting for approvals, waiting for humans in other timezones to complete their step, waiting for a DNS record to propagate before the next step can be verified. The work itself, if you compressed it, was maybe six hours. The rest was coordination overhead and implicit trust: one team trusting that another team had done their part correctly before handing off.

The automation did not eliminate the six hours of work. It eliminated the trust assumptions that made the other 42 hours necessary.

The trust-boundary question

Every step in a runbook encodes a trust decision. When an engineer manually provisions a node and marks it complete in a ticket, they are asserting: "this thing is in the state the next step assumes." A downstream engineer accepts that assertion without verification. This is how runbooks accumulate risk: not in any single step, but in the chain of unverified assertions between them.

When you automate a step, you are not just replacing a human action. You are replacing a human assertion with a machine-verifiable one. Ansible does not tell you a node is ready. It exits zero when the node is provably ready, and your pipeline will not proceed otherwise. The trust is no longer implicit. It is encoded.

This distinction matters because it is where most automation projects go wrong. Teams automate the mechanical steps and leave the assertion steps as human checkpoints. They get a faster runbook, not a fundamentally different one. The 42 hours of coordination overhead survives.

What we actually built

The framework that reduced provisioning time combines tools working in a trust chain:

Ansible handles the imperative provisioning work: node preparation, network configuration, storage attachment. It asserts completion via idempotent plays that fail loudly on any deviation from expected state.

Terraform manages the declarative infrastructure layer and produces a state file that serves as a verifiable record of what exists.

Kyverno enforces policy at the Kubernetes API layer. No workload lands without meeting the defined security and labeling standards, and no human approval is needed because the policy is the approval.

ArgoCD handles GitOps reconciliation, continuously asserting that the cluster's running state matches the declared state in git.

GitLab CI orchestrates the pipeline and provides the audit trail. Every provisioning run is a pipeline execution with a log, a git commit, and a timestamp.

The 15% that remains manual is not arbitrary. It is the set of steps where the cost of encoding the trust assertion exceeds the cost of a human making it once per cluster: physical datacenter access for initial hardware, and the final sign-off that a new cluster is ready to receive production workload. That sign-off will likely automate too, but only when we have enough observability data to trust the automated assertion over a human one.

The shift toward SATHOS

Once you have thought deeply about trust assertions in infrastructure automation, you start asking an uncomfortable question: if the pipeline is doing the work that humans used to do, who is verifying that the pipeline itself is trustworthy?

A human engineer provisioning a cluster is identifiable, auditable, and accountable. A pipeline job doing the same work has an identity too: a service account, a token, a role. But that identity is often static, broadly scoped, and poorly attested. We traded human trust assumptions for machine ones without fully thinking through what machine trustworthiness requires.

That question is the seed of the SATHOS research. The automation got us the 48-to-2 number. The security research is about making sure the two hours are not just fast, but provably safe.

What I would tell teams starting this

Start with the assertion layer, not the automation layer. Before you write a single Ansible play, map every step in your runbook and ask: what is this step asserting, and how would you know if that assertion were wrong? The places where the answer is "we would not know" are where your automation needs to be most rigorous.

The number will follow. What matters is whether the trust you embedded in the runbook is now verifiable, auditable, and encoded in the system: or just hidden inside a faster process.