Most website incidents do not begin with a crash. They begin with a small difference between what a team believes is running and what is actually running — a plugin version, a server setting, an environment variable changed during a Friday fix. Treating the symptom gets the site back online; treating the condition is what stops the pattern, and the condition has a name in modern operations: undeclared state. The diagnostic path below follows the standard clinical order — symptoms, underlying cause, supporting evidence, treatment — and lands on GitOps, because that is currently the most complete prescription for this particular ailment.

The symptoms: GitOps-shaped failures look familiar

If your delivery process produces several of the following, the diagnosis is already half written:

• A site that works on a developer’s laptop and misbehaves on the server, with nobody able to state the difference between the two.
• Deployments whose outcome depends on which person runs them.
• Hotfixes applied directly on production, remembered only by the person who typed them.
• A staging environment that drifted so far from production that testing on it feels cosmetic.
• Rollback treated as archaeology: digging through chats and shell history to reconstruct last week’s configuration.

None of these are character flaws. They are the natural output of a process where the true configuration lives in people’s heads and terminals rather than in a written record. Web teams are especially prone because content platforms blur the line between content, configuration, and code — but the failure mechanism is identical everywhere.

The underlying condition: undeclared state and silent drift

Engineers call the gap between recorded intent and live reality drift, and it enters through three ordinary doors. The first is manual access: anyone with a login can change production, and sooner or later someone does, for understandable reasons, at a bad hour. The second is the platform itself: autoscalers change replica counts and controllers fill in default fields, so parts of the system differ from the record for legitimate reasons. The third is untracked external change — a DNS update made in a registrar’s console, a certificate renewed by hand, a dependency updated by an unattended job. What turns these from curiosities into incidents is accumulation: each small divergence is survivable, and the sum of them is a production environment that no longer matches anything written down.

Evidence from a monitored week

Anecdote is cheap, so it is worth looking at numbers from a published teaching lab that instrumented a real cluster for seven days. During that week, one application change in five failed at the apply stage — a 20 percent failure rate that the team had never aggregated because the failures were scattered across people and days. The systems were out of sync with their intended state for 1.12 percent of all application-minutes measured, and two drift events accounted for 94 percent of that time. Of six drift events detected, four were reverted; more telling, four of five out-of-band changes discovered during the week had never been logged or ticketed by anyone. Read those figures as a portrait of the condition rather than an indictment of the team: without a record and a checker, drift is invisible until it compounds, and manual changes bypass governance precisely because no governance loop exists to catch them. A practice that makes the record authoritative and the checking automatic attacks every one of these numbers at once.

The treatment plan GitOps prescribes

The prescription has several parts, and they reinforce one another. First, declare the state: write the configuration as files that state what should exist, and keep them in a repository where every revision is preserved. Second, install an agent — Argo CD or Flux are the standard choices — whose job is to fetch that record and keep the running system matched to it, continuously and without holidays. Third, close the side doors: production changes happen through reviewed pull requests rather than logins, which converts every intervention into a documented, attributable, revertible event. Fourth, tidy the pipeline around the record: render configuration in CI and review the rendered diff, keep one value change per overlay so a diff states its own intent, and promote a tested version by its content digest rather than by a moving tag. A related rule earns its own sentence: mutable references — the :latest tag, an unpinned chart, a remote base pointing at whatever HEAD happens to be — quietly break the immutability that the whole model rests on. Pin everything; if a reference can move, sooner or later it will move.

One partial remedy deserves a caution. Teams sometimes respond to repeated deploy failures by bolting another script onto the pipeline, one that logs into the server and applies the latest build from outside. That automates the delivery but preserves the disease: the record still lives outside the system, the credentials to reach production still sit in the pipeline’s storage, and nothing watches the system between pushes. Automation of a broken model yields faster breakage. The distinctive move in the plan above is not “more automation” but a transfer of authority — the record becomes the only legitimate source of change, and the watcher sits inside the boundary rather than outside it.

Follow-up care: a drift policy per resource

Once a reconciler is watching, every divergence becomes visible, and the team needs a decided response for each class of difference rather than an improvisation per incident. Four options cover the field. Revert is the default for unauthorised edits: the agent restores the recorded state on its next pass. Tolerate fits legitimate self-adjustment — autoscaler decisions and webhook-filled fields are handled by marking those fields ignored, so the loop does not fight the platform. Alert suits differences that need a human judgement before action. Protect is for resources that must never be swept away by cleanup: the structural pieces of a cluster — namespace objects, storage claims, custom resource definitions — earn that status, because a careless prune against them cascades destructively. Writing this policy down, resource class by resource class, is an afternoon’s work that prevents the worst failure mode of automation — a well-meaning agent deleting something real at the worst possible time.

What recovery looks like

Teams that complete this treatment report a change in the texture of their work rather than a single dramatic moment. Deployments become boring, which is the point. Release weeks lose their dread because nothing about them depends on stamina or luck. Staging regains credibility because it is regenerated from the same records as production. The question “who changed this?” answers itself from the repository’s history, and rollback becomes a file operation rather than an investigation. Incidents do not vanish — nothing removes them — but their blast radius shrinks because divergence is caught in minutes instead of compounding for weeks. For teams rebuilding their delivery foundations, the diagnostic instinct generalises: the same “record it, review it, automate it” sequence that underpins the essential pages every business website should include or the discipline of planning a business website before development starts applies here at the infrastructure level. Deeper clinical detail — labs, runbooks, and worked policies — waits in the full GitOps reference notes, and teams plotting the wider journey will find their bearings in a practical roadmap for DevOps, SRE, and platform engineering. Diagnose honestly, treat in order, and the 6 p.m. pages stop repeating.