Most website incidents do not begin with a crash. They begin with a small difference between what a team believes is running and what is actually running \u2014 a plugin version, a server setting, an environment variable changed during a Friday fix. Treating the symptom gets the site back online; treating the condition is what stops the pattern, and the condition has a name in modern operations: undeclared state. The diagnostic path below follows the standard clinical order \u2014 symptoms, underlying cause, supporting evidence, treatment \u2014 and lands on GitOps, because that is currently the most complete prescription for this particular ailment.<\/p>\n
If your delivery process produces several of the following, the diagnosis is already half written:<\/p>\n
None of these are character flaws. They are the natural output of a process where the true configuration lives in people’s heads and terminals rather than in a written record. Web teams are especially prone because content platforms blur the line between content, configuration, and code \u2014 but the failure mechanism is identical everywhere.<\/p>\n
Engineers call the gap between recorded intent and live reality drift<\/em>, and it enters through three ordinary doors. The first is manual access: anyone with a login can change production, and sooner or later someone does, for understandable reasons, at a bad hour. The second is the platform itself: autoscalers change replica counts and controllers fill in default fields, so parts of the system differ from the record for legitimate reasons. The third is untracked external change \u2014 a DNS update made in a registrar’s console, a certificate renewed by hand, a dependency updated by an unattended job. What turns these from curiosities into incidents is accumulation: each small divergence is survivable, and the sum of them is a production environment that no longer matches anything written down.<\/p>\n Anecdote is cheap, so it is worth looking at numbers from a published teaching lab that instrumented a real cluster for seven days. During that week, one application change in five failed at the apply stage \u2014 a 20 percent failure rate that the team had never aggregated because the failures were scattered across people and days. The systems were out of sync with their intended state for 1.12 percent of all application-minutes measured, and two drift events accounted for 94 percent of that time. Of six drift events detected, four were reverted; more telling, four of five out-of-band changes discovered during the week had never been logged or ticketed by anyone. Read those figures as a portrait of the condition rather than an indictment of the team: without a record and a checker, drift is invisible until it compounds, and manual changes bypass governance precisely because no governance loop exists to catch them. A practice that makes the record authoritative and the checking automatic attacks every one of these numbers at once.<\/p>\n The prescription has several parts, and they reinforce one another. First, declare the state: write the configuration as files that state what should exist, and keep them in a repository where every revision is preserved. Second, install an agent \u2014 Argo CD or Flux are the standard choices \u2014 whose job is to fetch that record and keep the running system matched to it, continuously and without holidays. Third, close the side doors: production changes happen through reviewed pull requests rather than logins, which converts every intervention into a documented, attributable, revertible event. Fourth, tidy the pipeline around the record: render configuration in CI and review the rendered diff, keep one value change per overlay so a diff states its own intent, and promote a tested version by its content digest rather than by a moving tag. A related rule earns its own sentence: mutable references \u2014 the :latest<\/em> tag, an unpinned chart, a remote base pointing at whatever HEAD happens to be \u2014 quietly break the immutability that the whole model rests on. Pin everything; if a reference can move, sooner or later it will move.<\/p>\n One partial remedy deserves a caution. Teams sometimes respond to repeated deploy failures by bolting another script onto the pipeline, one that logs into the server and applies the latest build from outside. That automates the delivery but preserves the disease: the record still lives outside the system, the credentials to reach production still sit in the pipeline’s storage, and nothing watches the system between pushes. Automation of a broken model yields faster breakage. The distinctive move in the plan above is not “more automation” but a transfer of authority \u2014 the record becomes the only legitimate source of change, and the watcher sits inside the boundary rather than outside it.<\/p>\n Once a reconciler is watching, every divergence becomes visible, and the team needs a decided response for each class of difference rather than an improvisation per incident. Four options cover the field. Revert<\/strong> is the default for unauthorised edits: the agent restores the recorded state on its next pass. Tolerate<\/strong> fits legitimate self-adjustment \u2014 autoscaler decisions and webhook-filled fields are handled by marking those fields ignored, so the loop does not fight the platform. Alert<\/strong> suits differences that need a human judgement before action. Protect<\/strong> is for resources that must never be swept away by cleanup: the structural pieces of a cluster \u2014 namespace objects, storage claims, custom resource definitions \u2014 earn that status, because a careless prune against them cascades destructively. Writing this policy down, resource class by resource class, is an afternoon’s work that prevents the worst failure mode of automation \u2014 a well-meaning agent deleting something real at the worst possible time.<\/p>\n Teams that complete this treatment report a change in the texture of their work rather than a single dramatic moment. Deployments become boring, which is the point. Release weeks lose their dread because nothing about them depends on stamina or luck. Staging regains credibility because it is regenerated from the same records as production. The question “who changed this?” answers itself from the repository’s history, and rollback becomes a file operation rather than an investigation. Incidents do not vanish \u2014 nothing removes them \u2014 but their blast radius shrinks because divergence is caught in minutes instead of compounding for weeks. For teams rebuilding their delivery foundations, the diagnostic instinct generalises: the same “record it, review it, automate it” sequence that underpins the essential pages every business website should include<\/a> or the discipline of planning a business website before development starts<\/a> applies here at the infrastructure level. Deeper clinical detail \u2014 labs, runbooks, and worked policies \u2014 waits in the full GitOps reference notes<\/a>, and teams plotting the wider journey will find their bearings in a practical roadmap for DevOps, SRE, and platform engineering<\/a>. Diagnose honestly, treat in order, and the 6 p.m. pages stop repeating.<\/p>\n","protected":false},"excerpt":{"rendered":" Symptoms, causes and evidence: why websites suffer config drift, what a monitored week reveals, and the GitOps treatment plan that ends repeat incidents.<\/p>\n","protected":false},"author":14,"featured_media":5942,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1455],"tags":[],"class_list":["post-5941","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-development"],"_links":{"self":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts\/5941","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/comments?post=5941"}],"version-history":[{"count":1,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts\/5941\/revisions"}],"predecessor-version":[{"id":5960,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts\/5941\/revisions\/5960"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/media\/5942"}],"wp:attachment":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/media?parent=5941"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/categories?post=5941"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/tags?post=5941"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Evidence from a monitored week<\/h2>\n
The treatment plan GitOps prescribes<\/h2>\n
Follow-up care: a drift policy per resource<\/h2>\n
What recovery looks like<\/h2>\n