{"id":6268,"date":"2026-11-16T23:20:00","date_gmt":"2026-11-16T23:20:00","guid":{"rendered":"https:\/\/www.cmsgalaxy.com\/blog\/?p=6268"},"modified":"2026-11-19T23:20:00","modified_gmt":"2026-11-19T23:20:00","slug":"web-agency-server-hygiene-fictional-case","status":"publish","type":"post","link":"https:\/\/www.cmsgalaxy.com\/blog\/web-agency-server-hygiene-fictional-case\/","title":{"rendered":"A Web Agency Cleans Up Its Servers: A Fictional SysOps Story"},"content":{"rendered":"<p>Nobody real is described below: the team is invented, a sketch drawn to make the ideas concrete. Imagine a five-person agency that builds and hosts content-managed sites for local clients. It runs three Linux servers, one of them shared by a dozen client sites. For two years nothing dramatic happened, which is precisely why nobody looked. Then one Monday a client&#8217;s checkout stopped sending confirmation emails, and the quarter that followed became the agency&#8217;s introduction to SysOps, the discipline of looking after the host underneath the website.<\/p>\n<h2>The incident, in evidence<\/h2>\n<p>The team&#8217;s first question was not &#8220;who broke it?&#8221; but &#8220;what changed?&#8221; The logs told a plain story. A nightly job that should have rotated the mail queue had not run for eleven days. The job lived in an old scheduling table, and when the lead developer opened it, the command contained a percent sign inside a date format. In that scheduler an unescaped percent sign is treated as a newline, so the command had been silently truncated since the day somebody edited it. Nothing had crashed; a chore had simply stopped happening.<\/p>\n<p>A fuller account of why this class of fault is so common appears in <a href=\"https:\/\/www.xopsschool.com\/sysops\/\">the Ops Atlas page that the team later used as a checklist<\/a>. It points out that the older scheduler runs commands with a minimal shell and that systemd timers offer catch-up runs, randomised delays and a failure hook. The agency moved the three most important jobs to timers and attached a notification to each failure.<\/p>\n<h2>What the audit found<\/h2>\n<p>The incident justified a one-afternoon inventory. The results, written as a short table in the team&#8217;s wiki, were not dramatic but they were instructive.<\/p>\n<ul>\n<li>One shared administrator login used by everybody, with a password that predated two departures.<\/li>\n<li>A remote-access configuration with a stricter rule at the bottom of the file that was overridden by an earlier, looser one. The team had assumed the later line won; in that program the first matching value generally takes effect.<\/li>\n<li>A permission file for the sudo tool with a dot in its name, which the tool skips without complaint.<\/li>\n<li>Two servers with journals set to default behaviour and one with a log directory that had quietly grown to a large share of the disk.<\/li>\n<li>No record of which machines were waiting for a restart after updates.<\/li>\n<\/ul>\n<p>None of these was an emergency, and that is the analytical point: each was an ordinary configuration drift that would sit unnoticed for years, and each could be found with a five-minute check once somebody asked. The team treated the list as a backlog, not as a blame sheet.<\/p>\n<h2>The numbers behind the disk decision<\/h2>\n<p>The shared server carried a 500 GiB volume, 410 GiB of it used. Client uploads had added 63 GiB in a week. The team&#8217;s first reading was &#8220;we have 90 GiB left and about ten days&#8221;. The better calculation divided the growth over seven days, giving 9 GiB per day, and subtracted the filesystem&#8217;s reserved share for the administrator, around 25 GiB. That left roughly 65 GiB that ordinary programs could still use, which is about seven days, not ten. The difference mattered because a purchase order for storage takes longer than a week to clear. They ordered the volume extension that afternoon.<\/p>\n<p>The team added &#8220;days of runway&#8221; to its weekly review and set an alert at fourteen days. A figure that predicts is more useful than a percentage that reports.<\/p>\n<h2>Why the evidence-first habit paid off<\/h2>\n<p>Looking back, the most valuable decision was procedural, not technical. The team agreed that nobody would change a server without writing the intended change, the time and the expected effect in a shared log. When a later alert fired, the log narrowed the search from &#8220;anything&#8221; to &#8220;the three things touched this week&#8221;. It is a cheap discipline, and it makes every later investigation shorter.<\/p>\n<h2>Changes, in order of effort<\/h2>\n<p>The agency ranked its fixes by effort and risk and tackled the cheapest first. The shared login was removed and replaced by named accounts, each with a narrowly scoped privilege file checked by the sudo tool&#8217;s validator. The remote-access file was reordered, validated with the daemon&#8217;s test mode, and applied while a second session stayed open. Journals were given a clear size cap and an explicit storage setting. Reboots were scheduled monthly, with a short list of machines that still owed one.<\/p>\n<p>Two things were deliberately not done. The team refused to turn off packet filtering or SELinux just to coax a stubborn plugin into running; it relaxed one rule for one service and logged the reason. And it did not trust a disk snapshot as a backup, since the snapshot lived on the same storage as the data. It scheduled a quarterly restore of a sample onto a spare machine instead.<\/p>\n<h2>What the agency learned about its own work<\/h2>\n<p>Several lessons carry over to any team that runs a content platform. Content sites look simple from the outside; the page is the tip, and the host is the keel. The agency&#8217;s earlier reading on <a href=\"https:\/\/www.cmsgalaxy.com\/blog\/turning-clicks-into-clients-the-power-of-effective-service-pages\/\">service pages that convert browsers into clients<\/a> and on <a href=\"https:\/\/www.cmsgalaxy.com\/blog\/shaping-customer-perception-through-effective-homepage-design\/\">the way homepage design forms a visitor&#8217;s first impression<\/a> described what clients see. The new work protected what they do not.<\/p>\n<p>It also changed how the agency talked to clients. Instead of promising &#8220;100 percent uptime&#8221;, the team described the measures it tracked: how many scheduled jobs succeeded, how many days of disk remained, how many machines owed a restart. Those are the same metrics the Atlas suggests for judging operational health, and they gave clients something concrete to ask about.<\/p>\n<h2>A ninety-day shape you can borrow<\/h2>\n<p>The Atlas sketches a plan in three blocks of roughly a month each, and the invented agency followed it loosely. In the first month it took an inventory, removed the shared root login and set up basic performance history. In the second it brought configuration under a repeatable tool, moved the scheduled jobs to timers and capped the logs. In the third it tested a restore, practised reaching a server through its console, and put end-of-support dates for its operating systems on a calendar.<\/p>\n<p>By the end of the quarter the agency had fewer surprises, not zero. That is a realistic target for a small team, and it is measurable. Whether you run a CMS agency or a single blog, the sequence is the same: look, record, change one thing, look again.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>An imagined five-person agency hosts client CMS sites on a few Linux servers. Follow its first quarter of systems operations, with the numbers behind each decision.<\/p>\n","protected":false},"author":14,"featured_media":6269,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1455],"tags":[],"class_list":["post-6268","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-development"],"_links":{"self":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts\/6268","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/comments?post=6268"}],"version-history":[{"count":1,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts\/6268\/revisions"}],"predecessor-version":[{"id":6402,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/posts\/6268\/revisions\/6402"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/media\/6269"}],"wp:attachment":[{"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/media?parent=6268"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/categories?post=6268"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cmsgalaxy.com\/blog\/wp-json\/wp\/v2\/tags?post=6268"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}