133 lines
12 KiB
HTML
133 lines
12 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<style id="fort-styles">
|
|
[data-fort]{font:inherit;color:inherit}
|
|
ul[data-fort]{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.45em}
|
|
[data-fort] li{margin:0;padding:0;line-height:1.45;max-width:100%;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
|
[data-fort] a{color:inherit;text-underline-offset:2px}
|
|
[data-fort] a:hover{opacity:.72}
|
|
[data-fort] .src,[data-fort] .fm-type{opacity:.6;font-size:.85em}
|
|
.fort-citizen{display:flex;flex-direction:column;gap:.4em}
|
|
.fort-citizen .fc-name{font-weight:600}
|
|
.fort-citizen .fc-pages{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.3em}
|
|
.fort-media,.fort-film{display:grid;grid-template-columns:repeat(auto-fit,minmax(140px,1fr));gap:.6em}
|
|
.fort-film{list-style:none;margin:0;padding:0}
|
|
.fort-film li{overflow:visible;white-space:normal}
|
|
.fort-film a{display:flex;flex-direction:column;gap:.3em;text-decoration:none}
|
|
.fort-media img,.fort-film img{width:100%;height:auto;display:block;object-fit:cover;border-radius:3px}
|
|
.fort-film img{aspect-ratio:16/10}
|
|
.fort-nav{display:flex;flex-wrap:wrap;gap:1em;align-items:center}
|
|
</style>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<title>The 28-Minute Gap — a monitoring field note by Allen Lorch</title>
|
|
<meta property="og:type" content="website">
|
|
<meta property="og:title" content="The 28-Minute Gap — a monitoring field note by Allen Lorch">
|
|
<meta property="og:description" content="Allen Lorch on the 28-minute off-course flight: what monitoring deviation instead of state teaches us about alert thresholds, trust, and graceful degradation.">
|
|
<meta property="og:image" content="https://images.pexels.com/photos/5085921/pexels-photo-5085921.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=650&w=940">
|
|
<meta property="og:url" content="https://allen-lorch.4ort.net/28-minute-gap.html">
|
|
<meta name="twitter:card" content="summary_large_image">
|
|
<meta name="description" content="Allen Lorch on the 28-minute off-course flight: what monitoring deviation instead of state teaches us about alert thresholds, trust, and graceful degradation.">
|
|
<style>
|
|
:root{
|
|
--ink:#1b2a1a;
|
|
--paper:#eef0e4;
|
|
--accent:#7a1212;
|
|
--line:#4a5a44;
|
|
--code:#d8dcc8;
|
|
}
|
|
*{box-sizing:border-box; margin:0; padding:0;}
|
|
body{
|
|
background:var(--paper); color:var(--ink);
|
|
font-family:Georgia,'Times New Roman',serif; line-height:1.6;
|
|
max-width:820px; margin:0 auto; padding:2rem 1.25rem 4rem;
|
|
}
|
|
header.top{border-bottom:3px solid var(--line); padding-bottom:1.25rem; margin-bottom:2rem;}
|
|
.kicker{font-family:'Courier New',monospace; font-size:.72rem; letter-spacing:.18em; text-transform:uppercase; color:var(--accent);}
|
|
h1{font-size:clamp(1.9rem,5vw,3rem); line-height:1.05; margin:.4rem 0 .5rem;}
|
|
nav{font-family:'Courier New',monospace; font-size:.8rem; margin-top:1rem;}
|
|
nav a{color:var(--accent); text-decoration:none; margin-right:1.25rem;}
|
|
nav a:hover{text-decoration:underline;}
|
|
h2{font-size:1.4rem; border-bottom:1px solid var(--line); padding-bottom:.3rem; margin:2.2rem 0 1rem;}
|
|
h2 .num{color:var(--accent); font-family:'Courier New',monospace; font-size:.85rem; vertical-align:middle;}
|
|
h3{margin:1.4rem 0 .5rem; font-family:'Courier New',monospace; font-size:.95rem;}
|
|
p{margin-bottom:1rem;}
|
|
a{color:var(--accent);}
|
|
figure{margin:1.5rem 0;}
|
|
img{width:100%; max-height:320px; object-fit:cover; display:block; border:2px solid var(--line); filter:sepia(.15);}
|
|
figcaption{font-family:'Courier New',monospace; font-size:.72rem; color:var(--line); margin-top:.35rem;}
|
|
blockquote{border-left:4px solid var(--accent); padding-left:1rem; margin:1.25rem 0; font-style:italic; color:#33412f;}
|
|
blockquote cite{display:block; font-style:normal; font-size:.78rem; font-family:'Courier New',monospace; color:var(--accent); margin-top:.4rem;}
|
|
table{width:100%; border-collapse:collapse; margin:1.25rem 0; font-family:'Courier New',monospace; font-size:.82rem;}
|
|
th,td{border:1px solid var(--line); padding:.5rem .6rem; text-align:left; vertical-align:top;}
|
|
th{background:var(--code); text-transform:uppercase; letter-spacing:.08em; font-size:.74rem;}
|
|
.fortlet-block{border:2px dashed var(--line); padding:1rem 1.25rem; margin:1.5rem 0; background:rgba(255,255,255,.3);}
|
|
.fortlet-block h3{font-family:'Courier New',monospace; font-size:.72rem; letter-spacing:.15em; text-transform:uppercase; color:var(--accent);}
|
|
footer{margin-top:3rem; border-top:2px solid var(--line); padding-top:1rem; font-family:'Courier New',monospace; font-size:.72rem; color:var(--line);}
|
|
</style>
|
|
<fort-nav><nav class="fort-nav" data-fort="nav"><a href="/films/agc-triage/">Alarm 1201: Triage at 25,000 mph</a><a href="/">ALLEN LORCH // OPS-LOG 001</a><a href="/datacenter-ops.html">DATA-CENTER OPS // PUE, power, and the $105B Ohio build</a><a href="/28-minute-gap.html" class="active" aria-current="page">The 28-Minute Gap</a></nav></fort-nav>
|
|
<script defer src="https://analytics.4ort.xyz/script.js" data-website-id="d3ed927c-888a-4a6c-ae5f-0b1c613ddf5b"></script>
|
|
</head>
|
|
<body>
|
|
|
|
<header class="top">
|
|
<div class="kicker">Allen Lorch · field note · OPS-LOG 001</div>
|
|
<h1>The 28-Minute Gap</h1>
|
|
<nav><a href="https://allen-lorch.4ort.net">HOME</a> <a href="#stack">THE STACK</a> <a href="#tripwire">THE TRIPWIRE</a> <a href="#agc">COUNTEREXAMPLE</a></nav>
|
|
</header>
|
|
|
|
<figure>
|
|
<img src="https://images.pexels.com/photos/5085921/pexels-photo-5085921.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=650&w=940" alt="A dim-lit museum space capsule — the machine you must learn to read even when it is silent.">
|
|
<figcaption>A cockpit's worth of machinery that can hold a course for half an hour and tell you nothing. Photo: Pexels (royalty-free).</figcaption>
|
|
</figure>
|
|
|
|
<p>Last month a scheduled passenger flight drifted off its cleared route for nearly thirty minutes before anyone noticed. Reports described the flight crew as having fallen asleep in the cruise phase. I am not a pilot. But I have spent enough late nights in a monitoring rotation to recognize the shape of the incident, because it is the same shape as half my tickets: <strong>the machine performed perfectly, and that perfection was the problem.</strong></p>
|
|
|
|
<h2 id="stack"><span class="num">§0</span> READ THE STACK</h2>
|
|
<p>The autopilot did what autopilots do: it held a heading, kept the wings level, followed the programmed plan. For twenty-eight minutes it did this flawlessly. If you only watched the <em>state</em> — altitude stable, heading steady, engines cycling on schedule — nothing was wrong. The aircraft was doing exactly what it was told.</p>
|
|
<p>The failure lived one level up. Nobody was watching the <em>second derivative</em>: whether the current course still matched the <em>cleared</em> course. The plane wasn't malfunctioning. It was obediently going somewhere it had not been cleared to go — the way a misconfigured cron job, or a routing rule that points one subnet at a dead gateway, will keep a whole environment running <em>smoothly, wrong</em>, until someone happens to glance at the actual destination.</p>
|
|
|
|
<table>
|
|
<tr><th>Level</th><th>What it monitors</th><th>Why it failed here</th></tr>
|
|
<tr><td>State</td><td>"Is the plane flying?"</td><td>Yes. Altitude, heading, engines all nominal. Passed.</td></tr>
|
|
<tr><td>Course</td><td>"Is it where it was cleared to be?"</td><td>Not watched on this leg. This is the gap.</td></tr>
|
|
<tr><td>Intent</td><td>"Is the plan still worth following?"</td><td>Only the crew can judge this — and they were checked out of the loop.</td></tr>
|
|
</table>
|
|
|
|
<p>The lesson transfers straight to a server rack. Alerting on CPU use is monitoring <em>state</em>; alerting on "requests are being served but not reaching the database" is monitoring the <em>course</em>. State alerts keep you alive but blind. Course alerts tell you you're drifting off the map. Build your tripwires on course, and you shrink a twenty-eight-minute gap to a heartbeat.</p>
|
|
|
|
<h2 id="tripwire"><span class="num">§1</span> THE TRIPWIRE</h2>
|
|
<p>Here is the discipline I keep returning to, the one I write in every onboarding note: <strong>decide, in advance, which silence is acceptable.</strong> Alert fatigue is not caused by too many alerts. It's caused by alerts that fire on state and never on course, so the ones that matter arrive drowned in the ones that don't. You don't fix that by silencing more; you fix it by retuning what counts as news.</p>
|
|
<blockquote>
|
|
“A human being can be lulled into unreadiness by a machine that is, by construction, uninterruptible. The bravest engineering is often deciding to make the quiet thing loud.”
|
|
<cite>— field note, OPS-LOG 001</cite>
|
|
</blockquote>
|
|
<p>My rules of thumb, for myself and any team I land on:</p>
|
|
<ul>
|
|
<li><strong>Monitor deviation, not state.</strong> The baseline is noise-free by design; alarm on the bend.</li>
|
|
<li><strong>One loud alert beats ten polite ones.</strong> If a tripwire has to break through trust in the machinery — the hardest thing to break — it should be unmistakable.</li>
|
|
<li><strong>Nobody is watching for free.</strong> An automated check that exists but isn't seen is a liability, not a comfort. If you won't be woken by it, remove it.</li>
|
|
</ul>
|
|
|
|
<h2 id="agc"><span class="num">§2</span> THE COUNTEREXAMPLE: AGC 1201</h2>
|
|
<p>The Apollo Guidance Computer — built by <strong>Raytheon</strong>, written in <strong>assembly language</strong>, its entire human interface a keypad-and-numbers unit called the <strong>DSKY</strong> — is my favorite counterexample. Its source is <strong>public</strong>: open the repo and you can read the exact instructions that flew to the Moon.</p>
|
|
<p>During the Apollo 11 descent, the computer began dropping low-priority tasks under a workload overload, and raised <strong>Alarm 1201</strong>. The crew did not have time to become experts. The flight team had already defined what that four-digit code meant, and the meaning was: <em>the machine is dropping the safe tasks to keep descending — that is acceptable, continue.</em> The plan lived before the panic. That is graceful degradation — the system deciding, in advance, which part of the mission to give up first.</p>
|
|
<div class="fortlet-block">
|
|
<h3>Live knowledge — AGC</h3>
|
|
<fort-kg query="apollo-guidance-computer" limit="5"><ul class="fort-kg" data-fort="kg" data-query="apollo-guidance-computer"><li>Apollo Guidance Computer — computer</li><li>Gemini Guidance Computer — digital computer designed for Project Gemini</li><li>Apollo Computer — developed and produced Apollo/Domain workstations in the 1980s</li><li>computer appliance — single-purpose computing device with software or firmware dedicated to providing a specific computing resource</li><li>command guidance — missile guidance method using signals from an offboard station</li></ul></fort-kg>
|
|
</div>
|
|
<p>That is the whole difference from the 28-minute gap. In the aircraft, the automation was too good at being uninteresting and nobody defined the second tripwire. In the Command Module, the automation was loud — four digits, unmistakable — and the tripwire had already been built the slow, unglamorous way: documented, rehearsed, written down before the emergency existed.</p>
|
|
|
|
<div class="fortlet-block">
|
|
<h3>Live NASA headlines</h3>
|
|
<fort-news topic="NASA" limit="3"><ul class="fort-news" data-fort="news" data-topic="NASA"><li><a href="https://www.dongascience.com:443/ko/news/79518" rel="noopener">NASA '추락하는 우주망원경' 구조 실패…연말 대기권서 소멸</a> <span class="src">dongascience.com</span></li><li><a href="http://www.newzealandstar.com/news/279255068/miyu-yamashita-cards-8-under-62-and-leads-by-1-in-edmonton" rel="noopener">newzealandstar.com</a> <span class="src">newzealandstar.com</span></li><li><a href="https://timesofindia.indiatimes.com/science/my-best-shot-nasa-astronaut-shares-image-of-2025-maha-kumbh-taken-from-space/articleshow/133372831.cms" rel="noopener">indiatimes.com</a> <span class="src">indiatimes.com</span></li></ul></fort-news>
|
|
</div>
|
|
|
|
<footer>
|
|
<span style="color:var(--accent); font-weight:bold;">ALLEN LORCH</span> · allen-lorch.4ort.net<br>
|
|
Sources: public knowledge graph (Raytheon, assembly-language citation); image Pexels royalty-free. Written by hand, one ticket at a time.
|
|
</footer>
|
|
|
|
</body>
|
|
</html> |