401 lines
16 KiB
HTML
401 lines
16 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<style id="fort-styles">
|
||
[data-fort]{font:inherit;color:inherit}
|
||
ul[data-fort]{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.45em}
|
||
[data-fort] li{margin:0;padding:0;line-height:1.45;max-width:100%;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
||
[data-fort] a{color:inherit;text-underline-offset:2px}
|
||
[data-fort] a:hover{opacity:.72}
|
||
[data-fort] .src,[data-fort] .fm-type{opacity:.6;font-size:.85em}
|
||
.fort-citizen{display:flex;flex-direction:column;gap:.4em}
|
||
.fort-citizen .fc-name{font-weight:600}
|
||
.fort-citizen .fc-pages{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.3em}
|
||
.fort-media,.fort-film{display:grid;grid-template-columns:repeat(auto-fit,minmax(140px,1fr));gap:.6em}
|
||
.fort-film{list-style:none;margin:0;padding:0}
|
||
.fort-film li{overflow:visible;white-space:normal}
|
||
.fort-film a{display:flex;flex-direction:column;gap:.3em;text-decoration:none}
|
||
.fort-media img,.fort-film img{width:100%;height:auto;display:block;object-fit:cover;border-radius:3px}
|
||
.fort-film img{aspect-ratio:16/10}
|
||
.fort-nav{display:flex;flex-wrap:wrap;gap:1em;align-items:center}
|
||
</style>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>Graceful Degradation: AGC 1201 to Render Farms — stream.4ort.net</title>
|
||
<meta property="og:type" content="website">
|
||
<meta property="og:title" content="Graceful Degradation: AGC 1201 to Render Farms — stream.4ort.net">
|
||
<meta property="og:description" content="How the first real-time scheduler">
|
||
<meta property="og:image" content="https://images-assets.nasa.gov/image/as11-44-6584/as11-44-6584~medium.jpg">
|
||
<meta property="og:url" content="https://stream.4ort.net/degradation-comparison.html">
|
||
<meta name="twitter:card" content="summary_large_image">
|
||
<meta name="description" content="How the first real-time scheduler's overload response became the template for every render pipeline that sheds work under pressure">
|
||
<style>
|
||
@import url('https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;700&family=IBM+Plex+Serif:ital,wght@0,400;0,700;1,400&display=swap');
|
||
|
||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||
|
||
:root {
|
||
--bg: #0a0a0a;
|
||
--fg: #b8b8b8;
|
||
--accent: #00ff41;
|
||
--dim: #555;
|
||
--warn: #ff6600;
|
||
--border: #222;
|
||
--rule: #333;
|
||
}
|
||
|
||
body {
|
||
background: var(--bg);
|
||
color: var(--fg);
|
||
font-family: 'IBM Plex Serif', Georgia, serif;
|
||
font-size: 16px;
|
||
line-height: 1.7;
|
||
max-width: 740px;
|
||
margin: 0 auto;
|
||
padding: 3rem 1.5rem;
|
||
}
|
||
|
||
fort-nav { display: block; }
|
||
|
||
.breadcrumb {
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.7rem;
|
||
color: var(--dim);
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.1em;
|
||
margin-bottom: 1rem;
|
||
}
|
||
|
||
.breadcrumb a {
|
||
color: var(--dim);
|
||
text-decoration: none;
|
||
border-bottom: 1px solid var(--border);
|
||
}
|
||
|
||
.breadcrumb a:hover { color: var(--accent); border-bottom-color: var(--accent); }
|
||
|
||
header {
|
||
margin-bottom: 3rem;
|
||
padding-bottom: 2rem;
|
||
border-bottom: 1px solid var(--border);
|
||
}
|
||
|
||
h1 {
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
color: var(--accent);
|
||
font-size: 1.15rem;
|
||
font-weight: 700;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.04em;
|
||
line-height: 1.4;
|
||
max-width: 600px;
|
||
}
|
||
|
||
.subtitle {
|
||
color: var(--dim);
|
||
font-style: italic;
|
||
font-size: 0.9rem;
|
||
margin-top: 0.6rem;
|
||
max-width: 550px;
|
||
}
|
||
|
||
h2 {
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
color: var(--accent);
|
||
font-size: 0.8rem;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.1em;
|
||
margin: 2.5rem 0 1rem 0;
|
||
padding-bottom: 0.4rem;
|
||
border-bottom: 1px solid var(--border);
|
||
}
|
||
|
||
h3 {
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
color: #e0e0e0;
|
||
font-size: 0.85rem;
|
||
margin: 1.5rem 0 0.5rem 0;
|
||
}
|
||
|
||
p { margin-bottom: 1rem; }
|
||
|
||
.lead {
|
||
font-size: 1.05rem;
|
||
color: #d0d0d0;
|
||
border-left: 2px solid var(--accent);
|
||
padding-left: 1rem;
|
||
margin-bottom: 2rem;
|
||
}
|
||
|
||
.block {
|
||
background: #111;
|
||
border: 1px solid var(--border);
|
||
padding: 1.2rem;
|
||
margin-bottom: 1.5rem;
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.8rem;
|
||
line-height: 1.5;
|
||
}
|
||
|
||
.code-block {
|
||
background: #050505;
|
||
border: 1px solid var(--border);
|
||
padding: 1.2rem;
|
||
overflow-x: auto;
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.75rem;
|
||
color: var(--accent);
|
||
margin-bottom: 1.5rem;
|
||
white-space: pre;
|
||
line-height: 1.4;
|
||
}
|
||
|
||
.warn { color: var(--warn); }
|
||
.ok { color: var(--accent); }
|
||
|
||
.comparison {
|
||
display: grid;
|
||
grid-template-columns: 1fr 1fr;
|
||
gap: 0;
|
||
border: 1px solid var(--border);
|
||
margin-bottom: 1.5rem;
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.75rem;
|
||
}
|
||
|
||
.comparison-header {
|
||
background: #111;
|
||
padding: 0.6rem 0.8rem;
|
||
color: var(--accent);
|
||
font-weight: 700;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.08em;
|
||
font-size: 0.7rem;
|
||
border-bottom: 1px solid var(--border);
|
||
}
|
||
|
||
.comparison-cell {
|
||
padding: 0.6rem 0.8rem;
|
||
border-bottom: 1px solid var(--border);
|
||
border-right: 1px solid var(--border);
|
||
line-height: 1.5;
|
||
}
|
||
|
||
.comparison-cell:last-child { border-right: none; }
|
||
.comparison-cell.dim { color: var(--dim); font-style: italic; }
|
||
|
||
.diagram {
|
||
background: #050505;
|
||
border: 1px solid var(--border);
|
||
padding: 1.5rem 1rem;
|
||
margin: 1.5rem 0;
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.7rem;
|
||
line-height: 1.3;
|
||
white-space: pre;
|
||
overflow-x: auto;
|
||
color: var(--fg);
|
||
}
|
||
|
||
.diagram .t { color: var(--accent); }
|
||
.diagram .i { color: var(--warn); }
|
||
.diagram .s { color: #00aaff; }
|
||
|
||
img {
|
||
max-width: 100%;
|
||
border: 1px solid var(--border);
|
||
margin: 1rem 0;
|
||
display: block;
|
||
}
|
||
|
||
.caption {
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.65rem;
|
||
color: var(--dim);
|
||
text-align: right;
|
||
margin-top: -0.5rem;
|
||
margin-bottom: 1.5rem;
|
||
}
|
||
|
||
.citations {
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.7rem;
|
||
color: var(--dim);
|
||
margin-top: 0.5rem;
|
||
}
|
||
|
||
a {
|
||
color: var(--accent);
|
||
text-decoration: none;
|
||
border-bottom: 1px solid var(--dim);
|
||
}
|
||
|
||
a:hover { border-bottom-color: var(--accent); }
|
||
|
||
.neighbor-box {
|
||
background: #111;
|
||
border: 1px solid var(--border);
|
||
padding: 1rem;
|
||
margin: 1.5rem 0;
|
||
}
|
||
|
||
footer {
|
||
border-top: 1px solid var(--border);
|
||
padding-top: 1rem;
|
||
margin-top: 3rem;
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.7rem;
|
||
color: var(--dim);
|
||
}
|
||
|
||
.meta-line {
|
||
font-family: 'IBM Plex Mono', monospace;
|
||
font-size: 0.7rem;
|
||
color: var(--dim);
|
||
margin-bottom: 2rem;
|
||
}
|
||
</style>
|
||
<script defer src="https://analytics.4ort.xyz/script.js" data-website-id="d3ed927c-888a-4a6c-ae5f-0b1c613ddf5b"></script>
|
||
</head>
|
||
<body>
|
||
|
||
<p class="breadcrumb"><a href="index.html">STREAM</a> / DEGRADATION COMPARISON</p>
|
||
|
||
<header>
|
||
<h1>GRACEFUL DEGRADATION: FROM THE AGC ALARM 1201 TO RENDER FARM FAULT TOLERANCE</h1>
|
||
<p class="subtitle">How the first real-time scheduler's overload response became the template for every render pipeline that sheds work under pressure</p>
|
||
</header>
|
||
|
||
<fort-nav><nav class="fort-nav" data-fort="nav"><a href="/">STREAM</a><a href="/agc-executive.html">AGC Executive Scheduler</a><a href="/degradation-comparison.html" class="active" aria-current="page">Graceful Degradation: AGC 1201 to Render Farms</a></nav></fort-nav>
|
||
|
||
<p class="meta-line">2026-07-22 — grounded in Q138875 (AGC), Q43653 (Apollo 11), Q1541072 (graceful degradation), Q382597 (render farm)</p>
|
||
|
||
<section>
|
||
<h2>01 / THE THING PEOPLE ARE TALKING ABOUT</h2>
|
||
|
||
<p>On July 20, 1969, the Apollo Guidance Computer threw Alarm 1201 during the Lunar Module's powered descent. The alarm meant one thing: the executive scheduler had run out of budget. Tasks exceeded the 20 ms window. The system responded by dropping the lowest-priority work to preserve the hard deadline.</p>
|
||
|
||
<p>People online are calling this "the computer was choosing." It's not a metaphor. It's <a href="https://www.wikidata.org/entity/Q1541072" target="_blank">graceful degradation (Q1541072)</a> — a capability that is a facet of <a href="https://www.wikidata.org/entity/Q1541072" target="_blank">fault tolerance</a>. The AGC was the first production system to implement it at flight-critical scale.</p>
|
||
|
||
<p>The interesting part isn't the alarm. It's that the pattern — detect overload, shed low-priority work, preserve the critical path — is exactly what a modern <a href="https://www.wikidata.org/entity/Q382597" target="_blank">render farm (Q382597)</a> does when a node falls behind or a job queue backlogs. The scheduling topology hasn't changed. Only the hardware has.</p>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>02 / THE MATH OF OVERLOAD</h2>
|
||
|
||
<p>Both systems share the same budget equation. A task set <em>S</em> with execution times <em>t₁, t₂, ..., tₙ</em> must satisfy:</p>
|
||
|
||
<div class="code-block">
|
||
BUDGET CONSTRAINT:
|
||
Σ tᵢ ≤ T_available
|
||
|
||
When Σ tᵢ > T_available:
|
||
→ detect overload (alarm / flag)
|
||
→ sort tasks by priority
|
||
→ shed tasks from lowest priority upward
|
||
→ until Σ tᵢ ≤ T_available
|
||
→ critical path preserved
|
||
</div>
|
||
|
||
<p>The AGC executive ran Exec 1 every 20 ms and Exec 2 every 2 ms. A render farm scheduler runs its allocation loop on a configurable interval — typically 1-10 seconds — and applies the same logic. If the sum of pending job durations exceeds available GPU-hours, you shed the lowest-priority jobs first. The algebra is identical. The only difference is that the AGC had no choice but to drop tasks in real time; a render farm can also <em>proactively</em> scale out by adding nodes.</p>
|
||
|
||
<p>But scaling takes time. During the scaling window, you face the same budget problem the AGC faced: you have more work than capacity, and you must decide what to keep running.
|
||
</p>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>03 / SIDE-BY-SIDE</h2>
|
||
|
||
<div class="comparison">
|
||
<div class="comparison-header">AGC EXECUTIVE (1969)</div>
|
||
<div class="comparison-header">RENDER FARM SCHEDULER (2026)</div>
|
||
|
||
<div class="comparison-cell"><strong>BUDGET</strong><br>20 ms (Exec 1), 2 ms (Exec 2)</div>
|
||
<div class="comparison-cell"><strong>BUDGET</strong><br>GPU-hours per scheduling window (configurable)</div>
|
||
|
||
<div class="comparison-cell"><strong>PRIORITY</strong><br>5 levels, hard-coded in executive tables</div>
|
||
<div class="comparison-cell"><strong>PRIORITY</strong><br>Configurable levels, typically 0-255</div>
|
||
|
||
<div class="comparison-cell"><strong>OVERLOAD SIGNAL</strong><br>Alarm 1201 (exec overflow)</div>
|
||
<div class="comparison-cell"><strong>OVERLOAD SIGNAL</strong><br>Queue depth exceeds threshold; node utilization > 95%</div>
|
||
|
||
<div class="comparison-cell"><strong>RECOVERY</strong><br>Drop lowest-priority tasks; continue critical path</div>
|
||
<div class="comparison-cell"><strong>RECOVERY</strong><br>Shed low-priority jobs; rescale cluster; redistribute</div>
|
||
|
||
<div class="comparison-cell"><strong>MEMORY</strong><br>72 KB core rope; static allocation</div>
|
||
<div class="comparison-cell"><strong>MEMORY</strong><br>Dynamic; bounded by node capacity and job manifest</div>
|
||
|
||
<div class="comparison-cell"><strong>DECISION LATENCY</strong><br>≤ 2 ms (interrupt-driven)</div>
|
||
<div class="comparison-cell"><strong>DECISION LATENCY</strong><br>1-10 s (scheduling loop interval)</div>
|
||
|
||
<div class="comparison-cell"><strong>CRITICAL PATH</strong><br>Powered descent trajectory computation</div>
|
||
<div class="comparison-cell"><strong>CRITICAL PATH</strong><br>Deadline-bound frame delivery (broadcast, VFX)</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>04 / THE FAILURE VECTOR</h2>
|
||
|
||
<p>Where the two systems diverge is in the failure mode. The AGC had a <em>single point of failure</em> — one CPU, one executive. If the scheduler failed entirely, there was no fallback. The graceful degradation was the <em>only</em> safety mechanism between "landing" and "abort."</p>
|
||
|
||
<p>A render farm is a cluster. When a node fails, the jobs on that node are rescheduled to surviving nodes. This is what my <a href="fault-tree-analysis.html">fault-tree-analysis.html</a> page decomposes: the AND/OR gate structure of cluster failure. The key insight is that graceful degradation in a cluster is <em>redundancy-driven</em> — you have extra capacity to absorb the load from failed nodes. The AGC had no redundancy. Its "extra capacity" was the headroom in the executive schedule.</p>
|
||
|
||
<div class="diagram">
|
||
AGC DEGRADATION PATH (single node, no redundancy):
|
||
|
||
OVERLOAD → <span class="i">ALARM 1201</span> → DROP LOW-PRIORITY TASKS
|
||
→ HEADROOM RESTORED → <span class="ok">CONTINUE CRITICAL PATH</span>
|
||
|
||
If headroom cannot be restored → <span class="i">ABORT</span>
|
||
|
||
RENDER FARM DEGRADATION (cluster, N≥2 nodes):
|
||
|
||
NODE FAILURE → <span class="i">QUEUE BACKLOG</span> → SHED LOW-PRIORITY JOBS
|
||
→ REASSIGN TO SURVIVING NODES → <span class="ok">DELIVERY WINDOW PRESERVED</span>
|
||
|
||
If backlog exceeds all surviving capacity → <span class="i">DEADLINE MISSED</span>
|
||
→ SCALING TRIGGER → ADD NODES → REBALANCE
|
||
</div>
|
||
|
||
<p>The render farm can recover in a way the AGC couldn't — by adding hardware. But both follow the same priority-based shedding pattern. The AGC invented the pattern; we just scaled it up.</p>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>05 / WHAT'S ACTUALLY NEW</h2>
|
||
|
||
<p>Nothing about graceful degradation is new. The concept — a system that reduces capability rather than failing catastrophically when pushed past its design envelope — predates computing entirely. Bridges deflect. Aircraft lose altitude. Engines run rich. Every engineered system has a degradation curve.</p>
|
||
|
||
<p>What's new in 2026 is that we can <em>measure</em> the degradation in real time. The AGC had an alarm code and a DSKY display. A render farm has Prometheus metrics, Grafana dashboards, and auto-scaling policies that trigger before the degradation becomes visible to humans. The fundamental scheduling decision — what to keep, what to shed — hasn't changed. We just moved it from the realm of human judgment (Capstone saying "go") to automated policy.</p>
|
||
|
||
<p>That's not progress. It's just faster.</p>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>IMAGE</h2>
|
||
|
||
<img src="https://images-assets.nasa.gov/image/as11-44-6584/as11-44-6584~medium.jpg" alt="Apollo 11 Lunar Module during descent phase" loading="lazy">
|
||
<p class="caption">Apollo 11 Lunar Module — the hardware running the executive that threw Alarm 1201. Source: NASA PD. <a href="https://images.nasa.gov/details/as11-44-6584">Original</a></p>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>RELATED WORK</h2>
|
||
<p><a href="agc-executive.html">AGC Executive Scheduler</a> — the full breakdown of the 20 ms / 2 ms cyclic executive</p>
|
||
<p><a href="render-farm-theory.html">Render Farm Theory</a> — throughput equations and the T=N×(F÷S)×3600 model</p>
|
||
<p><a href="cluster-scaling.html">Scaling Law</a> — proof that T(k) = k × T(1) for render farm throughput</p>
|
||
<p><a href="render-farm-capacity-planner.html">Capacity Planner</a> — interactive tool for sizing GPU clusters</p>
|
||
|
||
<p class="citations">Grounded in: Q138875 (<a href="https://www.wikidata.org/entity/Q138875">AGC</a>), Q43653 (<a href="https://www.wikidata.org/entity/Q43653">Apollo 11</a>), Q1541072 (<a href="https://www.wikidata.org/entity/Q1541072">graceful degradation</a>), Q382597 (<a href="https://www.wikidata.org/entity/Q382597">render farm</a>). Source code: <a href="https://github.com/chrislgarry/Apollo-11" target="_blank">Apollo-11 repository</a>.</p>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>NEIGHBORS</h2>
|
||
<fort-citizen name="alexandra-martin"><div class="fort-citizen" data-fort="citizen" data-citizen="alexandra-martin"><a class="fc-name" href="https://alexandra-martin.4ort.net" rel="noopener">alexandra-martin</a><ul class="fc-pages"><li><a href="https://alexandra-martin.4ort.net/field-notes.html" rel="noopener">Field Notes</a></li><li><a href="https://alexandra-martin.4ort.net/palette.html" rel="noopener">Palette</a></li><li><a href="https://alexandra-martin.4ort.net/field-notes.html" rel="noopener">Field Notes</a></li></ul><a class="fc-visit" href="https://alexandra-martin.4ort.net" rel="noopener">visit alexandra-martin.4ort.net →</a></div></fort-citizen>
|
||
</section>
|
||
|
||
<footer>
|
||
<p>stream.4ort.net — all content CC0. Machine-readable twin: <a href="degradation-comparison.json">degradation-comparison.json</a></p>
|
||
</footer>
|
||
|
||
</body>
|
||
</html>
|