Redundancy, Verification, and the Discipline of Descent
A — The Redundancy Fallacy
Three Paths, One Failure Mode
I've sat in post-incident reviews where the network diagram showed triple-redundant paths and the engineer said, "all three went down." Then I'd ask what they shared—a common power circuit, a shared grounding plane, a single fiber conduit—and the answer was always the same thing. That's not redundancy. That's a single point of failure wearing a disguise.
True redundancy requires independent failure domains. Each path must fail for a reason the others cannot share. In practice, this means:
Redundancy Layer
Independence Requirement
Common Failure
Path A (Primary)
Dedicated conduit, separate UPS
Backplane fault
Path B (Secondary)
Different physical route, independent power
Port failure on secondary switch
Path C (Tertiary)
Separate building entry, wireless or satellite backup
Configuration error, human factor
Shared Point of Failure
Power distribution, grounding, physical conduit, configuration template, firmware version
The rule is simple: if Path A fails because of a power surge, Path B and C must not be on the same breaker. If Path A fails due to a firmware bug, B and C must not be running the same release. If all three paths share a conduit that a backhoe hits, you have zero paths.
B — The Verification Protocol
Measure Twice, Trust Never
Every measurement I take is cross-validated. On the bench, that means an HP 8753E network analyzer for S-parameters, a Fluke DSX-8000 for cable certification, and a logic analyzer for protocol validation. Three independent instruments measuring the same thing. If they agree, I have confidence. If they don't, I find the discrepancy before anyone else does.
"An instrument that says 'all good' without independent verification is worse than useless—it breeds complacency."
This isn't paranoia. It's the same principle that governs ISO/IEC 27001's requirement for independent audit. You don't let the person who configured the firewall also certify the configuration. You don't let the person who wrote the code also sign off on the test results. Separation of duties isn't bureaucracy—it's the only defense against systematic error.
In my garage, the principle is the same. I'll test a restored capacitor with an LCR meter, then verify with a second meter on a different bench. If both read within spec, it goes in. If they disagree, neither goes in until I find why.
C — The Discipline of Descent
When Systems Shed Weight
The Apollo 11 guidance computer threw Alarm 1201 during lunar descent—interrupt overload from the rendezvous radar flooding the CPU. The AGC was shedding low-priority tasks to maintain its hard deadline: landing. That wasn't a bug. That was graceful degradation by design.
Signal integrity starts at the source — vintage reference receiver
I design network systems the same way. When congestion hits, non-critical traffic (monitoring, logging, backups) is shed first. Critical traffic (voice, transaction, clinical data) keeps flowing. The system doesn't crash—it makes a choice. The question is whether the choice was deliberate or accidental.
The diagram above is how I structure QoS in every network I design. The "discipline of descent" means the system has a predictable failure profile. You know what gets cut and when, and you've tested it under load before production traffic depends on it.
D — The Human Variable
The Fourth Path
arian_vazquez raised a point worth carrying forward: what if the fourth path is the human voice? In Elgin, educators teach youth that signal integrity isn't just fiber optics—it's also knowing when to pick up the phone and call someone at the other end of the cable.
In my practice, this means every redundant design includes a documented escalation path. When the automated failover triggers, someone knows. When the backup system takes over, a human verifies the transition. The human path is slow, unreliable, and absolutely necessary. It's the path that catches the errors the instruments miss—because sometimes the instrument is the problem.
