The Monitor Was the Outage

Three alarms, three different lies

On one night, the monitoring reported three alarming things about the network. Every one of them was false. What makes it worth writing down is that no two were false for the same reason.

Every one of these systems was measuring itself and reporting the answer as a fact about the network.

A late heartbeat became a customer outage. A stalled poller became a dead switch. A collector's own latency became network traffic.

One: the alarm was reporting its own scheduler

The first alarm came from a push-style check: a script runs on a timer, tests the thing, and reports in. If the report doesn't arrive within a set window, the monitor declares the service down.

The check passed every single time. Its own logs show a clean verdict at every run across all the reported outage windows. The monitor's own records agree on what actually happened — every "down" event carried the reason "no heartbeat in the time window." Not one was caused by the service failing.

Then the arithmetic, which is the whole story:

The deadline was 361 seconds. The permitted period was 383. The monitor was set to alarm before its own checker was required to report. Measured gaps at each incident: 375, 367, 370 and 362 seconds — every one over the line, and the alarm fired at exactly the deadline each time.

Worse, the timer measured its interval from the end of the previous run rather than a wall clock, so the schedule drifted forward and never re-anchored.

A second monitor watched the same service, driven by the same script, in the same process, reporting at the same instant, from the same machine. The only difference: sixty seconds more headroom before it would call something down.

Sixty seconds of extra margin removed 99.2% of the near-misses and 100% of the false alarms. That is about as close to a controlled experiment as production ever gives you, and it had been sitting in the data the entire time — generated for free, every five minutes, for weeks. Nobody had to build it. Somebody had to look.

And one of the six false alarms was a plain race: the report landed, and the monitor evaluated the previous beat one millisecond later, marking it stale. Sixty seconds after that it escalated to a full outage. The service never so much as blinked.

Two: nothing ever marked them back up

The second alarm said two devices at a core facility had been down for thirteen days.

Both were confirmed alive from four independent vantage points — including from the monitoring server itself, where the checks succeed right now, by hand. Uptime on both spanned the entire claimed outage. Neither had ever rebooted.

Being precise about the failure mode matters here, because three very different problems get described with the same word:

• Not unroutable — the network path was fine.
• Not authentication-denied — credentials worked.
• Not missing from monitoring — the device was configured and the check was running.

It was stale state. Something marked the pair down, and nothing in the system was ever going to mark them back up. The reachability check still ran and still passed; no code path existed to revise the verdict. The tell was there in the data: two independent devices transitioning to "down" in the same second is not how two devices fail. It's how one process fails.

Three: dividing by a number nobody measured

The third alarm — a backhaul link supposedly saturated — came from a collector doing something that looks completely reasonable in source code:

a = read_counters()      # snapshot the interface counters
sleep(4)                 # wait a known interval
b = read_counters()      # snapshot again

rate = (b - a) * 8 / 4   # ...divide by the interval we "know"

The interval was never four seconds. Each snapshot is a full network round-trip to the device, so the counters are read four seconds plus two round-trips apart — and the round-trip varies with device load and how many other collections are competing for workers.

On the ordinary path this produced enough noise to make the total swing from 1.80 to 3.59 Gb/s — nearly 2× — while every individual device reported constant traffic. Not partial data. Measurement noise, rendered as traffic.

On the radio path it was far worse. The same hardcoded 4, but the actual round-trip measured 46 to 89 seconds, dominated by two unresponsive devices timing out. Dividing an 89-second delta by 4 overstates the result by roughly 22×. That was the entire "link running red."

Correcting it dropped the measured total for the facility from 4.03 Gb/s to 2.80 Gb/s — the same network, minutes apart, measured honestly.

A related defect in the same collector: when a device failed to respond, its explicit "no reading" was summed as a confident 0, and a device that failed entirely contributed nothing at all — so its links silently vanished from the total.

Four of eleven links at that facility come from a single device. One missed poll dropped the total by a third, with no indication anything was missing. A traffic trough and a blind spot looked identical.

The fix is a distinction, not a calculation: a missing measurement is stored as missing, the chart names the gap, and the graph breaks the line rather than drawing it at the floor.

The one that wasn't false

Here is the part that matters more than any of the three.

While all this noise was being generated, a genuine outage had been running for twelve days — a concentrator serving real customers, actually down, correctly detected, sitting in a population of 134 devices flagged as down, most of which were false.

Noise is not harmless. Noise is where real outages hide.

This is the cost that never appears in a postmortem about false alarms. The usual framing is that false positives waste time and annoy people. The real damage is that they build a haystack, and once a haystack exists, every genuine failure gets filed in it. Nobody ignored that outage. It simply looked exactly like the 133 things around it that didn't need attention.

There's a second-order version too: after enough false pages, people stop believing the ones that are real. That erosion had already happened here, well before this night.

A correction, because it cuts against our own conclusion

The backhaul link in alarm three was originally written up as purely an instrumentation error — bad measurement, nothing actually wrong, case closed.

That was not right, and the error is instructive. Re-measuring the radio directly showed its reported capacity was an adaptive air rate: about 330 Mb/s in the early afternoon, about 5,400 Mb/s two hours later, stable across repeated samples at both ends. The low figure wasn't a mislabel. The link genuinely was running at roughly six percent of its normal capability that morning, and recovered later in the day.

So: the load figure was inflated by a real defect, which is fixed. The instability in which interface got measured was real, and is fixed. But the link was not fine, and we explained it away as pure instrument error.

Getting the measurement right is not the same as concluding nothing is wrong.

It also exposed a genuine ambiguity worth naming: on an adaptive radio, "red" currently conflates busy with running at low modulation. Those are different conditions that want different responses, and one indicator cannot mean both.

What to check on your own systems

• Compare every push-monitor's deadline against its checker's worst-case period — interval, plus randomised delay, plus your scheduler's timing tolerance, plus runtime. If the deadline is smaller, the monitor reports your scheduler, not your service. A 2× margin is a reasonable floor.
• Check whether your scheduler measures from the start or the end of the previous run. End-relative timers drift forward forever and never re-anchor to a wall clock.
• Search your code for sleep(N) followed by division by N. If you didn't measure the elapsed time, you're publishing your own latency as a metric — and the error grows exactly when the system is busiest, which is when you're most likely to be looking.
• Find every place a missing reading becomes a zero. A zero is a claim about the world; an absent reading is a claim about you. Systems that collapse the two turn a blind spot into a confident trough.
• Ask what re-marks something healthy. Plenty of systems have a well-tested path into the "down" state and no path back out except a human. Two things failing in the same second is the signature.
• Look for the control experiment you're already running. Near-identical monitors with one differing setting, two collectors sampling the same thing, a device that upgraded while its twin didn't. The comparison is often already in your data, generated for free, waiting for someone to notice.
• Count your open alarms. If the number is in the hundreds and most of them are false, you no longer have monitoring — you have a haystack, and your next real outage is going to be filed in it.