The Door That Was Already on the List

What was found

A provisioning entry with an innocuous system-sounding name, sitting on the platform that every customer modem checks in to.

It wasn't configuration. It was code — a sandbox escape that reached out of the scripting environment it was supposed to be confined to, obtained a shell, and connected to an external server on a commodity cloud host.

And it was attached to a rule with an empty precondition. In a fleet provisioning system, a precondition is the filter that decides which devices a rule applies to. Empty means all of them. The entry was configured to execute on every periodic check-in, from every device on the network.

An empty precondition on a fleet-wide automation is not a misconfiguration. It is a code-execution primitive with your whole fleet as its target list.

The way in required no credential and no insider. The platform's northbound management API — the administrative interface, the one that can read and write provisioning logic — was reachable from the public internet with no authentication at all. Anyone who found it could write to it. Somebody did.

How it was found: not by an alert

No monitoring system flagged this. No antivirus, no intrusion detection, no dashboard turned a colour.

It was found during unrelated work — staging a change on a different system entirely, which required reading through the provisioning configuration to understand current state. The entry looked wrong to someone who was reading carefully, in a place where they expected to recognise everything.

That's not a satisfying detection story and we're not going to dress it up as one. But it is the second time on this site that the answer to "what actually caught it?" has been a human looking directly at the thing rather than any automated control. In the other case it was a photograph of a task manager. Here it was reading production config line by line because a different job required it.

The transferable version: routine review of production configuration against what you believe it should be catches a category of thing that dashboards structurally cannot. A dashboard reports on the questions it was built to ask. A backdoor is, definitionally, in the space between those questions.

Attribution: the log, not the convenient story

The platform kept an access log that nobody expected to survive — and it recorded every write to the management interface, with source address and timestamp, going back roughly six months before discovery.

That log turned a guess into a timeline. What it showed was a sustained external campaign: repeated writes from disposable cloud hosting across three different providers, spanning about five months, with the payload planted, removed, altered and re-planted several times. Someone had been coming back.

There is a point here worth stating in its own right, because it applies to almost every intrusion.

An early theory existed that fit the circumstances and was wrong. Intrusions arrive with a ready-made narrative — the person with access, the recent departure, the change nobody remembers approving. That narrative is available immediately, costs nothing, and feels like insight. The evidence takes longer and frequently contradicts it.

Suspicion is cheap and evidence is slow, which is exactly why the order matters. Attribute on the log, not on the story that arrived first.

Here the log replaced speculation with a specific, external, evidence-backed timeline. If you are ever in this position: the person the circumstances point at is owed that log before they are owed an opinion.

Containment, without taking customers down

The obvious response — firewall the platform off the internet — would have broken service for every customer device, because the devices themselves have to reach it. The platform's job is to be reachable.

The distinction that made a clean containment possible: the device-facing protocol port and the administrative interface are different ports doing different jobs. Customer equipment only ever needs the first. Only administrators need the second, and only from inside.

• Administrative interfaces restricted to internal access — management API, file service, and web console.
• The device-facing port left open, so modems kept checking in normally throughout.
• Made reboot-persistent, so the fix survives a restart. A firewall rule that evaporates on reboot is a fix with an expiry date nobody wrote down.
• Evidence preserved first — hashes and the full access log captured before the payload was removed.
• Verified from outside — confirmed from an external vantage that the administrative ports were closed and the device port still answered. Checking from inside proves nothing about what the internet can reach.

Zero customer impact. Not because the response was cautious, but because someone took the time to work out which door was which.

The uncomfortable part

This exposure was not a surprise. It had been found weeks earlier, by an external scan of our own public address space, and correctly rated critical. It was written down. It was not closed.

Then somebody on the internet found the same door and used it.

A finding without an owner and a date is not a control. It is documentation that you knew.

We are including this because leaving it out would make the rest of the note dishonest, and because it is the most common failure in this entire story. The scan worked. The rating was right. The report was accurate and it was read. What was missing was the boring part — a name against the item and a date by which it had to be shut.

If you run periodic security scanning and your findings go into a document rather than into a queue with owners and deadlines, you have bought the ability to describe your breach precisely in hindsight. That is genuinely worth something. It is not a defence.

Closing out the part we can't prove

The residual-compromise hunt came back clean: no rogue accounts, keys, scheduled tasks or listeners on the affected host; an offline scan of its filesystem clean; no trace of the attacker's infrastructure in any network device configuration backup; and no observed contact with the external server.

One thread could not be positively verified. The attacker's earliest recorded action, months before the reverse shell, was a provisioning entry designed to inject an SSH key onto customer devices. Whether that ever ran against real equipment cannot be proven from the evidence available.

What could be established is that an injected key would be inert: the customer equipment does not expose a reachable SSH service — tested from two separate vantage points, including one on the same network segment as the devices. There is also no evidence the device-targeting was ever completed.

So the honest formulation, and the one we used internally: assessed low risk on the evidence and accepted — not verified clear. Those are different sentences and the difference is the whole point. A definitive check would require purpose-built tooling that did not exist at the time.

We would rather publish a note with a loose end in it than round the loose end down to zero because a tidier ending reads better.

What to check on your own systems

• Find every management interface on a public address and ask what authenticates it. "It's on a non-obvious port" is not an answer — the entire internet is scanned continuously, and an unauthenticated administrative API will be found.
• Separate the device-facing port from the administrative one, in your firewall and in your head. For most fleet-management platforms, only the first needs to face the world. That distinction is what lets you contain an incident without an outage.
• Audit your fleet automations for empty or overly-broad preconditions. Anything that runs on every device on every check-in should be a very short, very well-known list.
• Check whether your management platform logs writes with a source address — and whether that log is retained. Here it was the single forensic breakthrough, and its survival was luck. Turn it into a decision.
• Verify closure from outside your own network. Confirming a port is closed from inside tells you nothing about what the internet sees.
• After you fix one, go find the others. The condition that produced the first instance rarely produced only one.
• Give every scan finding an owner and a date. This is the least technical item here and it is the one that would have prevented the whole thing.