OT Horror Stories blog image
Blog

The Night Shift: Five OT Horror Stories That Don't Need Monsters

Every OT engineer knows the sound of a Friday going sideways. It starts with somebody looking at a screen and saying, very quietly, “Huh.”

Then someone opens the network drawing. Someone else finds a change ticket from 2017. A third person asks whether the engineer who installed the thing still works here.

Nobody is leaving early.

People love Halloween horror stories. Ghosts, monsters, abandoned buildings, things making noises behind locked doors. Fine. But if you work in OT, you’ve seen enough terrifying things in perfectly well-lit electrical cabinets to last a lifetime.

So here’s our Halloween offering: five small nightmares from the industrial world. The scenes are fictionalized. The problems are real. If you recognize your facility in one of them, I’m sorry. Maybe buy the night shift some donuts.

The Connection That Wouldn’t Die 🧟

At 2:13 every morning, a remote connection woke up, exchanged a little traffic, and went quiet again.

It had done this for years. Nobody complained. Nobody even knew it existed until a network engineer, trying to reconcile a firewall rule with a current vendor list, asked a fairly reasonable question.

“Who owns this?”

The answer was an archaeological expedition through old change tickets, purchase orders, and an email chain that ended with, “We’ll remove it after commissioning.”

Commissioning had ended in 2020.

The integrator had changed names twice. The access account still worked. The original project folder thoughtfully labeled the connection TEMPORARY.

Apparently, it had better job security than the rest of us.

Before anyone unplugged anything, an operator asked what the connection supported. Good question. It was a legitimate remote-support path that had outlived its documentation. The team reviewed who needed it and whether it could stay disabled outside approved support windows.

That was the comparatively easy ghost. The harder one was the cellular modem someone found during the cabinet walkdown. It had its own connection, completely separate from the network everyone had been monitoring.

Network monitoring cannot report traffic it never sees. Sometimes the right discovery tools are a pair of boots, an old invoice, and the right cabinet key.

Defeating the monster: Check the paths you can observe, then go looking for the ones you can’t. Network evidence, firewall and access reviews, procurement records, vendor conversations, and physical inspections all have a job. We go deeper into that distinction in How to Find Undocumented External Connections in Your OT Network.

The Skeleton Key 💀

The remote-access logs showed a successful session.

Successful authentication. Valid account. No failed password attempts. The door had opened exactly as designed.

The only problem was that the account belonged to an integrator who said nobody from their team had logged in that night.

Someone suggested changing the password. Someone else wanted to check whether the integrator had confused time zones again. Both were sensible ideas. The operations lead asked a better question:

“What did the session actually do?”

The group compared the remote-access logs with the available workstation records and session activity. The connection had reached an engineering workstation, and the recorded activity showed software normally used during maintenance accessing equipment that wasn’t on the work order. None of it alone proved malicious activity. Together, it warranted investigation before authorizing another session.

Somewhere, a security scanner was still proudly reporting that the workstation had no critical missing patches.

Congratulations. The key worked.

A successful login confirms that the system accepted the credentials, not that the person using them was authorized to perform that work. That matters in OT, where vendor access is often essential to operations.

Defeating the monster: Know whose accounts exist, who authorizes access, and what normal work looks like after login. Use individual identities and strong authentication where supported, review sessions, and investigate activity in operational context. For the longer version, see OT Credential Abuse: They Logged In. They Didn’t Break In.

The Dashboard That Cried Wolf 🐺

The night-shift analyst had 10,412 alerts.

This sounded impressive in the quarterly report. On the screen it was less impressive. Most were variations of yesterday’s alerts, which were variations of the alerts from the day before. A protocol had been observed. A device had communicated. Something had changed somewhere, probably.

The dashboard was spectacularly good at counting.

One evening, a new alert joined the pile: an engineering workstation had issued a write command to a controller outside its usual maintenance window.

It could be an urgent repair, an unrecorded change, or a reason to investigate immediately. The command alone doesn’t settle the question.

Unfortunately, the alert was presented with precisely the same urgency as 86 other messages that essentially said, “Yes, the PLC is still talking.”

The analyst called the plant. The operator knew the workstation, but hadn’t heard of planned work on that controller. Now they had a starting point: a device, an action, a time, and someone who understood the process.

By the time it hit 10,412, the alert counter had become a vanity metric with its own login.

You can buy a smoke alarm that screams every time you make toast. Technically, it works. Eventually, you stop trusting it. A useful alarm earns attention because it tells you something you can act on.

Defeating the monster: Keep the underlying evidence, but prioritize what matters in the actual environment: the asset, the behavior, the expected operating pattern, and the people who can validate it. We’ve talked about the consequences of losing that trust in No Noise. Just Signal: Chaos Reduction in OT Networks.

The Switch Nobody Dared to Flip 😱

The message arrived late on a Friday: a service provider with remote access to the facility was investigating a compromise of its own systems.

There was no confirmed evidence of activity inside the plant. There was also no reliable answer to whether that vendor’s access had been used by anyone other than the vendor.

Security wanted the connection disabled until somebody could sort it out.

Operations had one question: “What breaks if we do?”

Nobody could give them a complete answer.

One diagram showed the vendor’s remote-support tunnel. Another showed historian replication. A third was titled Final Network Architecture and had a filename ending in _FINAL_v7_revised.

The emergency procedure had a beautifully formatted cover page. The part that identified which connections could be isolated independently said TBD.

There are horror movies with less convincing plot twists.

After some investigation, the team determined they could revoke that vendor’s access at the remote-access gateway without dropping other traffic. Good. But they also found that the procedure hadn’t been tested, and nobody had assigned ownership for approving the change outside business hours.

They spent the first hours of their response discovering options they should have practiced months earlier.

In OT, an isolation decision can affect remote support, monitoring, business-system data, or processes that depend on external services. Some disconnections are straightforward. Others deserve engineering review, a defined fallback, and a practiced procedure.

Defeating the monster: Know the connections, their dependencies, the authority to isolate them, and the operational consequences. Test the lowest-impact option in a controlled window before an incident forces the conversation. Your OT Isolation Plan: What You Can Actually Disconnect When It Matters walks through the choices.

The Phantom PLC 👻

Engineering counted 46 controllers.

The asset spreadsheet listed 47.

The monitoring team had 48 possible controller identities from observed traffic, several still unverified.

The meeting lasted ninety minutes. Everyone left with the same numbers they brought in.

The argument centered on an address associated with the old pump house. The engineering drawing said the equipment was removed during an upgrade. A more recent maintenance record mentioned a replacement panel. Nobody had updated the inventory because, naturally, everybody assumed somebody else had.

They checked the monitoring point’s coverage, reviewed addresses and protocol evidence, then visited the pump house with an operator.

There it was. A controller in a cabinet. Properly wired. Doing its job. With a laminated label dating to a project two revisions older than the drawing.

The ghost wasn’t haunting the plant. It was running production while three departments argued about whether it existed.

An observed network address is not automatically a confirmed physical controller. And a controller that hasn’t communicated during the collection window may still be in a cabinet. Different capture points, stale drawings, and unresolved identities can all explain inventory disagreements.

Defeating the monster: Use passive observations as evidence, validate identities, compare them with engineering records, and have operators confirm what’s really installed. Where passive data is incomplete, work with operations to choose suitable additional discovery methods. The Basics Behind Building an Asset Inventory in OT Environments is a good place to start.

The Part Where Everyone Survives

Every one of these stories ends with a person asking a sensible question.

Who owns that connection? What did the account do? Which alert matters? What stops if we isolate this path? Is that controller actually in the building?

The ending is less cinematic than Hollywood would prefer. Engineers, operators, and defenders compare what they think is true with what the environment is actually doing. Somebody updates a procedure. On a good day, somebody even updates the drawing.

Good OT security depends on context, people who understand the process, and the ability to act without making a bad day worse.

As for the monsters, leave those to Halloween. The undocumented VPN can come off the permanent guest list.

Happy Halloween from all of us at EmberOT. And please, for the love of everything that runs on a maintenance window, update the drawing.