Failure Is Like Onions: Physical Security Lessons for Your Server Room

Unlocked server room doors lead to chaos

Stop me if you have heard this before. A company suffers a massive IT outage. The culprit is not a hacker in a foreign country or a sophisticated piece of malware. It is an unlocked door.

Over the weekend, someone wandered into the server room and unplugged a critical uninterruptible power supply (UPS). That UPS powered the storage array hosting all of the company’s VMware virtual machines. When it failed, the storage disappeared from under the VMs. The result was a complete outage that took days to resolve.

Who did it? Nobody knows. There were no cameras. No access control. No accountability. The UPS

was not configured to notify anyone of the outage.

It sounds almost comical until you add up the cost. Days of downtime. Thousands of lost hours. A shaken reputation.

As a student of history, I recall the naval historian Drachinifel and his description of the Mark 14 torpedo in World War II. He called it “Failure is like onions.” The torpedo had two detonators and neither worked well. Its gyroscope sometimes stuck, causing it to run in circles and head back toward the submarine that launched it. Worse, the Navy Bureau in charge refused to admit the problems.

This is not just about torpedoes. It is about cybersecurity and physical security. Failures rarely occur in isolation. They stack like layers of an onion. Alone, each weakness might not cause disaster. Together, they make it inevitable.

In the server room story, multiple failures combined:

• The server room door was unlocked.

• No card key or biometric system logged who entered.

• No cameras monitored activity.

• The UPS had no monitoring or alerting.

• The infrastructure was not designed to fail safely.

Each of these problems is solvable. None require a massive budget. But stacked together, they created a business crisis.

So how do you prevent this in your own organization? Let us walk through the principles of physical security for server rooms and data rooms.

Location Matters

Start with location. A server room should not sit in a busy hallway or next to the break room. The fewer people who walk by, the less curiosity and temptation.  The less likely it is that someone grabs the server room door handle in an honest mistake of seeking the room next door, the easier it is to respond to reconnaissance and early intrusion attempts.

Ideally, your server room sits in a quiet part of the building. Even better if most employees do not know exactly where it is. Out of sight is out of mind.

When building or renovating office space, treat the server room as a security decision, not just a facilities decision.

Control Entry

Access control is critical. A simple key lock is better than nothing, but it does not scale and it does not log activity. If someone copies a key or borrows it, you will never know.

Modern systems use card keys, fobs, or biometrics. The key benefit is not only restricting entry but recording who entered and when. Logs deter misconduct and give you a starting point when investigating issues.

Limit access to those who truly need it. Not every IT employee requires a pass. Not every contractor deserves a card. If someone needs one time access, escort them. The fewer people with unsupervised access, the safer the server room.

Monitor Activity

Access logs tell you who entered. Cameras show what they did.

Position cameras at entry points and on critical racks. Store footage securely and review it when needed. The goal is not to watch employees, it is to protect infrastructure that runs the business.

A camera pointed at the storage rack can prevent days of finger pointing when an outage occurs.

Detect Problems Early

In our outage story, the UPS was unplugged and nobody knew until the business went dark. That is unacceptable.

Critical systems like UPS units, PDUs (power distribution units), and cooling units must be monitored. If one loses power or fails, alerts should trigger instantly. Email, text, or ticket, it does not matter. Someone should know within seconds, not hours.

UPS units should also be configured for safe shutdown. If power fails for too long, they can protect data by shutting down systems in sequence. A UPS that only provides temporary power without a graceful shutdown is a liability.

Add environmental sensors for temperature, humidity, and airflow. A tripped breaker, a failed AC compressor, or even a propped open door can be detected before it becomes an outage.

Guard Against Fire and Flood

Server rooms are not ordinary office spaces. Sprinklers dumping water on racks of electronics cause as much damage as fire. Data rooms should use specialized fire suppression, often inert gas systems that suffocate flames without harming equipment.

Flooding is another risk. Basements may be convenient, but they are vulnerable to broken pipes and groundwater. If your data room is below grade, install water sensors and drainage protection.

Build for Resilience

Physical security is not only about doors and cameras. It is about resilience. A single point of failure, whether a UPS, a cooling unit, or human error, should not take down the business.

Design for redundancy. Use dual power circuits, multiple network connections, clustered storage, and tested failover. Audit these systems regularly. A UPS with dead batteries is worse than no UPS at all, because it gives people the false belief that power will not be interrupted.

Culture and Accountability

Technology is not enough without the right culture. If IT staff prop server room doors open, access control systems will not help. If executives pressure facilities to “just let someone in,” monitoring will not matter.

Executives set the tone. If leadership treats physical security as a box to check, the rest of the organization will follow. If leadership enforces accountability, the team will respect it.

Why It Matters to Executives

Executives often view cybersecurity as a technical issue. It is also a business continuity issue. Physical failures cost just as much as cyberattacks.

The outage in our story did not involve ransomware or phishing. But the impact was identical. Systems were down. Revenue was lost. Productivity stalled. Customers were frustrated. Trust eroded.

Physical security is part of cybersecurity. Ignore it and the rest of your defenses can collapse.

Conclusion

Failures are like onions. They stack layer by layer until something breaks. Peel them back and you find each one was avoidable.

An unlocked door. A missing camera. An unmonitored UPS. Each problem alone might not cause an outage. Together, they almost guarantee one.

Do not wait until you are in the middle of a crisis to discover how many layers of failure exist in your company. Audit your server room security now. Place server rooms wisely. Control entry. Monitor activity. Configure alerts. Build resilience.

The cost of prevention is measured in thousands. The cost of recovery is measured in millions.

If you would like help assessing your physical and digital security controls, Stoic Cybersecurity is here to help.