Executive Missteps: Fire Drill Down Below

This is the fourth installment in an ongoing series called "Executive Missteps" on how business leaders, often unknowingly, sabotage their organization's cybersecurity...and how not to be that person.  Each is inspired by a true story (or several). Find previous installments here: Above the Law (1), Shadow (IT) Man (2), On Deadly Ground (3).

Avery (not his real name) is busy, like most CEOs.  His company is a regional powerhouse in the process of building out a handful of new state-of-the-art facilities, and a leveraged buy-out of a smaller competitor.  Things like this don't happen fast: it's been a year-long project so far.  When Avery's assistant says that the company's cybersecurity team would like a date within the coming quarter when the C-suite is available to participate in a cybersecurity tabletop exercise for two hours, he tells her that sounds like a matter for the technical staff, and that executive leadership won't be participating.

Steam release from a piece of manufacturing equipment

Time marches on.  Almost a year later, the company had a serious cybersecurity incident.  The security team is on it.  Working with general IT staff,  they disconnect two manufacturing facilities from the internet and from the corporate network 

 and begin a deeper investigation.  They find that several HMIs -- human-machine interfaces, or general-purpose computing units that act as an interface with ICS/SCADA systems throughout the plant -- have been infected with malware and are sending programming changes to controllers throughout the facility.

One of the facility managers takes initiative and stops work at his plant after learning from the CISO what is going on.  The other facility's leadership is more of a stickler for protocol: he points out that IT has no authority over the manufacturing process, only the datacenter and office computers.  He will only accept a stop-work order from the COO or CEO.

When this happens, our CTO is on a plane to an international conference and can't be reached.  The company CISO, Mark, can't get a call with the COO or CEO: they barely know who he is.  They buried cybersecurity under the CTO so they wouldn't have to think about it.  Mark soon realizes he needs someone with more pull, and a real stake in cybersecurity.  He reaches out to the company attorney he works with on compliance matters, and gets a meeting with General Counsel to explain the likelihood of permanent damage to equipment, loss of intellectual property, and threat to human life if any of the compromised equipment malfunctions while in use.

General counsel gets it, and makes some calls, but before the COO can have the second plant shut down, a cooling system serving several manufacturing lines fails, and some machinery continues to operate instead of detecting the overheat condition and shutting down.  At least two million dollars in equipment is destroyed, and two employees are rushed to the hospital, having been burned by escaping steam.  There were several more minor injuries from debris ejected during the steam escape and as machines ground to a halt under high heat and pressure.

Avery has a board meeting tomorrow, and he fully expects it will be taken over by this latest incident.  He's several steps behind.  Mark asks to convene a tiger team including the COO, CMO, the CTO's second in command, General Counsel, himself, and the managers of the two impacted facilities.  Avery agrees, but progress is slow.  No one but the CISO and the CTO's stand-in has experience with cybersecurity incidents, or any process for working through these problems.

The company eventually gets through the incident, but one of the burned workers dies, and the other faces a lifetime of disability.  It takes weeks to get both plants online.  The company's financial losses mount.

Creating a Culture of Security

A tabletop security exercise sounds like a very technical thing, but it's not.  Think of it as a dress rehearsal for a true emergency.  It's in that rehearsal process that people not normally involved in cybersecurity get a handle on their roles and responsibilities in a true cybersecurity incident.  It's how executives avoid being blindsided, and how organizations come to understand the gaps in their response plans before an emergency happens, rather than in the middle of one.

An exercise once per year for a couple of hours, followed by reading a short report, is worth the time of every executive.  It's a small investment with big returns.  If you are disciplined enough to make it to the top job, you should understand that practice matters.

Many of the organizations I see with deep cybersecurity problems have leadership that choose to distance themselves from cybersecurity generally.  Part of having a culture of cybersecurity has to come from the person at the top.

Wrapping up

If you're a CISO, push hard to build relationships at the top of your organization, especially the CEO (and, if possible, the board).  Ultimately, these people set the tone, and they carry the authority to get the organization to do something unusual (like shut down a manufacturing plant) if something goes sideways and can't be handled in the usual way.

If you're a CTO, help your CISO get prepared to step up and act like a real executive.  Many people, as you've probably seen, find the transition from an engineering field to a new executive role difficult to navigate.  Instead of being a barrier between the CISO and the C-suite, you can enable those relationships through coaching and helping your CISO get the exposure they need.

If you're a CEO or COO, make it your mission to become literate enough in your organization's cybersecurity risk and posture.  Build a relationship with the CISO.  Get regular reports directly from the CISO, not through a third party.  Participate in a tabletop cybersecurity exercise at least annually.  It's another thing on your plate, but with a good CISO you'll get a better handle on your organization's security strategy and investments, and be on stable footing in the event of any emergency, for a very small amount of attention.