Manage Your Security Ops Like a Pilot

Having no infosec incident response plan is "controlled flight into terrain". Oooh, headlines!

Security incident response. It's a hot phrase right now in whitepapers, marketing material, podcasts, all those good things. Many of you are getting tired of being blasted by everyone relabeling their products as "automated incident response", to the point where the phrase is losing meaning. It won't be long until the disruptive paradigm shifts are collaboratively next-gen interoperating with-- you get the idea.

I want to focus specifically on what this automation is all about, and why it matters. One of my hobbies is flying small aircraft (private pilot), and there are a lot of similarities. Let’s indulge in some analogies, shall we?

Info security incidents and aviation emergencies have a lot in common.

Incident response is a kind of emergency. Like when your oil temperature is in the red, you look down and suddenly it's adrenaline-time. You have mere minutes to get your head together, to figure out what's going on, to take control of the situation and fly it safely to the ground. If you fail, you fall out of the sky and end up in the headlines in a bad way. How are pilots trained to handle this?

We live and die, literally, by our checklists. Security incidents are resolved or turn into disasters by workflows (or lack thereof). That's what a workflow is: a checklist on steroids. When X happens, do Y, and practice it until it's muscle-memory. The bulk of pilot training is 1) not allowing emergencies to happen to begin with (process) and 2) emergency management (response); these are both checklists/workflows. You're smart enough to see how these map to each other, so let's talk about airplanes falling out of the sky.

A pilot starts their flying day by preparing a flight plan. "I'm going to go from Here to There. Here's when I'm going to go. Here are the waypoints I'm going to hit along the way, and how I know I'm there. Here's the course I'll take on each leg, taking into account winds, and yes I’ve checked the weather forecast. Here's the altitude and airspeed I'll be at. Here's how much fuel I'll burn. Here are all the radio frequencies of local airports and Air Traffic Control along the way. Here's how I plan to approach the end airport and land." Then the flight plan is filed so the air traffic controllers know what to expect and where to look if you suddenly disappear from radar and go silent.

It's a checklist. It's a workflow.

Security workflows are not just for emergencies; they are also for making sure one doesn't happen to begin with. Onboarding, offboarding, recording what you did, security posture reviews, getting systems into place to prevent problems from happening to begin with, then have an outside consultant check your work. It's the same thing.

You then preflight your airplane. You walk around it looking for leaking hydraulic fluids, dents and dings, blocked airspeed/pressure sensing vents, oil on the ground. You check if there's water in your fuel, and that your tanks are full and the gas-caps tight. Flash the landing lights and flight markers. Turn on the radios and make sure you hear what you expect to hear. Wiggle all the control surfaces with the controls (not your hands!) and look to see they do what they are supposed to. Have a peek at the maintenance log to see when the last inspection was done.

This also is a checklist. This also is a workflow. Okay, time to fly!

Workflow. Checklist. Either way you look at it, it's a process for not falling out of the sky.

Review your security posture with a checklist implemented as a workflow. Endpoint protection. Network hardening. Authentication. Communications protection (email, etc.) Storage security. Patching. Automated or not, always be checking. Monitoring and security are best buddies.

Talk to the tower, tell them who you are, what type of plane you're in and which direction you're headed. "CLEAR OF PROP!" and fire it up. Check those gauges right away! Voltage, fuel, oil pressure, altimeter, engine RPM, all that. Okay, taxi to the runup area and let the engine warm up. Stand on the brakes and take it from idle to full power. Magneto check (you have two, but can fly on one; make sure both work). Okay, tell the tower you're ready to go. Take your position when you're told, then... go! Flaps down throttle in mixture good accelerate accelerate plenty of runway ahead approaching vX lift a little off the ground accelerate a little more lift the nose assume best rate-of-climb check those gauges again start easing in the flaps trim for climb okay good take a breath say byebye to the tower frequency and change to regional ATC hi it's me I'm activating the flight plan I filed and am in the air. WHEW

A mental checklist but a checklist nonetheless. A workflow, and it include abort-plans if something goes wrong.

You get the idea. We're cruising along a mile above the ground, on course, blue sky, pretty mountains and forests below and ENGINE SPUTTER SPUTTER STOP CRAP CRAP CRAP.

The most important checklist of all:

Aviate. Navigate. Communicate. In that order.

Keep your head and keep control of the plane; you've practiced this a million times. Immediately point the plane at the emergency landing site that you ALWAYS are picking out visually as you fly, trim for best glide-ratio, and get your head in the game. That field. That lonely road with no power lines next to it. That airport I'm in gliding range of. That lake for a water landing if nothing better is available. And I already know it's there because I've been mentally picking it out. I'm a mile above the ground, when I glide I lose a thousand feet per minute, I have a 10-to-1 glide ratio so I have ten miles and five minutes to deal with this. No panic.

When an incident is happening, step zero is ALWAYS to get your head in the game.

Basic checklist right away. Do I have fuel? Check the gauges and which tank I'm pulling from. Do I have ignition? Check that I didn't bump the ignition key or throttle with a knee or pop a circuit breaker. Are my temperature gauges completely out of where they should be? Maybe it's simple as that. Switch to the other tank, restart, berate yourself for not checking fuel gauges periodically like you should, and fly on. Your checklist saved you.

But if that doesn't help... okay, you're not panicking. You’re not wasting precious minutes on speculation. You're headed to an emergency landing site; you can glide there. Get out your preflight checklist card, flip it over, and if you have time go through it. NOW you call Mayday and let ATC you're in trouble, if you have time. You flip on your 121.5 MHz beacon. And you make the best soft-field landing you've ever done before getting your phone out and calling for help. Nobody died or bent the plane.

Checklist. Workflow. Success.

Security response is like this. Don't panic; keep control; figure out roughly what the problem is then activate the appropriate workflow. You can worry about details later; the priority is to stop the emergency and stop the damage. Forensics are what happens afterward. Response is what you do now. Workstation is showing malware or leaking info? Knock it off the network, you can talk to the user about it after. Firewall letting through an unauthorized SSH session originating from one of the T7 countries? Kill that session and kill that firewall (it's obviously compromised if it's doing something you explicitly have configured against) and pull the flow logs and firewall syslog later. Communicate when you have a moment to breathe; stopping the damage comes first. You've reached for your workflow playbook and based upon the broad category of incident you've done the right thing.

Be the pilot of your infrastructure.

You’re in better shape because you're not improvising. You've thought about these scenarios in advance, you've developed checklists for what to do when they arise, you've had someone else check your work to see that you didn't miss something, and you've practiced. Your workflow is shouldering part of your burden and is your copilot through the turbulence. You're on the runway, not smeared into a cliff.

Be the pilot of your infrastructure. Develop your workflows. Automate them. Practice them. And if the moment ever arises, you're ready.

Of course, if you have a world-class SOAR platform with your workflows implemented to automatically trigger or to be swiftly done at the press of a single button, you're in much better shape. It's like the "Safe Return" button (also known as the "pilot had a heart attack button") in a Cirrus small airplane that will take you to the nearest big airport and land all by itself squawking maydays all the way.. just sayin'... I really wish I could afford my own VisionJet... sigh...

And in full disclosure, I work for a company that makes a kickass, fully-modern SOAR product (Security Orchestration/Automated Response, not like, a glider or something). <mccoy> Dammit Jim, I'm an engineer not a marketeer! </mccoy>

my checklist of hashtags: #infosec #incidentresponse #soar #securityautomation #aviation #pilot #toomanyhashtags #terrainpulluppulluppullup