The Many Hats of an Incident Responder

The world of Cyber Security has grown exponentially in the last decade. It's gone from more of a niche field to front and center on the world stage. Look no further than the Colonial Pipeline incident and the sheer amount of attention that had not only in the United States, but also throughout the rest of the world. Ransomware is more prevalent than ever and is growing in scale each year with threat actors becoming more aggressive in their approach. DFIR (Digital Forensics and Incident Response) has become a high demand career field within the Cyber Security sector, and yet like many Cyber Security jobs there's a talent deficit to fill these roles.

Today I'll be talking about what it's like to work in the field as well as the many hats that a typical Incident Responder has to wear. This is going to vary between organizations - especially with the size of said organization - but you should have a general idea of what it really takes to do this job. In my experience, recruiters will give you a general sense, but even they sometimes don't have a full grasp on what this role really takes. So, I'm hoping that this article will be informative for both sides of the equation.

It's all in the Foundation

Analysts that work within DFIR greatly benefit from having a core set of skills they've already developed. For example, I worked as a Linux Administrator for a few years before I went to college and eventually graduated with a degree in Digital Forensics. The skills I developed during my time as a Linux Admin - I still use those in just about every case I work despite most of my cases involving compromised Windows systems. Linux provides a ton of power in the command line that's simply irreplaceable in my day-to-day.

However, it's about more than being able to run a few simple awk and grep commands.

What really sets you apart in this field is a solid understanding of network architecture and how operating systems work. Can you look at a firewall and identify poorly constructed rules? Where do you find the password policy for a typical Windows domain? If a server is configured with RAID-5 and was powered off prior to you being called in, how would you properly image that system without booting back into the OS? If someone dumps 500 GB of logs in an atypical format into an S3 bucket, how would you go about parsing and analyzing them? If you find an obfuscated PowerShell script, can you understand what it's actually doing once it's been de-obfuscated?

These are just a few of the questions that might pop up in our day-to-day as Incident Responders - even before we get to the actual forensics which requires a totally different set of knowledge! Having a solid foundation working as a Linux Admin or Windows Admin, a Network Engineer, or even a Datacenter Tech are all extremely helpful going into this field. When you're thrust into a brand new case you're often working with a foreign network that you didn't build and it's a 50/50 chance as to whether you'll actually be provided a network map so it's to your benefit to have a strong understanding of how a network is actually built, how the various pieces function, and especially what role a domain controller plays in the network. After all, most threat actors are trying to get to the domain controller for a reason. It helps to understand why.

Having some form of scripting ability (see the above PowerShell example) is also extremely helpful in your day-to-day. This can be anything from Bash to Python or even Ruby. You're often working with weird log formats that you've never seen before, disk images and all sorts of other oddities and being able to automate some aspects of working with these various sources of evidence is a huge boon in your favor.

And let's not forget the actual analysis side of things. Elasticsearch or Splunk are massively helpful for what we do. Understanding not only how to run searches in the UI, but also actually getting data into these platforms is going to be really key to speeding up your investigations.

The list above is hardly all-inclusive, but it hopefully gives you an idea of the core skills and understanding that you really need to excel in Incident Response.

But is that all? Nope! We're not even close to being done yet. Because next we have...

Curiosity, Capacity and Drive

These three are much tougher to measure. You can't slap them onto a resume and call it good. They have to be demonstrated by on-the-job work to your employer. Let's break each down a bit and go through how I attribute them to Incident Response. Full disclosure: This is my own opinion based on my experiences. There may be some differences based on your own.

Curiosity

This is the difference between scrolling past something you don't understand in a timeline versus stopping and googling it or asking a co-worker. It may slow down your analysis in the beginning, but as you continue to do this you'll build your knowledge up to where you'll be able to understand these things at a glance. This is critical because as threat actors continue to mature in their methods and grow more sophisticated the bread crumbs they leave behind become much tougher to find. It's up to you as the Incident Responder to be able to filter through the noise and find the bad. You are, in many respects, a true detective and curiosity is at the forefront of what we do.

Capacity

One of the toughest parts about Incident Response is determining if someone has the capacity to learn on the job and evolve over time with the work. I mentioned above some of the foundational skills that help tremendously in this field, but that doesn't mean the person will necessarily have the capacity to learn the full gambit of what this job requires. You see, in DFIR you're diving far more deeply in the inner workings of operating systems, networks and various applications in a short span of time than most individuals will ever need to. Incident Response is rapid - meaning you don't have weeks or months to try and wrap your head around how all this stuff is working, figure out where to collect the right evidence from, and then perform a forensic investigation that is both accurate and informative for your client. Forensics requires you to dig deep and understand what kind of information you're looking for and where to find it. And the worst part? Because the bad guys are constantly evolving their tactics, we need to evolve right along with them to keep up - and that is where the capacity to understand the job and to continue to learn really kicks in.

Drive

Lastly, one needs drive to succeed in Incident Response - though one could say that drive is something one needs in any career field. It's especially necessary in Incident Response however because without it you'll be quickly left behind by not only the various threat actors out there - but also your peers. Most of us are happy to share the knowledge we know, however the reality is that none of us are going to wait for you to catch up. You'll eventually run into a situation that may take you hours of trial and error, googling, and beating your head against the proverbial wall to get past. We've all been there at some point. It may be frustrating at the time, but when you come out on the other side you'll be a finely honed blade ready for your next battle.

Many Hats

Okay, so at this point you should hopefully have a pretty good understanding of what it takes to work in DFIR. You might be wondering about the "many hats" part of the article title and so I wanted to bring it all together with a couple of real-world examples of what you might run into in a typical case. My goal here is to arm you with information not scare you off. It's important to set realistic expectations going into this field. When I was in college the degree I took barely covered the tip of the iceberg. The pool of knowledge in DFIR is both wide and deep. It's important both to acknowledge and respect that you have a lot to learn and that, realistically, you'll probably never know everything. This is where curiosity, capacity and drive come in.

Situation 1

In this example we're given 20 triage captures from 20 different Windows server that contain your typical artifacts (Windows Event Logs, Registry Hives, Amcache, SRUM, etc), 50 GB of VPN logs, and there's firewall logs - but there's a catch. You see, the client isn't sure exactly when the threat actor got in, so they gave you a full 30 days of firewall logs which are massive in size. The log format is wonky with double and triple quotes around some fields and annoyingly the date, time and time zone fields are all split apart. How can you get these logs into a sane format that's easy to work with?

Situation 2

You're given 50 servers that have been infected with crypto miners and asked to analyze them and see if you can determine point-of-entry and if there were signs of lateral movement or exfiltration. By the way, the servers are all Linux and there's no local storage to store an image on. The majority of breaches involve computers running Windows, but there are still cases where Linux analysis needs to be done. I've seen many people working in this field that have no understanding of Linux (lacking even basic file system navigation through the command line) and they tend to stumble whenever the OS isn't Windows-based.

Situation 3

This last one is one that actually happened to me last year. Here I am working on mostly Windows-based and Linux-based cases with the occasional Mac case popping up and then suddenly... SCADA. What the heck is SCADA you ask? It stands for Supervisory Control and Data Acquisition. These are the systems used to power our power plants, water treatment plants, etc. Many of them haven't been updated since they were put into production since they're running critical infrastructure. They're vastly different than your typical operating system and will require thinking way outside of the box. You'll often find yourself in situations where you're thrust into the unknown and figuring it out may require some late nights reading through a 14 year old mailing list where a bunch of crusty old SCADA engineers are arguing about the caveats of open mail relay settings - as an example.

The Life

As you can see from the above, the world of an Incident Responder is extremely varied. This is a large part of what makes the field simply fun to work in though. No two cases are the same.

So, what's the actual life of an Incident Responder like? Well, this is the part that recruiters usually get right. It does typically require some late nights and weekend work. You may also need to travel (less so now since the Pandemic) to client sites as well. The reality is that threat actors are less likely to strike in the middle of the business day during the work week and are actually more likely to hit a network outside of business hours somewhere between Friday evening and Monday morning before everyone comes in for the work week. Because of this, we tend to see a lot of weekend cases coming in.

If you're a parent... this can be tough on family life. There's no getting around it.

Most organizations employ rotations so that you don't have to work every weekend, so don't worry - you can still have a life. Just expect that there will be times when you're called in on a weekend to help with the case load so keep your phone close and have a backup plan in case you need to cancel your plans.

One of the most common issues that we have in the DFIR industry is burnout and that's an extremely important topic to address - but I'll do that in another article as it really deserves to have its own spotlight. What I will say here is that there are typically two types of employers in our industry: The first are employers that pay you more, but work you until you collapse in exhaustion and quit and then they just replace you in this constant revolving door of people. Then there are those that pay a bit less, but are extremely mindful of your mental condition and provide a better work environment. Be aware of this when job hunting and really do your research to find the employer that's the best fit for you.

Conclusion

Phew. That was a lot of words, but we made it to the end. I hope this was helpful in some way for you whether you already work in the DFIR industry or you're looking to get into the field. Working as an Incident Responder in the DFIR field has been one of the most challenging and yet the most rewarding jobs I've had over the course of my career. It demands much, and yet you're rewarded with a certain pride in knowing that you're armed with a set of skills very few people in the world have.

And the best part? You're working with people that are just as passionate about the field as you are. It makes such a huge difference when you're surrounded by a team of people who genuinely love what they do. Few people have that in this world and I've been lucky to find myself in a career that provides exactly that.

Thanks for reading and see you next time!