The Two Best Ways to Pick a Computer Defense

There are a lot of research and articles on how to pick the right computer security or network defense. It goes without saying that the product candidate you are considering needs to solve a particular critical security problem you are having, work with your devices and platforms, work well, be a good operational fit in your environment, and be cost effective.

Some Vendors Exaggerate

I know you’re going to find this hard to believe, but some vendors exaggerate the capabilities of their products while underselling the weaknesses. Shocking, I know!

I don’t blame vendors for overselling their products, skipping over weaknesses, and dogging their competitors. That’s their job. That’s capitalism. They aren’t paid to help you make the best selection for your environment and to let you know all the strengthens and weaknesses of their solution or to guide you to other solutions. They usually truly think their product or service is the best one for you and will say yes to nearly anything to get you to buy it.

“Our product detects and stops 100% of all malware known and unknown!”, “It has zero false positives and false-negatives”, or “It can’t be hacked!” are common claims I see out in the marketplace.

So, how do you cut the wheat from the chaff, teasing out any potential weaknesses that the vendor may have not willingly discussed? How do you know if what the vendor is claiming is really true? Well, trying it before you buy it is the best way. But committing yourself to an extended test of a product requires time, planning, and resources; all of which translates to money. How can you figure out ahead of time if a computer or network defense product/service will do what it is advertised to do?

Here are two suggestions that I have always used:

Work the Problem

You are buying a product because it solves a particular problem you are having. The vendor is claiming it solves that problem, and they are likely claiming it solves that problem perfectly or nearly perfectly. But the devil is in the details.

Make up a set of realistic attacker/malware scenarios that you are worried about…the very reason(s) why you are considering this very product. Create as many realistic scenarios as you can. If you like, you can use MITRE’s ATT&CK matrix framework (https://attack.mitre.org/) to help develop attacker scenarios beginning from the initial root cause exploit to final payload action. Document some realistic scenarios and steps in those scenarios. Don’t create every possible scenario, stretching the boundaries of believability and capability. No, you want to create common, realistic scenarios (“attack chains” or kill chains”), which have happened before and are likely to happen to your organization. Best yet, if you have experience with particular attack scenarios from the recent past, use those scenarios as your test case scenarios. Or use ones you’ve heard about in your industry. The closer to them being 100% realistic to your organization the better.

Then propose each scenario to the vendor and ask them exactly how the scenario would be detected, prevented, or responded to. Don’t just take the vendor’s casual word that their product can handle a particular scenario. Ask for details. Ask them to show you the screens or event logs and alerts that would be generated. Even better, simulate the scenario while their product is involved and see what really happens.

In doing this, I’ve seen most products fall short of their proposed expectations. Many times, the promised claims and what I’ve been told are drastically different than what happened during the practice simulation, but often it’s just much less than what I had been told to imagine. The details matter.

Example Scenarios

Let me give you three examples. Suppose someone is trying to sell you a comprehensive intrusion detection system, consisting of both host and network IDSes. And the vendor claims it can detect anything. You spend this $100,000 and your computer security worries are over.

Now, you build your scenarios. Perhaps you start with a common advanced persistent threat (APT) scenario. The APT sends a spear phishing email to 1,000 company employees. A few of them open the email, click on the link, and end up executing a brand new trojan compromised only of PowerShell scripts and some self-encrypting, self-compiling executable code. The malware script isn’t detected by anti-malware software. It runs more scripts and captures the login name and password for an Active Directory, forest-wide admin service account used to do remote backups. The script uses the service account to download additional copies of itself to other computers, to steal additional passwords, and eventually sends the captured credentials outside the organization to the malware’s command and control servers over port 433 using encryption.

Or scenario number two, a trusted insider who is an admin for Salesforce is running a report which essentially captures the organization’s top customers and their contact information. She prints the report to PDF, attaches it in an email, but never sends. After a day, the email is saved to her Drafts folder. She then logs into her email remotely from home, opens the draft email, and edit/copy/edit pastes the data from the report into a local text file.

Or let’s throw in another common third scenario, one of the organization’s programmers uploads code to their public, shared GitHub, which contains hard-coded login credentials. An attacker downloads the code, sees the hard-coded credentials, and tries them against the organization’s publicly accessible API. It works and allows the attacker authenticated access to download as much data as they like.

Now, take all three, very common, real-world scenarios and ask your “fantastic” IDS vendor how their product would detect, alert, and prevent them…in their default configuration state. I’ve done this many times in the real world and mostly, I hear crickets.

Many vendors respond so badly to my created scenarios that their IDS detects none of it. Or they come back and say I have to do all this customized configuration or coding to pull it off. Hey, didn’t you say your solution was the answer to all my prayers out-of-the-box? Why do I have to do all this customized coding and configuration to detect what are very common attack scenarios?

I’ve had vendors tell me I was being unfair. The only thing I’ve been unfair about is actually expecting their product to work as promised and advertised.

I had another very popular vendor, loved more by CEOs who are wined and dined by the vendor’s sales people than by the IT security folks, tell me how their product could instantly retrieve any value or information I needed. I needed to only type in a human-like query language statement and bam!...my answer from my tens of thousands of machines is back in a flash. And they are only too happy to show you the same 20 queries they perform all the time to impress people.

So, I then asked them to get me some information I truly needed to maintain security in a current ongoing project. I told them to tell me, “Show me all the types of encryption algorithms and key sizes used by all programs and services on my workstations and servers.” Crickets. Second question, “Show me all my expiring digital certificates, but only the ones that are actively, currently being used by an application or service.” Third question, “Show me all firewall policies which have an <ANY><ANY> rule essentially defeating the whole purpose for the firewall.” Fourth question, “How many co-workers clicked on the rogue URL link in the spear phishing email sent across the organization yesterday?”

Crickets. I was told that all of what I requested could be done, but it required custom coding, and they were not even sure if they could retrieve all the cipher information from the question even with lots of custom coding because of everything that would be involved.

These were not “gotcha” queries. I wasn’t trying to make the vendor fail. I literally just thought of the information I needed to secure my organization, and what questions I needed answered lately, and asked that vendor to tell me how their “fantastic” product could get me those answers. In the end, they could not retrieve a single query that I suggested that was beyond their 20 pre-canned answers. In the end, I wasn’t buying.

I don’t always end up jamming vendors. It isn’t my intent. Sometimes vendors surprise me. This was the case when I was interviewing KnowBe4, my current full-time employer, when I was a customer. In researching the first edition of my opus book, A Data-Driven Computer Defense (https://www.amazon.com/Data-Driven-Computer-Defense-Way-Improve/dp/1092500847/), I realized that the data was showing that social engineering was the biggest threat to most organizations by far. Today, it’s commonsense, but five to 10 years ago, it wasn’t quite as recognized.

But I realized that every organization needed security awareness training (SAT) as a part of their anti-social engineering strategy. I called several SAT vendors, including one run by one of my old co-workers, whose SAT firm was not the industry leader at the time. I also called KnowBe4. I asked questions like, “How do you automate training?”, “Can I send different training to people based on how they respond to different simulated phishing tests (i.e., success or failure)?”,”Can you detect whether or not someone clicked on a link or typed in their login credentials versus just opening an email?”, “Can you test and detect if someone plugs in a USB key?”, and “Do you do testing of SMS phishing?”, and “Do you do testing of voice call social engineering?” It was KnowBe4’s flying color, impressive responses that led me to not only select their product as the review winner, but eventually led me to them as a full-time employee. I couldn’t wait to work for a company delivering a real product and truly significantly decreasing cybersecurity risk.

These questions were a little less scenario-based, but were based on actual social engineering scenarios my co-workers faced (e.g., simulated phishing testing failures, SMS phishes, employees picking up and plugging in randomly found USB drives, and phishing over the phone).

KnowBe4 wasn’t the only vendor to do this…I had also loved BeyondTrust and Bit9 Parity as top-notch solutions. But there are far more vendors that over promised and under delivered than the ones that actually worked as advertised, or even exceeded my expectations, in real-world scenarios.

No matter what type of computer or network security product you are considering, you should think of common, real-life scenarios that you are trying to address with that product. Don’t let the vendor’s sales people control the topics and tempo. If you do that, they will always do their normal dog and pony show that highlights their product’s strengths and avoids the weaknesses. There is no better way to figure out if a vendor’s product will actually address your critical and substantial issues than using it in real life, and if you can’t do that, do the next best thing by posing real-life scenarios and asking the vendor how their product would specifically perform against that scenario.

Ask This Question

My second hint is a short one. Ask the vendor for three customer references who have been using the product for longer than a few months. I’m surprised by how many vendors, who claimed to have “AWESOME” products, were hesitant to provide even a single customer reference. If you come across this, don’t walk, run away from this vendor.

But many vendors will give you one to a few customer references. You know these particular references were handpicked because of their glowing satisfaction with the product and vendor. They oftentimes have very friendly relationships with the vendor and might even feel guilty if they were to say even the tiniest little real criticism about the product you are calling them about. This is to be expected.

So, you need to ask questions that peel back the innate protection these customer references feel to get at the true story. No product is perfect. Every product has strengths and weaknesses. Your job is to find the weaknesses. Empathy works well here.

I call the references with a handful of questions ready to go, including how the product would handle and perform using the various scenario questions I had asked above. I’ve found that when I ask very real-world scenario focused questions that the people I call tend to open up. Like you, they are real-world warriors defending their environments against the world. And when you ask them about common, specific scenarios, they can picture themselves in those scenarios and they tend to tell you the truth about how the vendor’s product performs, including what it misses or doesn’t do well.

The last question I always ask is, “If you could change anything about the product or could get a feature added, what would it be?”

I’ve never had a real-world user of a product, if they are being honest with me, not give me some usable information about a particular weakness. That existing customer has been using the product and is frustrated by something. This question brings out that frustration. I’ve had customer references who were giving nothing but glowing, super-proactive statements, finally give me the real dirt with that last question.

There are a lot of products and solutions vying for your time and money. Most security vendors are over promising and under delivering. I don’t blame them. Most aren’t even aware of it. They are just trying to sell their product and make a living. Your job is to figure out if the product can truly do what it says it can do. All that takes is some scenario creation and a well-timed question.

Good luck!

Continue to fight the good fight!