Best Questions To Ask When Picking a New Computer Security Product

Finding and picking a new computer security defense solution is hard. Well finding it may not be. We all get besieged by vendors non-stop all the time. They all claim that if you just buy their product, your worries will be gone. Just buy and forget.

There is a lot that goes into figuring out if a particular solution is right for you, including:

● Functionality

● OS Support

● Ease-of-Use

● Cost: Initial purchase price, installation, ongoing operations and maintenance, etc.

● Impact on productivity

And more.

But to me, the biggest question I need to ask before considering purchasing a product is, how well does it actually prevent the cybercrime that it claims to prevent????

The antivirus world is a great example. There is hardly an antivirus product that does not make some 100% or 99% claim of being able to detect whatever malware that comes your way. They have reams of proof, papers and independent studies backing up how great they really are. 100%! You can believe them, I'm sure.

Except I have never found a single AV product that can detect most of the new malware samples I submit...ever! Even today, I can submit a new malware program to Google’s VirusTotal (www.virustotal.com), which currently runs 83 popular antivirus programs, and rarely get a detection hit. All of them usually miss what I am presenting, and I am always presenting real malware that I have verified using my own virtual machines and/or testing. Even after a week or month, the vast majority of them still do not detect the sample. Some of the AV programs seem to detect more legitimate files as malicious than the other way around.?

I am sure anyone in the AV industry just hates me right now, but if AV (or even EDR solutions) detected 100% of maliciousness all the time, the world would beat a path to their door, pay a big premium for the product and our malware problems would be over. But nothing like 100% detection exists. In fact, malware attacks have never been more pervasive and successful than now. That is the real fact.?

So, how do you know if you are being taken for a ride when a vendor says their product detects 100% of something or can do anything they claim 100% of the time?

Well, make the vendor prove it before you buy it.

Start with declaring the goal of the product. What does it claim to do? What does it claim to prevent? What do you think it will do? Clarify with the vendor that your hopes of what it will mitigate match what it can do. Then document the goal. Success of the test then becomes the product meeting that goal. It is important to do this step because reviewers often get sold on overhyped marketing of what the product might do instead of what the product actually can do. So, get agreement from the vendor about what it is capable of and what it cannot do. This is documenting expectations and understandings.

A good example of this is a claim my employer, KnowBe4, makes. KnowBe4 says any customer using our security awareness training products as recommended (which means at least monthly training and simulated phishing tests) will significantly reduce the percentage of their employees who click on a real (or simulated) phishing attack. We claim that the average organization has about 30% of employees who will click on a standard-looking phishing email/link without training. And after recommended training and testing, that figure goes below five percent???.?

Try it yourself. Do a “phishing prone” test. Send out a simple, standard-looking, simulated phishing email and see what percentage of participants respond to it. It is probably going to be 30% or higher. OK. Then start to do the recommended testing and training. Did the phish prone rate fall significantly to below five percent or not? If you cannot test, ask for the contact information for customers who did buy and test. Or even better, just ask a friend who is running the product AS RECOMMENDED, to see what they say. Proof is in the pudding…whatever that means.

For most computer security products, you are going to test and develop, various attack scenarios that the product claims to mitigate ahead of time. Figure out how to realistically simulate the various attacks the product claims to mitigate or detect. Document the inputs you will use to accomplish the simulated attacks and expected product outcomes (e.g., detection, mitigation, block, alerting, reports, etc.). Create a broad set of real-world scenarios. The more real-world the better. It is key that you create your own real-world attack scenarios that your organization is likely to be threatened by. Do not let the vendor get involved in this step, or at least the first set of attempts. That is because vendors will always lead you into scenarios where they know their product excels. They will always knock those scenarios out of the ballpark. So, instead, make sure you push the likely scenarios you want to mitigate. You lead, not the vendor.

Then perform the tests.

Next, analyze the results. What happened? How well did the test go? Figure out strengths and weaknesses. Can you live with the weaknesses? If the product is missing something that is a key expectation and the vendor promises it is coming, get the commitment in writing with a timeline. Sales people will often promise the world, but when you ask for dates in writing, the commitments become real.

Figure out how many people hours are necessary to appropriately install, maintain and operate the solution. Do you have the available hours to run the solution? The worst thing you can do is buy and implement a solution and it becomes shelf ware or simply neglected.?

Finally, the decision. Does the solution solve a critical gap in your computer security defenses? Can you appropriately implement and support it over the long run? If the answers are yes to both, then it is a winner.

You would be surprised by how many solutions fail the realistic testing and expectations of your own environment. Once, a company I was consulting at for an entirely different project asked me to sit in on a call with a vendor of the product they were getting ready to buy. The vendor was promising the world. They could do everything. Stop all attacks, known and unknown, and so forth.?

So, knowing that this customer had recently suffered their biggest breach because the attacker obtained a user’s password from another compromised, unrelated website on the Internet (where the user had inappropriately re-used their corporate password) and then logged in remotely using Microsoft Remote Desktop Protocol (RDP) as the user, I asked how the vendors of the product would detect and mitigate that attack. How would the product know it was an attacker logging in remotely with the user’s correct password and not an attacker with the user’s correct password? And since the user had tons of access to different databases, how would the product detect the user (or attacker) siphoning data from data sources the user had valid access to? All anyone heard was crickets.

In another instance, a very popular vendor claimed that you could ask its software any question and it would immediately return that answer for every device (running the vendor’s agent) in the organization. You want to know if ‘xyz’ process is running on how many computers? “We can answer that in seconds,” said the vendor. You want to know what product is missing what Microsoft patch? “We can answer that in seconds,” said the vendor. You want to know what devices are running an up-to-date antivirus program? “We can answer that in half a second,” said the vendor, with a sly smile.

What the vendor did not know was that I had collected the company’s IT security team ahead of time and asked them to come up with ten questions about client configurations, which if they knew the answer to, would make securing their company significantly easier. It took some prodding and initial examples, but eventually they all came up with ten really good queries.?

After the vendor’s demo that featured every predetermined answer coming back in seconds on their demo environment ended, we asked if we could ask ten different queries. They said, “Yes”, uncomfortably. I do not think they were used to anything other than the “When can we get this product installed?” question after their awesome dog and pony show.

So, we asked our ten queries. And their product delivered none of them. They did not attempt many of them. They said they would have to install additional agents, make additional configurations or write custom queries sets. One of the few queries they did try never came back…it seemed to lock up the system and became unresponsive after that. All I know is, none of the ten best queries the IT security team wanted to know could be answered by that vendor’s solution.?

I think most of the potential customers the vendor presents to are so wowed by the awesome demo accomplished in the predetermined environment, they just see how “great” the product is. Of course, it is! The vendor controlled everything. They pushed what the customer was going to see and experience.

What I recommend instead is figuring out ahead of time what you are expecting the product to do, create real-world test scenarios, and then make your new vendor go prove how well their solution works with your needs and environment?and not the other way around.

Using this methodology, I have been able to pick many very good solutions. Some vendors welcome the challenge and are ready to show how your needs can be answered by their system. There are many good systems out there. They are just surrounded by a lot of junk. You just need to know the right questions to ask.?