A Call To Action: It’s Time We Changed Our Malware Testing Methodology Standards

It’s time for the information security industry to have a serious discussion about changing our standards for malware threat detection. Otherwise, vendors will continue to develop and market products that put vulnerable enterprises at risk of attack, protected only by misguided confidence in fuzzy math.

Here’s what I mean. If a product is tested against a malware sample and fails to detect that sample on the first try, but succeeds the next 99, is the success rate an impressive 99% -- or a dismal 100% failure? If you manufacture bullet proof vests, you don’t put a product in the market knowing the first bullet will penetrate but promise the customer it’ll stop the next one. Yet, many security vendors adjust their threat “verdicts” when the results of malware sandboxing eventually trickle in. Some even tout their ability to do this as a feature -- “continuous malware analysis” or “file retrospection".

Hackers know their chances of successfully compromising a network with a malware attack are greatest when the sample is new and there is no signature for traditional defenses to identify. That’s why they have adopted automation to generate new malware and bombard their targets at an astounding rate.

Recent research by AV-Test found that more than 121.6 million new malware samples—four new samples every second of every day—were discovered in 2017. At that scale traditional defenses can’t rely on their professed "99 percent" success that equates to a 100 percent failure rate when that first attack consistently slips through. That’s the threat our industry must face and the metrics against which our success must now be measured.

As the old saying goes, you don’t get a second chance at a first impression. Neither do you have the luxury of time to analyze a potential threat when the malware may activate within seconds.

The misleading nature of current industry success metrics is furthered whenever there’s a new, high profile attack that succeeds in penetrating a high-profile target or causing mass disruption. Vendors whose products were not installed may breathe a sigh of relief while the chorus of cybersecurity experts who relish commenting on these attacks blame the victimized organization for making mistakes that led to the breach. Perhaps their biggest mistake was trusting in outdated success metrics. Instead of blaming the customer there should be a level of industry-wide introspection that acknowledges we can do better, and that the time has come to establish new standards for performance that more accurately reflect today’s threat environment.

This is why Richard Seiersen and I have partnered with Blue Hexagon, an innovative deep learning innovator, on a private CISO event entitled “CISO Manifesto: The Future of Security Metrics”. Nayeem Islam, CEO and co-founder of Blue Hexagon, and his team have developed a unique deep-learning based approach to network threat protection that excels in speed, efficacy and scope of attacks. But how would we measure this solution versus sandboxes, signature-based systems and other alternative products?

The CISO Manifesto event is intended as a first step towards a common standard of practitioner-defined metrics. We invite our peers -- all security leaders to join us. The CISO Manifesto event, expert panel and table discussions will take place on Sunday, March 3rd from 6:00 - 9:00pm on the eve of the RSA 2019 Conference in San Francisco at the Four Seasons Private Den. The output of this event will be a metrics manifesto we can all rally behind.

Here are some considerations to get the dialog started:

• What is the value of pushing detect/prevent further away from the defended asset ("left of boom") and does it matter?
• Example: Time to Detect (TTD) should be measured from the point of last packet of first observation to when a payload is first flagged as malicious. Here is where the standard must be set and not on subsequent analyses on the same sample. If TTD is hours or days, that’s an unacceptable risk for the user
• As offensive versus defensive AI continues to push the limits of a real-time arms race, how important is decision speed to security operations?
• Example: Network Time to Determine (NTD) should be how long it takes your security control to make a determination for a file after the very first time it was observed on your network or touching your assigned cloud resources? If NTD is measured in minutes when a payload may activate in seconds, it’s time to do better.
• What is the true cost of missing threats?
• How effective do the controls need to be?  
• Example: False Positive Rate of First Determination (FPFD) and False Negative Rate of First Determination (FNFD): False positives and false negatives are self-explanatory, but must be measured in the context of first determination, otherwise subsequent determinations will skew the metrics and we’re back to fuzzy math.

This debate is necessary if we are to challenge ourselves as an industry, and it must include security leaders tasked with protecting public and private infrastructure. For every second we delay in having this important discussion, the adversary will launch four new attacks.

Follow this link for more information and to register for the event. Security leaders only, please. I'm looking forward to spending some time with you.