Misleading Metrics

The misuse of statistics for marketing is a bugbear of mine.

• 'Small businesses are 10 times more likely to be affected by ransomware.'
• '90% of cyber security incidents are down to human error.'
• '60% of businesses affected by ransomware fail within 6 months.'
• 'There are 214,719,292 vulnerable DNS servers.'

I've posted about invented numbers and how to spot them before, and doubtless will again, but this morning I was sent a white paper which annoyed me.

The two key highlights from the report, which appears to be a marketing effort to sell security consultancy (hey, I get it, I like to sell security consultancy too - being able to eat and have a roof over my and my people's heads is one of my favourite things) are about the link between revenue and cyber security. To quote:

• 91% of companies that lagged in Cybersecurity missed revenue goals
• 57% of companies that led in Cybersecurity exceeded revenue goals by 7%+

Alarming! Clearly, doing better at cyber security (UK spelling has a space, I get more nitpicky about this every time someone throws a tantrum about something so inconsequential) makes you better at business.

In fact the report reinforces this by talking about how these successful businesses are not only more mature in their security, but more likely to see security as a business enabler. Who wouldn't draw the conclusion that investing in security maturity is the key to success?

Correlation != Causation

I'll give the AT&T white paper points for never using the word causation, even if the language and framing strongly hints that way. They state that there's a correlation, but never use the word causation. Unfortunately while the data and findings are presented fairly, there's a passage in the exec summary which I take issue with:

The goal of the research was to validate if, and to what degree, organizations more in alignment with best practices prescribed by the NIST CSF can help to operate more secure environments and better enable their businesses. - AT&T Business, The Relationship Between Security Maturity and Business Enablement

There is absolutely no relationship between the stated goal, and the outcome of the research. Yes, they've established correlation, but they have not established that the NIST CSF lead to more secure environments, or that more secure environments better enable their business.

Let me put forward an alternate hypothesis. From much later in the report, where people may or may not read to (most readers of white papers don't get past the exec summary, statistically speaking, it's why they exist):

The research points to a relationship between business success and cybersecurity acumen. This connection is likely anchored by trust, communication, and collaboration between people—managers and staff from lines of business (LOB) and cybersecurity teams. - AT&T Business, The Relationship Between Security Maturity and Business Enablement

Could it, just maybe, be that more mature businesses with better trust, communication, and collaboration between different business areas are inherently more successful in terms of revenue targets (or, for a more cynical take, more realistic about their revenue targets - as these were declared by the businesses) and that being more mature in cyber security by the carefully chosen criteria is an outcome of that?

It's a pity that this research has been promoted this way, because if the link between overall business culture and security maturity were actually explored I can see interesting questions coming out of it about the way cyber security is approached and marketed.

Sadly, a lot of the time this research is mostly about marketing and doesn't delve deeper into the areas where it could make a real difference.