Cybersecurity Frameworks, Maturity Models, and Effectiveness
Several years ago I started applying #cyber maturity models and frameworks to assessments of global enterprise #cybersecurity programs. We worked hard to build our own model and even worked through three generations of that model. The larger team performed well over 100 assessments across multiple industries both globally and in the US. I worked with CISOs and their teams, reviewed their documentation, looked deeply at their programs, technologies, and processes. In some cases we even went deep on a teams' expertise and certifications. We produced great reports with valuable insights for the CISO, their executives and the boards of client companies.
Mark Twain popularized the saying, "There's three kinds of lies: lies, damn lies, and statistics." In this case, the value of statistics (and some interpolation) let me see that the financial state of a company, along with its industry and geographic peers were powerful predictors of a company's #cybermaturity. My prior experience with top 50 global banking and the intelligence community gave me a perspective on where a really mature cybersecurity program would need to go and what highly mature cyber programs could look like. For example, I was fortunate to build a strong identity and access management program before 2010, and still watch companies struggling to build good IAM programs today. We can readily predict that where the best companies are leading, more companies will follow.
Looking back, its clear we missed something vital in these assessments. While the assessment process was straightforward and I stand behind the advice we provided, we were missing #cybereffectiveness. When I look across the global alphabet soup of cybersecurity frameworks and models they are all missing measures of broad cybersecurity effectiveness. We used maturity and widespread practices as a proxy for effectiveness. The widely practiced effort to buy the so-called best tools is another proxy for the effectiveness of tools.
Outside cyber, in finance, Sarbanes Oxley (or SOX) led to a widespread adoption of specific financial controls which are regularly checked by auditors in public (and private!) companies. For SOX, the financial teams establish controls and test the efficacy of those controls on an annual basis. There's both an underlying (if sometimes unspoken) maturity model for SOX controls and a check of effectiveness internally and externally.
For cybersecurity, our proxy's for effectiveness tend to address the last major breach while trying to anticipate the next one. Assessing cybersecurity maturity against a framework has a strong correlation to reduced cyber incidents. Maturity remains a proxy for effectiveness.
领英推荐
Real effectiveness is something different. Effectiveness is hard to measure. In a world where there's a nearly infinite opportunity for new cyber attacks tomorrow, testing the tools, processes, and team to measure effectiveness is the next challenge we need to explore.
Security practitioners have struggled with tool effectiveness for years. We have asked questions like, "what's the best firewall?" What's the best anti-virus? What's the best CSPM? And with no objective testing, experts share their feelings and experience in different environments. At this large bank, product X worked really well. At this manufacturer product Y was the best in our comparison test. We use another layer of proxies like analyst ratings to find the best product to buy. Coming from technology backgrounds, the #cyberindustry often focuses on tools and products (and services); we often skip thinking about the effectiveness of processes, or teams. Rick Howard has been vocal in his #cybermoneyball thinking about building effective teams.
For this weekend, consider this thought exercise: are your current tools, processes, and people effective simply because you haven't detected an attack? Is your measure of effectiveness the lack of a detected, successful attack? With average dwell times measured in weeks, and breakouts from initial infection to detrimental effects measured in minutes, how should you measure effectiveness? How do you assure the CEO, the board, and other stakeholders that you've built an effective program? Are you measuring effectiveness with lies, damn lies, or statistics? Or are you measuring effectiveness at all?
#cybersecurity BoardCyber #CEOCyber #effectivecyber
Chief Information Security Officer at Centene
7 个月I think your comments around Effectiveness are spot on. CMMI (arguably the most popular construct for assessments right now) even says it in the name - Capability. So even if the assessment is deep and intrusive (not some survey, really into the inspection genre), it is providing a view of what an organization is Capable of doing. That is perfectly fine, as long as we use the results appropriately. Understand the difference between capability maturity and explicit effectiveness measures. Use the other tools to dive into the "how well do your teams and tools actually perform" question. Whether that is a Pen Test, Red Teams, Purple Teams, Table Top Exercises, and so on, do not represent the CMMI assessment as the end all, be all moment. I like to think of the maturity assessment as a guide for continued focus and a way to help bolster the argument for continued investment. But I still need to continually test my teams and tools, work their training, improve our policies and procedures and adapt to the changing threat.
At the Corner of Cyber Risk and Business Success.
7 个月Todd Inskeep, CISSP When it comes to cyber security there is a fundamental problem with trusting a self-assessment - even the more formal audit approach is flawed. Both types are typically A) point in time based (annually on average) and B) they are based on personal opinion or a small statistic sampling of evidence. "Show me evidence you're patching software - provide me a satisfactory 2-5% view of your total inventory and I, the auditor, will report that your patching control(s) are cyber secure for the coming year." That approach just doesn't cut it in the dynamic world of cyber risk. In my view the cyber crime industry (yes, consider them an organized industry!) are demonstrating an innovation velocity that outruns everyone. I know there are better ways as we have come up with quantitative data science to lift companies from opinion based / old time audit sampling to operational data analysis on 100% of technology inventory that can call out if stated / self-assessed governance controls are truly implemented effectively or is just a paper standard.
Cyber Risk Management Leadership | Speaker | Author | Board Advisory | v-CISO ???#BringThemHomeNow ???? ???? ????? ?????
7 个月Great perspective! We need to catch up - would love to chat live about this topic.
Managing Partner at Jobplex, Inc. a DHR International Company. I help CIO's and other line of business leaders build diverse, high-performing teams.
7 个月Well said, Todd! Thx for posting