Information Security Risk Indicators: Bank CISOs Compare Notes

Whether it's for the Board of Directors, the CEO, the Regulators, or just for your team: CISOs always struggle with:

" What are the RIGHT Key Risk Indicators (KRIs) and Key Performance Indicator's (KPIs) that we should be measuring, monitoring, reporting, and using as actionable triggers?" 

This challenge was taken up by a group of Chief Information Security Officers of large and mid-size U.S. Financial Institutions. Most of these CISOs are members of the Mid-Size Bankers Coalition of America (MBCA). The Collaboration and sharing of data and ideas (one core focus of the MBCA) led to the execution of a survey among members to inventory, and analyze over 120 KRIs and KPIs submitted. 

Ground Rules:

Let us agree (or disagree) on a few things before we dive into the survey results. 

First: Nothing in this article represents the opinions, policy position, or activities of my employer, the MBCA or any member institutions. This article is not to be considered professional advice, and is presented without warranty, guarantees of completeness, or infallibility. Use any content, conclusion, inference, or suggestion at your own risk. Consult with an industry certified Information Security Professional before taking any action as it relates to your unique situation. 

Second: The below definitions are being used throughout this article. 

-Risk: A probability or threat of a bad thing happening, that may be mitigated through preemptive action. 

-Risk Indicator: A signal of future increased exposure or probability of the bad thing happening, and possible future loss of value to the Organization. 

-Performance Indicator: A signal of past business unit activities, and value gained by the Organization. 

-Risk Trigger: A condition that, if met, manifests a bad thing happening, typically resulting in a loss of value. (at 90% capacity, latency is experienced)

-Control Trigger: Based on a Risk Trigger; it is a lower threshold for execution of additional mitigating activities to prevent the negative event from occurring. (If at 90% capacity, latency is experienced, at 75% capacity, an additional widget is added to prevent latency)

-Measure: A piece of objective data derived from observing the characteristics of an object or event. (number of high vulnerabilities in a system as a result of a vulnerability scan)

-Metric: Often subjective, a combination of a measurement and another relative point of reference. (number of high vulnerabilities, older than 7 days)

Third: These types of surveys and opinion articles can only be written with the selfless inputs of other professionals, and I could not have put this together without their help. Thank you to those who contributed to this effort. Specifically @Jeff Jancula, @Eric Fisch, @Brent Tjarks (MBCA), @Mark Watson (EY), and @Fellipe Velloso (WBR).

Finally; I did not ignore what the Industry says we could be measuring. The Center for Internet Security's Critical Security Controls (CSC), National Institute of Standards and Technology (NIST), Information Systems Audit and Control Association (ISACA), International Information System Security Certification Consortium (ISC2), and the International Organization for Standardization (ISO), among others, all have various advice on KRIs and KPIs that could be used. 

This article is real world reporting, translating theory to action.

The Survey

The Survey simply asked for the respondents to report the Metric, the Target, and the Tolerance. It also asked if there was a Control Trigger, and what corrective action they would take when the trigger is met. There are varying degrees of maturity when it comes to metrics. Most start with "Policy Exceptions", then simple KPIs in an attempt to show some semblance of Return on Investment (ROI) (or at least the value InfoSec brings to the organization). Every other business unit is (or should be) reporting performance to show their contribution to the organizational objectives, right?

It is a separate, well known challenge, to show Security Investment ROI. While not the objective of this article; in any domain, you cannot definitively bifurcate ROI from Metrics.

Of the 51 respondents, the bulk of the 120+ metrics are from 33 CISOs who diligently populated the survey questionnaire. One could infer; nearly 25% of CISOs do not collect and report metrics, do not have metrics readily available, or do not have time to grab their last quarterly report to submit answers. To that point - if I get a survey asking for the exact data I submit to the board, executive management, and the regulators; it should be at my finger tips at a moments notice.

Not all CISOs "get it"

One respondent said: "Metrics have proven to be somewhat meaningless....if I block a lot of malicious events, does that mean anything about our overall security? What did we miss (and therefore are unaware) that could be really, really bad?"

Another simply said: "This just got too complicated/time-consuming to complete."

My only response is; If you don't think Metrics are important, go look at the stock markets, and also go say that to your CEO.

General Findings

It is no surprise that that some CISOs try to measure a lot, and some only have data on the basics. The Survey revealed that 92% of the metrics were moderate to low in effectiveness, and maturity, 78% had little to no automation, and 65% had too few to report on.

Most organizations are looking to move away from tactical, manual metrics, and focus on automated, strategic metrics.

Critically - Know your Audience - the reporting lines differ on some metrics, but were exactly the same on others. Everyone reported Phishing Fail rate to the Board, but not everyone reported patch coverage/Vulnerability Scan data that high. This speaks to the maturity and the differing importance perceived from one metric to the other (or no one wants to freak out the Board by showing lots of red on a slide). More on that in another post.

The KRIs submitted could be logically grouped.

Bottom Line - Start collecting the data and measure these to get data-centric decision support on where your resources should be focused.

The Top 10 Metrics (by organizational maturity):

• InfoSec Training Completeness
• Policy Exceptions / Deviations
• Phishing Campaign Fail Rate
• Audit Findings Outstanding
• Risk Assessment Ratings
• Patch Coverage by system
• Incidents and Events
• Breaches
• Vulnerabilities by criticality and age
• Account Management

Part 2 of this Article will break these down further, but I hope this has helped stir the KRI/KPI Data collection juices!

-Brian Fricke, CISSP, CISM, CCSP, CSSLP

Brian is a business-centric technology professional, specializing in strategic Enterprise Information Operations, Security Policy, and Risk Management. He is currently the SVP & Chief Information Security Officer of Bank of the Ozarks. Nothing in this article represents the opinions, policy position, or activities of my employer, the MBCA or any member institutions. LinkedIn: LinkedIn.com/in/brianrfricke