KPIs for Cyber Security

Not All Security Metrics are KPIs

?

The statement, “Ignore/avoid cybersecurity metrics” is a contentious topic in the field of cybersecurity. While many practitioners believe that security metrics are crucial for an effective enterprise cybersecurity program and the pursuit of cyber resilience, I argue otherwise. It’s time to challenge the prevailing wisdom! Let me explain the rationale behind this statement. Far too often, cybersecurity professionals get caught up in measuring security attributes simply because they can be measured. However, it’s important to recognize that key performance indicators (KPIs) are a subset of metrics and offer more substantial value to an enterprise when assessing the health of core processes and the underlying controls. Here’s the crux of the matter: measuring something solely because it is easy to measure doesn’t necessarily contribute to enterprise value.? On the other hand, measuring something meaningful can significantly enhance value creation within an enterprise.

?

Let’s delve into a specific example. During one of my early roles as a CISO, I didn’t fully grasp the significance of security metrics in an enterprise information security program. I relied on a book I read about security metrics and proudly presented reports to the Executive Leadership Team illustrating the number of security vulnerabilities in the enterprise from various sources. As we added more capabilities to detect and remediate security vulnerabilities, the numbers predictably increased. This led to the CEO questioning my tenure, as he pointed out that the security vulnerabilities had surged more than 50% since I assumed the CISO role. This outcome was certainly not what I had intended. I had committed some classic mistakes for CISOs: measuring something that was easily quantifiable instead of focusing on essential information and failing to consider whether the metrics truly reflected the health of the business process and underlying controls.

?

The right approach is straightforward. Instead of chasing after metrics that can be measured for the sake of producing information, strive to define what “health” means for a process and its associated controls. Think practically. For instance, the length of this article, measured in words, doesn’t inherently determine its quality. Increasing word count might improve it, but only if the additional words add value and clarity. In most cases, concise language eliminates ambiguity. A better measure of health, in this case, could be the number of people who respond favorably to a survey question about the article divided by the total number of readers who received the survey.

?

Now, let’s shift our focus to practical considerations in cyber resilience.

?

For any CISO or cyber leader, producing KPIs that gauge the health of key processes and associated controls is crucial. Many KPIs measure processes that aren’t and shouldn’t be under the CISO’s control. When I transitioned from my CISO role at American Express to DTCC, I realized that nearly 70% of all cyber policies were beyond the purview of the information security organization. This compelled me to become the authoritative source of KPIs that directly improved cyber resilience. I disregarded the guidance from numerous books on cybersecurity metrics and concentrated solely on measuring the health of core processes.

?

For example, I noticed variations in server and workstation configurations during deployment, each with its unique standard “build.” To address this, I developed a way to measure configurations for each device type (and standard build) once the configuration work was completed. I shared this information weekly with the IT Infrastructure team and sent monthly reports with the same data to senior IT leaders. This process was extended to cover all device types and was eventually automated. Within a few months, the number of devices deployed with incorrect builds decreased significantly across the board.

?

?

Reporting to Three Levels

?

Initially, I shared KPI information with the deployment team leaders on a weekly basis, but the most substantial improvements in vulnerabilities occurred after the monthly report was distributed. In some cases the senior leaders contacted the deployment team leaders directly about the results but in most cases, the team leaders demonstrated more proactive behavior simply knowing the KPI information was shared with executives. The implication was that this information mattered to the executives, so it was clearly a priority for them. The primary focus of the executives was the potential to lower the cost of IT ownership by preventing defects in the deployment processes whenever feasible. The prevention of defects meant less work to fix them and lower cost.

?

Over time I refined the KPI reporting process by introducing and implementing it for a month or two before sharing the KPI results with others. That gave the team leaders an opportunity to revise/optimize the configuration management process and demonstrate better results when the KPIs were shared at three levels of the enterprise.

?

The most important thing that I learned is the power of transparency in any organization when focus was applied to the things that mattered most…the health of core processes. This is more important when the CISO does not own the core process. KPIs enable accountability to be applied correctly while metrics may or may not.

?

Implementation KPIs vs. “Run the Business KPIs”

?

Any cyber security organization at any point in time is accountable for some level of transformation within the organization, the program and/or the enterprise. This is somewhat connected to the fact that when threat actors adjust their tactics, the enterprise has to reevaluate the effectiveness of the controls in place. It is also the direct result of change in technology and the corresponding attack surface. Therefore, there are generally programs that require implementation with controls that change or evolve over time. KPIs that measure the health of a new process use “implementation” KPIs that differ from “run the business” KPIs once the controls are operational. When tools that automate the build process within a CI/CD pipeline are initially installed, the KPI might be a measure of when the tool(s) is successfully implemented and in use (usually a date). Once the tool(s) are fully implemented, an operational KPI is necessary to measure the health of the software pipeline (e.g.: number of defects remediated/#of builds in production). Complex processes often take months to fully implement changes or new controls, so the implementation KPI is the best way to capture health. In production, the implementation KPIs no longer have value and should be discarded replaced with an operational or “run the business” KPI.

?

KPI Targets

?

Once KPIs are defined and aligned with consensus on how to measure the health of a process, there are techniques the CISO can and should consider for reporting purposes. The use of color indicators (Red for action required, Yellow for challenges, Green for all good) is helpful for senior leaders to know what issues require action or decisions. This allows a CISO to clearly identify targets tied to a specific date for both implementation and operational KPIs.

Successful control implementation, process changes, and technology deployment require efficient allocation of limited resources. ?Technology implementation, in particular, demands specific skills and a focus on continuous improvement. CISOs that understand how technology management works as a business process within an enterprise tend to achieve greater success than those with limited IT operational experience. KPIs play a pivotal role in influencing resource allocation decisions as conditions evolve. They provide a tangible way to capture the true health of processes. To further enhance KPI quality, consider involving individuals with a data science background who can identify opportunities for improving data quality while ensuring data availability and potential automation.

?

?

Conclusion

In conclusion, it is crucial that KPIs are defined by program owners, individuals who bear responsibility for specific programs within the cyber organization, in addition to selected policy owners in other organizations. While the CISO should review all KPIs to ensure they accurately represent process health, the decision regarding the KPIs rests with the program owners.

?

It is essential to avoid implementing KPI definitions sourced from senior executives, as I learned from my previous experience. Relying on senior executives for KPI definitions, even with their willingness to provide input, will lead to inefficiencies and a lack of quality.

?

Executive decision-makers excel at generating ideas and making high level decisions but are out of their element when tasked with architecting KPIs for core processes and controls. Program owners are better positioned to determine KPIs at any given moment. Input from others, especially those with data science skills, is always welcome, it is paramount that the CISO enforces the responsibility of program owners to define and evolve KPIs.

?

I encourage you to explore the more than twenty books describing and emphasizing security metrics. However, I urge you to adopt a pragmatic approach by focusing primarily on measuring the health of processes, controls and workflows that the enterprise genuinely relies on. This approach will lead to more meaningful insights and drive positive changes improving cyber resilience and lowering the total cost o