#4. Measuring the Seemingly Unmeasurable

Dear friends,

I hope this letter finds you well. In my last letter, we explored the concept that anything we care about can be measured. Today, I'd like to dive deeper into specific measurement methods I've been learning about from "How to Measure Anything in Cybersecurity Risk" by Hubbard and Seiersen.?

When considering measurements, many of us might think of direct, straightforward processes - like using a tape measure to determine the length of a piece of lumber. However, in cybersecurity, the process is often more complex. We frequently deal with what the authors term "indirect inferences" - deducing unseen information from observable data. This approach is akin to detective work, piecing together evidence to uncover hidden truths.

For instance, imagine trying to measure the effectiveness of your firewall. You can't directly observe every attempted intrusion, but you can track things like the number of blocked connection attempts or the frequency of rule violations. These observable metrics help you infer the overall effectiveness of your firewall.

Here's an intriguing insight about sample sizes: even small samples can be more informative than we often assume. Consider the Rule of Five, a useful concept in statistical inference. It states that in any random sample of five items from a population, there's a 93.75% probability that the median of the entire population falls between the smallest and largest values in that sample. This principle can be quite powerful in practical applications.

To illustrate, let's apply this to a cybersecurity scenario. Suppose you're estimating the typical duration of security incidents in your organization. If you examine five past incidents and find they lasted between 2 and 8 hours, you can reasonably infer that the median duration for all incidents likely falls within this range. While this doesn't provide a complete picture, it offers a solid starting point when working with limited data.

To further illustrate, let's apply this to another cybersecurity scenario. Suppose you're assessing the impact of a new security awareness training program. If you randomly select five employees and find that their phishing test success rates range from 75% to 95%, you can be quite confident that the median success rate for all employees falls within this range.

Addressing rare events, such as major data breaches, presents a different challenge. For these situations, the book introduces Laplace's Rule of Succession, a method for estimating the probability of an event, even if it hasn't occurred in our observations.?

Laplace's Rule states that if an event has occurred m times out of n observations, the probability it will occur in the next observation is (1 + m) / (2 + n). For example, if your organization hasn't experienced a major data breach in six years, Laplace's Rule would suggest the probability of a breach in the coming year is (1 + 0) / (2 + 6), or 12.5%.

To put this into perspective, think of Laplace's Rule as a conservative estimate that errs on the side of caution. It's like assuming there's always a small chance of rain, even if you've had clear skies for weeks.?

This result might seem high, and it's appropriate to question it. This is where the concept of "reference classes" becomes relevant. We can refine our estimates by considering a larger set of similar situations. Rather than relying solely on your company's history, you might examine breach frequencies across your entire industry. This broader perspective, or "baseline," often yields more accurate estimates than relying exclusively on individual experiences.

Practical Application:

Let's put these concepts into practice. Consider the following scenario:

Your organization has implemented a new intrusion detection system (IDS). In the first month of operation, it has detected 3 genuine threats out of 100 alerts. How would you estimate the probability of the next alert being a genuine threat?

Using Laplace's Rule of Succession: (1 + 3) / (2 + 100) = 4/102 ≈ 3.92%?

This gives us a starting point, but remember to consider your reference class. How does this compare to industry standards for false positive rates in IDS?

?

Key Takeaways:

1. Small samples can provide valuable insights (Rule of Five).
2. Even rare events can be estimated (Laplace's Rule of Succession).
3. Always consider broader context (Reference Classes).
4. These methods are starting points - refine as you gather more data.

?

I encourage you to consider applying one of these methods the next time you encounter a seemingly "unmeasurable" aspect of cybersecurity. You may find that even a small amount of data can yield useful insights.

I'm interested in your thoughts on this topic. Have you employed similar methods in your work?

Keep learning and stay curious.

Michael

P.S. While these methods are useful, they serve as starting points. Always consider the broader context and be prepared to refine your estimates as you gather additional information.