Is Cyber Risk Quantification (CRQ) Baloney?

Cybersecurity practitioners and risk managers are increasingly adopting Cyber Risk Quantification (CRQ) to assign financial values to risks, aiding decision-making. However, with the growing reliance on complex formulas, assumptions, and tools, CRQ is not without controversy.

Applying Carl Sagan’s “Baloney Detection Kit” — a set of cognitive tools for distinguishing sound reasoning from fallacy — we assess if CRQ is indeed valuable, or if it risks becoming another example of pseudo-precision in cybersecurity.

Let’s dive into Sagan’s framework and apply his principles step-by-step to CRQ.

Carl Sagan’s Baloney Detection Kit: A Brief Overview

Sagan’s Baloney Detection Kit consists of principles for evaluating arguments or claims objectively. Some of the most relevant tools from his toolkit include:

• Independent confirmation of facts
• Encouraging multiple working hypotheses
• Avoiding reliance on authorities alone
• Rigorous testing of assumptions
• Ensuring falsifiability
• Following Occam’s Razor — simpler explanations are preferable
• Testing with reproducible results
• Being cautious of logical fallacies like appeals to ignorance or false analogies

With these tools in hand, let’s analyze CRQ.

Applying Sagan’s Baloney Detection Principles to CRQ

1. Independent Confirmation of Facts

CRQ frameworks like FAIR (Factor Analysis of Information Risk) depend on internal and external data to quantify risk. However:

• Challenge: Much of the data is subjective or based on industry surveys, which might lack independent confirmation.
• Assessment: CRQ runs the risk of being based on insufficiently validated or biased data sources.

Conclusion: CRQ lacks consistency in independently confirmed data, which may compromise its reliability.

2. Encouraging Multiple Working Hypotheses

Cyber risks are complex, involving various threat vectors, assets, and consequences. CRQ frameworks typically reduce this complexity to a monetary figure representing expected loss.

• Challenge: CRQ might oversimplify a multi-dimensional problem by focusing on only one quantifiable outcome (financial loss).
• Alternative Hypothesis: Qualitative assessments, heat maps, and threat models might better capture non-financial risks.

Conclusion: CRQ may suffer from reductionism by dismissing alternate ways to evaluate risks.

3. Avoiding Reliance on Authority Alone

CRQ tools and models are often presented as objective science. However, organizations may adopt them primarily because industry experts or vendors endorse them.

• Challenge: Tools like FAIR are promoted by institutions, but they often rely on proprietary assumptions that aren’t fully transparent.
• Critical Question: Are decisions being made because of evidence or simply due to expert recommendations?

Conclusion: Heavy reliance on authority could reduce the critical evaluation of CRQ’s claims.

4. Testing Assumptions Rigorously

Many CRQ models depend on assumptions like:

• The probability of a breach can be estimated.
• Losses follow predictable patterns.
• Controls affect the probability linearly.
• Challenge: Many of these assumptions are difficult to test in real-world conditions, especially because cyber events are rare and unpredictable.
• Risk: If assumptions are invalid or misapplied, the resulting risk quantification becomes meaningless.

Conclusion: CRQ struggles with untestable assumptions and often lacks scientific rigor.

5. Falsifiability

Falsifiability means that a claim must be testable and capable of being disproved. Can CRQ models be proven wrong?

• Issue: Cyber incidents are highly unpredictable, and CRQ models often give probabilistic estimates (e.g., “95% chance of $2 million loss”). These estimates are almost impossible to falsify.
• Risk: Models may be retrofitted to justify outcomes after incidents occur, leading to confirmation bias.

Conclusion: CRQ faces limited falsifiability, making it hard to assess its true predictive power.

6. Occam’s Razor: Are CRQ Models Overly Complex?

CRQ frameworks often use complex statistical models, making them inaccessible to many decision-makers.

• Challenge: In some cases, simpler, qualitative frameworks could provide equally actionable insights without adding the burden of misunderstood metrics.
• Risk: Overcomplicating risk with financial quantification may lead to misinterpretation or inaction.

Conclusion: CRQ may violate Occam’s Razor by creating unnecessary complexity that does not improve decision-making.

7. Testing with Reproducible Results

For a model to be scientific, results should be reproducible under similar conditions. However:

• Challenge: CRQ outcomes often vary depending on the organization’s data quality, assumptions, or the tool used.
• Risk: Inconsistent results make CRQ look more like art than science.

Conclusion: CRQ struggles to achieve reproducibility, raising doubts about its scientific reliability.

8. Cautiousness About Logical Fallacies

Several logical fallacies can creep into CRQ:

• Appeal to ignorance: “We can’t manage what we can’t measure.”
• False precision: Assigning exact dollar values to inherently uncertain events.
• Confirmation bias: Adjusting risk models after incidents to match the outcomes.

Conclusion: CRQ is prone to logical fallacies that may distort the real picture of cyber risk.

So, Is Cyber Risk Quantification Baloney?

Based on the application of Carl Sagan’s Baloney Detection Kit, CRQ exhibits some signs of pseudo-precision and over-reliance on assumptions. However, it is not entirely “baloney.” Here’s the balanced takeaway:

1. Valuable Aspect: CRQ offers a way to link cybersecurity to business outcomes, enabling better communication with executives.
2. Flaws: CRQ is not a panacea. It can oversimplify complex risks, rely on questionable assumptions, and mislead decision-makers with false precision.
3. Recommendation: CRQ should complement other frameworks and qualitative methods, rather than being treated as the only truth. Organizations must apply CRQ cautiously, testing its assumptions and recognizing its limitations.

Final Verdict

Cyber Risk Quantification is not pure baloney, but it should not be taken as gospel either. Applying Sagan’s principles, CRQ appears useful as one tool among many. However, decision-makers must stay vigilant about its assumptions, limitations, and possible fallacies — keeping a critical mindset to avoid being misled by the illusion of precision.

CRQ works best when balanced with other frameworks and continuously tested and validated. In short, CRQ is only as good as the critical thinking we apply when using it.