A CISO said, “I’m fed up with APPSEC errors! I must make better product choices.” This is my How To Measure Anything based response.

What’s Your Error Rate?

Meaning, what’s your false positive and false negative rate?

Easy to answer if you have the data.

Maybe all you have is a sample?

We can work with that.

Or, we can start with nothing.

Let’s start there – with nothing.

First, Ask You SMEs For Their Rate!

Expert guesses are better than nothing.

They’re a starting point.

And, ironically enough, expert guesses can fix biased data. 

They add wiggle to small biased data.

Conversely, SME guesses can add more certainty where there is none. 

You’re looking for comments from SMEs like this: 

“I’m pretty sure the error rate fluctuates around 20% – it’s just as likely to be below 20% as it is above.”

Now, ask your SME’s for their stretch rate.

This is also known as their 90% boundary rate.  

That’s the highest error rate (give or take) they think they would ever see.

You’re looking for a statement like this.

“I would be shocked if the error rate was much more than 45%! ”  

Credible Wiggle Room

Those two numbers: 20% and 45% gives us a model for getting at total wiggle.

That “total wiggle” has a formal name. 

It’s called the credible interval.

It’s slightly different than the confidence interval you may have learned about in school.

The credible interval is a bayesian concept.

This credible interval states that there is a 95% chance that the error rate wiggles between 2% and 59%. 

That’s a lot of room for error!

The Uncomfortable Forecaster

Some people have difficulty making forecasts.  

Yet, entrepreneurs, sales people, scientists and bookies eat forecasts for breakfast. 

If your SME says, “It’s impossible to know the rate!” Take them on their word.

They may not be an SME.  

Experts can forecast. It’s why you hired them.  

They have some vague sense about future impact given current direction. 

Show Me The Money

Time is money.

It takes time to triage appsec errors.

This includes both false positives and false negatives.

It may take multiple hours to triage a single issue.

What’s the average time to triage errors?

What’s the lower bound time to triage?

What’s the upper bound time?  

Maybe it’s two hours on average with a lower bound of one hour and an upper bound of five hours?

Do the same thing with the financial impact.

You are creating ranges of impact with plenty of wiggle.  

Competition Of The Fittest

Now you need to test products.

Run each appsec product against the same target at the same time.

Spend the time to triage the findings.

Get totals findings (that matter) for each vendor.

Count the total errors. You may need to include developers in this (in fact...you must).

Count false negatives too.

If one product finds something another product misses (false negative)….that’s counts as an error for the product that missed it!

Mashing It All Up

All those inputs are put in a box and shaken together….thousands of times.

The graph above simulated 100,000 tests with those wiggly inputs.

It’s what we do when we don’t have a lot of data.

We simulate data generating processes.

We do that by creating what's called a generative model.

We make models because it may be too expensive to collect all the data we need.

Or, we lack time and need to make a decision now.

Perhaps we are just lazy?

Beat The Competing Model

We are shooting for slightly better bets.

That is what the graph above shows us.

And, it’s better than the alternative.

The alternative is unaided, overly biased, intuition (wild unstructured guesses).

Or, it's small, noisy, and biased data.

This model beats both by first and foremost being mathematically unambiguous.

It’s also a consistent yardstick.

And it’s not moody...unlike our unaided intuition!

Yet, it is still very wrong.

All models are wrong in fundamental ways.

And they are very wrong if you expect them to mirror reality closely.

Closing Thoughts

I have written a fuller blog post here with access to the code – to run this yourself.

 https://get.soluble.cloud/posts/2020/06/three-steps-to-better-security-roi-when-fast-times-clash-with-lean-reality/

Let me know if you catch any errors! Be it in code or stats.

Happy Modeling.