Are You Caught in the Retrospective Detection Trap?

We spend a lot of money and time on our detection systems, but we cannot always tie an observed event to an ongoing incident. That means there's always enough blame to go around when we have a breach. People up and down the hierarchy want to know why the IDS/IPS/AV system or security analysts missed the evidence of a breach. The retrospective analysis often "clearly" shows that there were observable indicators and we spend time berating the SOC analysts.

There is a famous math problem called P versus NP, which simply stated is "If the solution to a problem can be quickly verified...can it also be solved problem quickly?" (paraphrased Wikipedia definition). We deal with a variant of that in this case: If we can quickly verify that observable evidence of a breach existed after we declare a breach, can we as easily identify observable evidence before we know something bad is occurring?

The answer is often not. Yes, it's easier after an insider performs a malicious action to say "we should have seen the indicators they were headed towards this action because of bad performance reviews, social isolation, financial need, etc." But as every cop will tell you, if we could predict crime solely based on the presence of defined observable criteria, their jobs would be a math problem, not a people problem. The combination of observables doesn't always definitively lead to a negative event. And the retrospective analysis often forgets that and makes the claim that the analyst should have recognized the combination as definitive evidence of an impending incident without taking into account that that combination may well have occurred hundreds of times and not led to a negative event.

In conclusion, when we perform retrospective analysis, we should analyzing not just the observables that led to the breach, but the same observables that did not lead to a breach as they both inform the perspective your analysts have. And be nice to your experienced and sincere analysts when they raise something that turns out to be nothing in retrospect as the last thing you want is for them to not report something for fear of being berated.

[All opinions are my own and do not necessarily represent those of my employer and should not be construed as reflective of the practices/strategies of my employer]