Remembering what the end goal is

Longer ago than I really want to admit I was one of a two member team at a major energy company in Australia charged with the strategic design, implementation and coding of a range of systems automation solutions. The IT environment at the time was an IBM MVS/ESA mainframe and a Tandem NonStop system, supported by a 24x7 operations staff of 20+ people. The IBM and Tandem environments supported the company's customer facing systems, bulk produce loading, sales and marketing functions as well as the usual financial and business operations.

Previous to the initiation of the systems automation project the overall availability of the systems was pretty good, as was the handling of any systems interruption by the ops and systems/applications staff. That said, the goal was to further improve the timeliness of detection of problems and automate the actions when problems occurred and during routine system tasks.

The project success metrics were pretty simple (if only as simple to achieve!)

1. reduce system downtime both scheduled and unscheduled
2. reduce operations headcount
3. improve accuracy and timeliness of problem reporting so as to provide a a basis for continual improvement

Sparing you all the details of how we achieved these goals suffice to say that we met all three goals.

Keeping an Eye on the Goal

Now here is the fascinating part of the story and the pivot to the relevancy today...

At the time it was all the rage to measure systems automation projects through the metric of console message suppression. MVS consoles were chatty (according to the message flow rate norms of the day) and if the consoles became flooded with messages requiring operator action the entire system could grind to a halt. Also when humans were expected to read the console flow a fast flowing message rate meant important messages might be missed - messages that might mean that a problem was slowly unfolding.

On the conference circuit my peers would proudly announce in their user case studies that they had achieved 30, 40, 50 even 70% message suppression rates using the same or equivalent automation software as we used.

Cue: Applause! Back-slaps! Bravo!

Then I'd stand up....and announce that our overall message rate had actually increased across the consoles because we'd realised that the console log was basically a message bus that we could use to loosely connect the three major automation platforms we were using (In the absence of API's...Improvise!).

Cue: Silence! Stunned silence! What the...?!?

But you see...our goals did not include making the consoles easier for humans to read. Automation software can read log traffic at a phenomenally faster rate that a human so it didn't matter that we were adding more messages to the stream because we achieved our project objectives quite well (very well in fact). Interestingly the same three objectives were purported to be the project goals for the folk doing the boasting about their success in message suppression ie. reduce downtime, head count and drive long term operational improvement.

The Security So What

Today I visit Security Professionals and some of them seem to be falling into the same trap as all those Message Suppressors of Yesteryear. They have implemented a tonne of automation capability and processes in their SOCs with the purported goals of reducing stopping business critical information being compromised, exfiltrated or interrupted. They seek to reduce security incidences, achieving faster detection and recovery and wish to drive longer term and continual improvement.

Many have achieved measurable and laudable successes against a determined and well resourced set of adversaries who continue to improve their attack methods and skill sets, all while the systems and applications architecture being protected is ever changing.

Cue: Applause! Back-slaps! Bravo!

Some though seem to have taken their eye off the ball a little. Sure they have SIEMs that are automating log collection, collation and event correlation. Sure they have Defense-in-Depth through an implemented and deep stack of security acronyms. Sure they have skilled and dedicated staff who really ought to have medals pinned on.

But....

More than a few have been distracted from achieving their ultimate goal because they have started to measure success through the metric of how many security messages their SIEMs are handling and how many security alerts get logged. As a colleague pointed out to me this week though an alert is simply an avoidable error by the attacker.

The big problem is that all too often the attack isn't triggering any alerts, hence we see industry stats (and after-the-fact news stories) about attacks that have gone unnoticed for weeks, months and sometimes years.

Getting Back to the Real Goal

Those achieving true success in their goals of reducing stopping business critical information being compromised, exfiltrated or interrupted aren't waiting for the attackers to make an avoidable error and thus trigger an alert in the SOC. Instead they're actively hunting - humans ingenuity aided and augmented by investigative technology against human ingenuity augmented by attack and penetration technology.

Hunt, find, chase, learn and shut down.

Remember your security goals and focus on achieving them. Our adversaries are determined but we can win and we will succeed.

Happy Hunting.

Please note that all views expressed in this post are solely those of the author and do not represent those of my current or any former employer or association.