Thoughts on Metrics

Alright let's talk about metrics. I am a data nerd and this is a topic near and dear to my heart. I was at Anomali #Detect17 last month and the idea of “showing value” came up in every single talk I attended. It was like “trending” in real life. So, what did we figure out during the discussions? Well it depends…

The reality is that metrics and data in general are only as good as how you’re using them in your environment to answer your specific business needs. Personally, I prescribe to the saying “what gets measured gets managed”, so I believe that metrics are a critical element of an effective program. If you’re only collecting metrics to “send them north” to executive leadership, chances are that your metrics program isn’t really working. Also, you’re wasting your analysts’ time when they collect those metrics and they will slowly lose respect for you the longer they collect pointless numbers. (true story).

So how do we get to “good” metrics…

As an infosec community, we must realize that there are different measurements that we need to capture for different decision makers in the corporate food chain. Quite simply, one size does not fit all. At the lowest level of management (e.g. team lead/ first line supervisor) there is a need to count pure productivity. These leaders need to assess the effectiveness of their team members and their execution of specific processes. This is often captured by tracking the number of widgets (W) completed and the amount of time (LOE) it takes to complete those tasks. For example, a manager may want to know the number of incident tickets reviewed and the number of minutes it took to review each ticket. A Cyber Threat Intelligence (CTI) team lead may account for the number of reports and IOCs reviewed per analyst/shift/day/week/etc. These data points can help the Team Lead identify several decision points, including 1) which analysts deserve bonuses or other perks 2) which analysts may require additional training 3) which analysts/shifts are overtasked.  

Hopefully this is an automated calculation and they're not asking analysts to “put a check mark on a Post-It note every time you…”. Yes, I've actually had a manager ask me to do that before- “just for a couple of months until we could collect a meaningful number”- whatever that means! See my point above about losing respect for management…

From our example above, we can start looking at our metrics from the next level in the management chain. At the section/branch level, these same data points (#Widgets & Level of Effort) can be used to identify 1) the average number of minutes spent per ticket 2) the average number of widgets per shift/week/month, etc. These calculations can help the branch manager argue for additional resources (people and/or tech).  Within those tickets, you may track the initial DETECTION point- notice the emphasis on the point in which you detected the breach instead of the initial threat vector. Tracking the initial detection point can help determine where additional resources can be applied to “move left of bang”.  

Let’s not forget that our leadership must go before executive board members and the Chief Financial Officer (CFO) to justify their budget each year. We need to get better at arming our CISO with the data-points they need to fight the good fight. This includes high-level summaries of the metrics that we have already collected and discussed. For example, the CISO may take the annual cost of IR tickets and the fact that the primary detection point is C2 to present a quantitative argument for better host-based defenses.  

This is also a great opportunity to tell your “good news story”. You need to be able to arm your CISO with the team’s “wins” in clear & concise bullet points. While sales are hitting record quarterly numbers and the events team crushed last month’s party, what did Security do for us lately? Settle down Captain America, we know you saved democracy- but time to show up and show it off. Arm your CISO with bullet points like- “this quarter we reduced the average downtime from incidents by X hours, with an estimated savings in $XXX based on the sales team’s productivity during that time frame”.  

Ultimately the whole metric process is a lot easier if you know your data and processes intimately. The best leaders I have worked with could tell you how their entire team operated and what the average throughput was for their team. They knew this because they understood their team’s capabilities. They weren’t micro-managers but they were effective leaders. Remember: fight the urge to report numbers for the sake of reporting numbers. We must get better at showing our value to the business and delivering that “so what” statement when we talk about the great things that our teams are doing. We must remember that we support operations and we must make direct connections to the impacted business units.  

---Recap of Examples---

1) Capability Analysis- Count Widgets, Level of Effort (LOE) per Widget, over certain length of time, # of analysts assigned to that task

a. Number of Widgets per week (time) divided by Total LOE for those Widgets = Average LOE per Widget 

b. #W/T= average number of Widgets for that timeframe.

c. Analyst counts can help identify if anyone is overworked, what impact will it have to move or promote an analyst, and other personnel decisions.

d. “Spending $xxxx on this appliance will have us $xxxx annually in IR costs and minimizes the need for redundant hardware.”

2) Thread Feed Analysis- track True Positives, False Positives, number of “actionable” (as defined in your environment) vs. “noise”.  

a. Compare TP/FP and determine an acceptable ratio for your organization. When that ratio is exceeded, it may be time to term off that feed.

b. Compare Actionable / Noise ratio. Consider turning off feed when acceptable ratio is violated for a certain period (30 days of noise?) or certain number of violations (3 strikes rule?). 

c. “the paid service threat vendor notified us of an emerging threat that we were able to detect and mitigate against before it impacted our organization. Based on my team’s initial assessment, this relationship with Vendor A may have saved us $30,000 and 4 hours of downtime” (or whatever your LOE calculations predicted based on calculations of previous incidents).

---Recap for analysts---

Metrics are important. They help us justify getting more resources. They can also help justify your pay raise or bonus. Find a way to make your metrics “tell a story” without impacting your day-to-day operations.

---Recap for leaders---

Be a leader that captures meaningful metrics that support business operations. Learn to “tell your story” using a hybrid approach of quantitative and qualitative analysis. Automate your metrics as much as possible. Your job is to remove roadblocks for your analysts, not create them.  

Well that's my take on it. What am I missing? What have you seen that works? What are some good examples of effective SOC metrics? Effective threat team metrics? Other ways to measure success?  

LIVE FREE | STAND PROUD