MITRE releases results for 2024 ATT&CK Evaluations
What I’ve Been Thinking About
Since 2018 MITRE has been conducting annual evaluations of security products to measure their detection and prevention effectiveness.? These evaluations are conducted objectively and in conjunction with each vendor to provide objective feedback on their efficacy for the market and vendors alike. With each year, MITRE improves its evaluations and the standards on how they’re conducted, all in an effort to reflect what defenders must deal with on a daily basis.
MITRE recently released its 2024 results of the ATT&CK evaluations. A good summary of the process can be found here and here. Overall, nineteen vendors participated and their detections and protections were included in the results. Tee evaluations included simulations of a few forms of ransomware and emulation of attack patterns conducted by the Democratic People’s Republic of Korea (DPRK). I highly recommend you explore the results and see how each tool stacked up. One important aspect to keep in mind is you can see how each tool, as well as the cohort, performed “out of the box” as well as with some configuration changes. These changes were allowed midway through the evaluations. Configuration changes naturally yield better results, whereas the goal is to provide the best form of detection and prevention with little to no configuration. I won’t delve into the ins and outs of the results. I’d rather focus on why I think this evaluation process is so valuable.
As you know, I’m a huge fan of MITRE ATT&CK as it continues to provide the framework for how companies can speak to their protections against known threats. As it also turns out, these threat actors use common techniques that lead to most breaches. So the more you know how to test for and defend against these techniques, the better security posture you’ll have. Not only can you use the model of these evaluations to help inform potential security tools and vendors to invest in, but you can also use this model for your program.
Here’s my hot take–MITRE is showing you how to do your job. Well, maybe not your entire job, but definitely sets the example for how to evaluate your security tools and their effectiveness in your environment. If you were to take their lead and conduct these types of evaluations on your own security stack, think of the progress you could report on and how well you could speak to stakeholders about your risks to known threat actors. Just some food for thought, but this also plays a role in my discussion when we get to the news section of today’s newsletter. In any event, I recommend you delve into the evaluations and methodology to see what you can glean.
Start with “Who”
My Musing on Leadership
As leaders, we all understand the notion of ensuring you have the right people around you to help achieve the mission. Every team desires to be a high functioning and achieving team. The principle of "right people, right jobs" is a critical factor in the establishment of a high-performing organization.? It is predicated on the understanding that high-performing individuals thrive in environments where they are surrounded by equally dedicated and skilled colleagues. This fosters a culture of mutual growth and excellence, where each member contributes optimally and inspires others to do the same.
However, organizational growth and evolution necessitate adaptation. Individuals who excelled in previous roles may find themselves less suited to new challenges and responsibilities.? In such instances, it is imperative to prioritize the needs of the team and organization, while remaining mindful of the individual's well-being and potential.? This may involve reassignment to alternative roles within the organization where their skills and talents can be better utilized. The key factor to remember when talking about having the right people is not always a notion of replacing them, but identifying the best fit for them in the current context of the business or team.
Jim Collins, in his seminal work "Good to Great," eloquently articulates this concept:? ensure the right individuals are in the right roles before determining the organization's direction. This necessitates the cultivation of a rigorous, not ruthless, culture, where performance is valued and individuals are supported in reaching their full potential. Collins eloquently states to start with “who” and then determine the direction for the team. This is important because we see that great talent brings innovation and great ideas, which can influence or even change strategy.
It is essential to acknowledge that performance challenges may arise from factors beyond individual capability. When an individual's performance declines following a role transition, it is crucial to examine the role itself, the adequacy of professional development and support provided, and the overall organizational environment.
The "right people, right jobs" principle is essential for creating a thriving organization where individual talents are optimized, collective performance is maximized, and continuous improvement is embraced. It requires leadership to demonstrate discernment, empathy, and a commitment to fostering an environment where every individual has the opportunity to excel. It sounds basic, but it’s not easy and as leaders it’s important to spend a large amount of your time focusing on your team, making sure you have the right people on the bus, the right ones off the bus, and then figure out its direction.
FCC Issues Classic Response in Wake of Salt Typhoon Saga
My Thoughts on the Latest Cybersecurity Headlines
As you know, I’ve been following the latest Salt Typhoon saga quite closely. For those of you who are not familiar with the story, Chinese spies have infiltrated the major US telecom providers and have access to all unencrypted communications transmitted through their networks. This includes text messages and phone calls. As a result, the FBI recently urged Americans to move to encrypted communications for voice and text and assume that your phone calls and SMS messages are compromised. Shortly thereafter, the FCC issued a statement requiring “swift steps” to secure their networks. The memo then proceeds to propose a solution to this by proposing carriers comply with a yet-to-be-determined “modern” framework that adapts to the evolving threat landscape. I am finding it hard to contain my disappointment in this kind of response.?
First, the last thing these carriers need is another security compliance and management framework. Second, the “modern” proposals outlined in the memo state that carriers would require annual certification and attestation with said framework, annual security assessments, and implement a cybersecurity risk management plan. What about that outline is modern? Furthermore, I guarantee that these companies are already doing these activities in multiple facets. They have large security teams with really smart people working for them. They have red teams, threat intel teams, and IR teams. In fact, Verizon puts out one of the best breach reports every year. This kind of proposal will only cost them more time and money and distract them from alleviating the problem. Obviously this memo is quite short and we don’t have details on what else would be included in this “modern” framework, but I know that adding another compliance framework to their list of things to do is the last thing the carriers need. What they need is an innovative approach from governing bodies and the industry to combat these threats. We don’t have the full details on how the breaches occurred, but the Salt Typhoon breaches only highlight how difficult security is, not necessarily expose any blatant gaps in carriers’ current security practices.?
I really hope the FCC’s guidance moving forward is truly a modern approach and that we as an industry can support a true paradigm shift. With their current proposal, I’m not hopeful. However, I’m not one to simply complain about a problem and not offer solutions. If we’re going to be proposing new frameworks, I think we should be reporting on newer ways to approach the actual problem, perhaps in a way that MITRE does it? We should already be taking MITRE’s lead on evaluating threat actors in our environment. Let’s focus on a way to continuously report on known threat actors efficacy within our environments. Maybe showing evidence over time as to how you’ve improved in your detections and preventions against known threat actors is a good first step? Drop the notion of annual compliance and attestation, let’s focus on oversight (if we’re going the oversight route) related to continuous reporting and try to remove as much bureaucracy and red tape as possible. That’s where I would start. This is a hard problem and it requires innovative solutions, otherwise we will continue to succumb to the same attacks at the expense of our privacy or worse.?