How Fast Do We Need to Patch?

We need to re-think our whole patching expectation!

According to Mandiant, 33% of all successful exploits involve unpatched software and firmware (https://www.action1.com/patching-insights-from-kevin-mandia-of-mandiant/).?

The recent Comcast breach that exposed 36M customer records (https://techcrunch.com/2023/12/19/comcast-xfinity-hackers-36-million-customers/) is begging the question of how fast do we need to patch after a security critical patch is released by the vendor?

Comcast was exploited by a vulnerability known as Citrixbleed (https://nvd.nist.gov/vuln/detail/CVE-2023-4966). Here’s the known timeline:

1. Citrix releases patches for Citrixbleed on 10/10.
2. Comcast breached using Citrixbleed on 10/16.
3. Active exploitation was publicly announced on 10/17 and CISA adds Citrixbleed to KEVC on 10/18.
4. Comcast notices exploitation 10/25.

I’ve got to say based on the known facts, that I’m not going to be one of those people who shout out that Citrix should have patched faster. I’m not sure when they were going to patch their Citrixbleed vulnerabilities, but it was already a non-issue 6 days later. For better or worse, a ton of organizations would not have been patched 6 days later.

But there have been a growing number and percentage of zero-days over the last few years and attackers have stepped up their game to exploit unknown and publicly known vulnerabilities faster. I’ve been in the computer security game for 35 years and something has changed. We just don’t have as much time to patch anymore.

Note: The past multiple exploited Microsoft Exchange vulnerabilities (in 2021 and 2022) and the more recent MOVEit (https://en.wikipedia.org/wiki/2023_MOVEit_data_breach) 0-day (2023) has become a leading indicator of the changing nature of 0-days. There are more of them being exploited more often far more broadly than before.

So how fast does an organization need to patch in order not to be seen as negligent?

Conventional wisdom’s answer has always been, “Fast as you can after thorough testing.” Fast, fast, fast. Do it now!

But the reality is that organizations, fearing operational failure because our patches often cause problems (often not due to what the vendor did), cause most organizations to intentionally back away from the “now, now, now” mantra. I get it. Causing operational downtime is a career-limiting move.

Traditionally, for decades, this operational worry has turned into an official or unofficial recommended patching requirement for most organizations that was “within 30 days or less”. In my many years of doing security reviews across hundreds of organizations, I’ve seen faster and slower patching requirements. I’ve seen some orgs require patching of critical vulnerabilities within 1 week and I’ve seen some say within 6 months, or even longer. But the average organization that I reviewed had a 30-day requirement. The ones I considered best had a 2-week requirement. It’s just difficult to test and deploy patches faster than that.

This is something CISA seemed to acknowledge a few years ago when they set up their Known Exploited Vulnerabilities Catalog (https://www.cisa.gov/known-exploited-vulnerabilities-catalog) patching list. In their initial release they said patching for new items placed on the list should be within 2 weeks: https://www.cisa.gov/news-events/directives/bod-22-01-reducing-significant-risk-known-exploited-vulnerabilities

But is that quick enough?

I think we are coming to a point in time where it isn’t. I think the best patching schedule for critical vulnerabilities exposed to the Internet is a week or less. This is going to be seen as “impossible” in many organizations, but that’s only true given the current expectations and resources. I think any patch management system not capable of faster patching needs to be re-engineered and better supported by senior management and budget. We live in different times.

We Need New Patching Timing Expectations

I will say that I’m still a huge proponent of everyone patching what is on CISA’s Known Exploited Vulnerability Catalog list as soon as possible. And if you can’t do it right away, soon as possible! I don’t think 2 weeks is safe for most organizations anymore.

I’ll add to that list any announced Internet-exposed security vulnerability with an 8.0 or above Common Vulnerability Scoring System (CVSS), https://nvd.nist.gov/vuln-metrics/cvss. CVSS scores 7.0 and above are considered “High” criticality, but in my anecdotal experience, only the ones that were ranked 8.0 (and really 9.0 and above) become the “big ones” that require super fast patching. I think patching every 7.0 CVSS patch will lead to a ton of unnecessary patching. I could be wrong.

Here's my current thinking on patching schedules.

What if you can’t patch faster? What if your senior management will not allow you to or give you the right resources to patch faster?

?Well, you’re in a hard place that you shouldn’t be. The traditional response would be that you do any offsetting mitigations that you can do to prevent successful exploitation, but those usually take more energy and testing than just deploying the patch. And some vulnerabilities just don’t have any sane offsetting mitigations, unless removing the impacted asset off the network is something you can consider. Most organizations and people can’t do either.

Another answer is to use an “inline patching” application firewall or proxy that attempts to remediate incoming exploits as if the patches were already applied to the (unpatched) protected resources. These are great options, but tend to be a little like antivirus systems needing constant updating and application testing. It might just be easier to apply the patches sooner in a lot of cases.

This is to say, if your critical patching schedule is longer than a few days to a week, you probably need to strongly consider trying to get faster. Attackers and exploits have changed. They are getting faster. We need to be responsive to that trend and be faster ourselves. Existing status quo needs to change.

Personally, I’d love to see the day when all software and firmware is immediately patched within a day. Vendors force it and automatically patch everything right away, and reboot if needed. They can do it in off hours or let the client select which hour in the day to do. Yes, this would initially cause more operational interruption because of the lack of appropriate testing. But customer complaints would force vendors to make better patches with guaranteed seamless reversals. Customers could still apply to small numbers of assets earlier, within hours, before forced to take everything within 24 hours.

I know I have a lot of readers tasked with patch management that are yelling at me right now. I get it. I’m a writer dork telling people in the trenches with real responsibilities what to do. But attackers and attacks are getting faster. We need to do the same.