Rushing Security Fixes, Organizations May Risk Self Imposed Meltdown

ANUARY 4, 2018 BY RBS

On the heels of a nightmare year for security, as most have heard by now, three unique new vulnerabilities were disclosed on January 3, 2018. The vulnerabilities have been given two codenames; Meltdown and Spectre.

TL/DR

The new vulnerabilities are in microchip processors and affect just about every modern day computing device, including desktops, cloud providers, and cell phones. One of the vulnerabilities (Spectre) can in theory be used remotely and potentially allow an attacker to access sensitive information. Spectre while lower risk, is going to be a challenge to fix and will be problematic for the foreseeable future. Meltdown can be fixed by software updates, and despite a low CVSS score, is considered critical by many and poses a real risk specifically to cloud environments.

Should you rush to do something in order to fix it? The answer is not so clear. Many cloud vendors have already implemented patches without you even knowing it. Some software fixes that have been quickly published by vendors impacted are rumored to have negative impacts. The impacts include CPU slowdowns potentially up to 30% and others are more severe with new Microsoft patches supposedly breaking systems and certain antivirus software even causing the dreaded BSOD (Blue-Screen Of Death).

Reports and details have changed substantially over the past 36 hours since the initial media coverage. Intel has suggested they have “made significant progress” in rolling out security patches and firmware updates to protect against the issue.

While more patching will be required for organizations to truly protect their organizations, the truth is that this event is still unfolding and not all of the information is available to make the proper risk management decisions for your organizations at this point.

Vulnerabilities Details

Google indicates that “every Intel processor which implements out-of-order execution is potentially affected, which is effectively every processor since 1995 (except Intel Itanium and Intel Atom before 2013).” They successfully tested Meltdown on Intel processor generations released as early as 2011 and note that it is unclear whether ARM and AMD processors are also affected by Meltdown. This attack is independent of the operating system, and it does not rely on any software vulnerabilities. On a press conference phone call on January 3rd, Intel emphasized that this is not a “procedural flaw or bug“, rather it is a side-channel attack, while some news articles describe this as a “fundamental chip/processor design“. As such, it is unknown if this can be patched by Intel; thus, software vendors are creating patches to workaround the issues.

At RBS, we published an initial entry in our VulnDB service on January 3rd that covered what was rumored to be one vulnerability based on existing information. We were holding off until more details became available because the initial information wasn’t actionable (e.g. there wasn’t certainty if it was a processor or OS vuln, which vendors were impacted, etc.) but ultimately published without as much detail as we would have liked due to the increased media.

Within a couple of hours of publishing that entry, Google broke the January 9 embargo (more on this point later) and published considerable details. Once we had more detail we ended up abstracting to create a total of three entries that have been updated several dozen times since. We continue to monitor the news, social media, and researcher offerings to ensure our coverage is accurate and timely.

The Rush To Patch

While Meltdown and Spectre were fully disclosed on January 3, 2018, many companies had been working behind the scenes to develop patches for their software and services. With the lead time before disclosure, the goal was to proactively patch services in a way that wouldn’t disrupt operations, and for software vendors to have patches ready for customers. One critical aspect of patch development is testing; every patch has to be tested for all supported cloud instances, operating systems, web browsers, and other software impacted. With the disclosure happening six days before planned, some of the patches seem incomplete and problematic. Despite some vendors having months to develop and test the patches, they didn’t work out so well for customers.

In no particular order, a sampling of the problems these patches have caused:

• Symantec Endpoint customers may experience Blue-Screen Of Death (BSOD) after patching. Customers will have to wait a full day for a second patch. [Source]
• The Register reports that Azure cloud customers are similarly experiencing issues after Microsoft’s patch. [Source]
• Microsoft is warning customers that their patch may conflict with anti-virus software and require the anti-virus vendor to set a specific registry key. [Source] For those curious if you are impacted, Kevin Beaumont is maintaining a spreadsheet listing each anti-virus vendor’s disposition and/or response.
• There is a lot of speculation, along with some testing, showing that patches to mitigate the flaws may slow down computers by up to 30%. [Source] Intel disputes this saying that over time the effect will be negligible.
• Many Amazon AWS customers are reporting significant CPU performance drops after the patches. [Source]

While patches are pushed out, end-users can look at workarounds to help mitigate the issue. For example, users of the Chrome browser can follow simple steps to enable an experimental feature that should stop such attacks until the next version of Chrome is released, with additional mitigations built in. Users of Microsoft IE / Edge can install the patches released by Microsoft last night. Users of Firefox are not as fortunate, as they have to wait until the release of the next version which is said to have mitigations in place.

Attribution and Collisions

When word of these vulnerabilities broke in the news, largely on January 2, 2018, we didn’t know it was even more than one vulnerability. Further, we didn’t know who had discovered the vulnerabilities which is interesting. Since there was already word that an embargo was in place, it was pretty clear that security researcher(s) had discovered the issue and were coordinating with vendor(s). This is a much better alternative to a major flaw being discovered while being exploited in the wild by criminals. Once the details were released on January 3rd, the case of discovery became even more interesting.

The vulnerability dubbed ‘Meltdown’ was independently discovered and reported by Jann Horn of Google Project Zero, Werner Haas and Thomas Prescher from Cyberus Technology, and a team of four researchers from Graz University of Technology in Austria. The vulnerabilities dubbed ‘Spectre’ were independently discovered by Jann Horn of Google Project Zero and a group of five researchers from academia and a commercial company. On the surface, three separate researchers, or groups of researchers, finding the same vulnerability may seem incredible. However, as we have written about in the past, vulnerability rediscovery or vulnerability collisions are actually a lot more common than many realize. To us, the interesting part about this collision is that the discovery came from three relatively different areas; a researcher with a large service company that was given leeway to find vulnerabilities, a commercial company that specializes in cryptography, and an academic research group. This shows that a wide variety of security researchers can be interested in the same type of research and ultimately find the same issues.

Last, one aspect we haven’t seen reported, that we find particularly interesting, is that the group of researchers (largely from academia) did this work under several government grants. According to their footnotes, their work was “supported in part by NSF awards #1514261 and #1652259, financial assistance award 70NANB15H328 from the U.S. Department of Commerce, National Institute of Standards and Technology, the 2017-2018 Rothschild Postdoctoral Fellowship, and the Defense Advanced Research Project Agency (DARPA) under Contract #FA8650-16-C-7622.” In short, we can thank the U.S. government for funding the research that led one group to discover these vulnerabilities, while not helping to coordinate the disclosure.

Exploitation

One more thing to ponder on the back of exploitation and vulnerability rediscovery is that we can’t assume the three parties are the only ones to have discovered these vulnerabilities. There may be several other parties that discovered this, and potentially a lot sooner, but did not reveal that fact. Governments and so-called “nation-state actors” as well as criminal organizations that use computer crime as their business model have the expertise to find such vulnerabilities, and no reason to share the details once found.

So if bad actors had discovered this vulnerability and been using it all along, would we know? Despite the boilerplate disclaimer we frequently see, ala “Microsoft has not received any information to indicate that these vulnerabilities have been used to attack customers at this time”, that doesn’t have much weight in many cases. As the Meltdown site says:

Has Meltdown or Spectre been abused in the wild?

We don’t know.

That is fair, they discovered the vulnerability in their own environments, so they weren’t examining real-world deployments for signs someone else figured it out. But, can you even detect these vulnerabilities? From the same website:

Can I detect if someone has exploited Meltdown or Spectre against me?

Probably not. The exploitation does not leave any traces in traditional log files.

This is a lot more problematic for organizations! Even if bad actors were exploiting this for years, we wouldn’t necessarily know or be able to detect if they were. That is an unfortunate but painful reality we all face when trying to determine the risk of a vulnerability.

The Cloud Will Save Us?!

Since “the cloud” became a household term many years back, the debate over the value of outsourcing services and infrastructure to the cloud has been hotly debated. There are clearly merits to using cloud-based services for both businesses as well as some aspects of your personal life (e.g. email). In the case of Meltdown and Spectre? There are advantages and disadvantages! On the upside, the companies that manage the cloud infrastructure are not only responsible for patching the vulnerabilities for you, the big players were part of the coordinated disclosure. That means by the time you learned of the vulnerability, a significant portion of major cloud providers had already mitigated the vulnerability or were close to completing the rollout of the patches.

On the downside? Your entire cloud infrastructure was just as vulnerable as your on-premises computers. Even worse? One of the two named vulnerabilities, Meltdown, poses a much more serious risk to cloud environments where a single host can be shared among multiple companies. Meltdown, despite being “only local”, allows someone with access to a system to potentially disclose arbitrary portions of the system’s memory. So a user with legitimate access to an Amazon AWS instance could in theory use this to disclose memory that contains sensitive information from a different company that shares the same instance. With a well-designed exploit, an individual could rent an AWS instance for a few hours and potentially steal a wide variety of sensitive information, including customer details, passwords, and more.

As always, using cloud resources has tradeoffs, and this is one more reminder of what should be in the ‘risk’ column when deciding.

Disclosure

At some point in the week leading up to this article, rumors of a “big Intel vuln” started circulating. It is difficult to pinpoint where the rumors started, but there were signs of it well before the news articles started coverage. On November 17, 2017, and again on November 30, posts to the linux-arm-kernel mail list discussed large proposed patches to the Linux Kernel referencing them as the ‘KAISER’ patches along with a link to the associated paper. On December 4, a post to the unofficial Linux Kernel Mailing List discussed similar patches, calling them “Kernel Page Table Isolation (was KAISER). The referenced name and paper, “KASLR is Dead: Long Live KASLR” was published June 24, 2017 by Daniel Gruss et al. In hindsight, Gruss et al were one of the three parties to discover the vulnerabilities that would become known as Meltdown and Spectre. Despite all of that information being public, it wouldn’t be until January, 2018 before the full impact would be realized. Jump to December 14, when Amazon AWS customers were notified via email notice of an upcoming wave of reboots on January 5 to fix a security issue.

That and other hints of something big to come were made clear in a January 2, 2018 article by The Register. At the time, the article was comprehensive, well-written, and did a good job of explaining what the community collectively knew. Unfortunately, a day later that article became largely obsolete when Google, Intel, and Gruss et al broke the planned January 9, 2018 embargo on the vulnerability information. By this point, there were a dozen or more news articles and thousands of Tweets speculating about the “big Intel vulnerability”. Intel broke the embargo due to “inaccurate media reports” while Google broke the embargo due to “growing speculation”. With the details released, we realized that there were two major vulnerabilities, Meltdown and Spectre, and that Spectre was itself two separate variants giving us three distinct vulnerabilities.

In the months leading up to the disclosure, we subsequently learned that Google had started patching their infrastructure in early 2017, a “majority” of Microsoft’s Azure cloud offering had been patched, macOS Server had been patched quietly as of version 10.13.2 (released December 21, 2017), and Microsoft’s “Windows Insiders” received preliminary patches for these vulnerabilities in November. The rest of the Windows world had to wait until January 3, 2018 for the patches. Note that according to the Verge, the patches were to include only the “Intel vulns” for Windows 10. In reality, Microsoft decided to release their full “Patch Tuesday” set of patches, including the “Intel vulns” and 33 additional issues, six days ahead of schedule. Meanwhile, more mainstream news outlets like the New York Times were covering the vulnerabilities, albeit, with some errors. By mid-day January 4, 2018, many major vendors had released advisories with varying states of mitigation including VMware, Citrix, Red Hat, Microsoft, and more, with some vendors still evaluating the impact.

For those relying on CVE/NVD for vulnerability intelligence, they got a partial win? Unlike some prior named vulnerabilities, MITRE opened up the three CVE IDs for these issues (CVE-2017-5754, 2017-5753, 2017-5715) in a timely fashion (January 3rd), but didn’t even include the names of the vulnerabilities in each entry. The sparse descriptions also give no real insight into the nature and real severity of the vulnerabilities. Meanwhile, NVD still hasn’t opened them up on NVD let alone generate CVSS scores or CPE data. Unfortunately, MITRE did not offer any guidance to CVE assignments to the CNAs either, prompting one CNA to consult RBS for advice on assigning for their products.

Multi-vendor disclosures, especially to this scale, are tricky to say the least. Coordinating patches and disclosure across dozens of vendors, while leaving hundreds more out, will always be messy. In this case, it isn’t clear when the coordination between the various researchers, Google, AMD, and ARM began, but it seems the embargo lasted for several months. Ultimately, the information was disclosed six days shy of the targeted disclosure date (January 9, 2018). Regardless of the fallout, this is yet another clear reminder of the benefits and pitfalls of large-scale embargos.

FILED UNDER: NEWS, VULNERABILITIES