Issue #21 | May, 2024
More challenges for the cloud
In yet another packed edition of Alice in Supply Chains, cloud issues are once again at the forefront.
Our first headline this month is the Cyber Safety Review Board report on the breach that Microsoft (and, by extension, the US government) suffered last year when a Chinese threat actor gained access to cloud-hosted email accounts. Our second story is the Sisense breach, which has been oddly under-reported, in part due to how little information the company has decided to release publicly.
We also have our usual round-up of third-party breaches and government news, as well as some follow-ups on the ransomware attack against UnitedHealth Group and the XZ backdoor story (or, more generally, social engineering attacks against open-source project maintainers).
Enjoy!
Cyber Safety Review Board releases report on Microsoft cloud email incident
The United States Cyber Safety Review Board (CSRB) carried out an independent review of what the government now calls the “Summer 2023 Microsoft Exchange Online Intrusion.” The report is now publicly available, with news outlets highlighting that it blames Microsoft for a “cascade of errors.” The Associated Press says:?
In a scathing indictment of Microsoft corporate security and transparency, a Biden administration-appointed review board issued a report Tuesday saying “a cascade of errors” by the tech giant let state-backed Chinese cyber operators break into email accounts of senior U.S. officials, including Commerce Secretary Gina Raimondo. […]
It concluded that “Microsoft’s security culture was inadequate and requires an overhaul,” given the company’s ubiquity and critical role in the global technology ecosystem. Microsoft products “underpin essential services that support national security, the foundations of our economy, and public health and safety.”
Several other outlets are covering the story, such as The Washington Post. You can read the press release from the Department of Homeland Security or just get the 34-page report PDF from the CISA website.
Since keeping up with all the recent Microsoft security news may be a bit difficult, here’s a refresher: this is the incident from June 2023 in which government email accounts were accessed with a token authorized by a leaked Microsoft Services Account (MSA) key. It was detected by the State Department, not Microsoft, but was later found to impact other organizations.
The government was only able to detect the intrusion because it had acquired premium logging features. Many have pointed out that paywalling such important security tools makes Microsoft appear rather unconcerned about their customers’ cloud security. CISA quickly started negotiating with Microsoft to make this feature freely available to government agencies that were already paying for cloud services. The CSRB review was announced a while later, in August.
Aside from pointing out how Microsoft was to blame for the incident, the report reveals that Microsoft developed a total of 46 hypotheses to investigate how this hack took place, and that the investigation is still ongoing. The CSRB also consulted with other cloud providers to come up with a list of recommendations.
If you want to read what others have to say about this report, there are posts from Adam Shostack, Rich Mogull, Adrian Sanabria, and a thread from Eric Geller.
On the very same day that the CSRB report was released, CISA published an Emergency Directive (ED) to mitigate “the significant risk from nation-state compromise of Microsoft corporate email system.” As natural as it would be to think they are related, that’s not the case — this Directive is about Midnight Blizzard, which is said to be a Russian state-sponsored actor. The incident from June covered in the report is attributed to the Chinese.
A few days later, researchers revealed that Microsoft employees exposed internal passwords.
This flood of issues is slowly giving form to repercussions in the market and national security discussions. Acceleration Economy points out these incidents are creating customer?doubt and opportunities for competitors. Bloomberg weighed in with a story on how this problem was years in the making, forcing the company to “embark on [its] biggest security reboot in two decades.”
Meanwhile, The Register spoke with former senior White House cyber policy director AJ Grotto, who commented that Microsoft is a national security threat due to the comfortable position they enjoy, which gives them little to no incentive to improve security. Eric Geller wrote a similar story for Wired, titled “The US Government has a Microsoft Problem” (restricted).
As we reported in the past, CISA has been pushing for vendor accountability to balance out market incentives. CISA Director Jen Easterly reiterated this view at the recent Institute for Security and Technology’s annual ransomware event: “There’s a lot about the villains. There’s a lot about victims. We do not talk enough about vendors,’ she said, according to Cybersecurity Dive.
While technology vendors have successfully maintained the position that they should not be held accountable for security failures like these, they have also given enough ammunition for the government to support a policy change.
Regardless of how this unfolds, there are likely going to be several implications for third-party risk management – or at least for the security of cloud services and applications hosted on Azure and Microsoft 365.
The Sisense breach and why we should move to better authentication
Business intelligence vendor Sisense suffered a breach of some sort and is keeping some details reserved for customers. Journalist Brian Krebs initially posted on LinkedIn about “a supply chain attack affecting many millions of credentials and hundreds of tenants.” Risky Business picked up the story too, noting that the Sisense breach had CISA “panicking.”
Krebs himself then published a more in-depth story:
The U.S. Cybersecurity and Infrastructure Security Agency (CISA) said today it is investigating a breach at business intelligence company Sisense, whose products are designed to allow companies to view the status of multiple third-party online services in a single dashboard. CISA urged all Sisense customers to reset any credentials and secrets that may have been shared with the company […]
The latest advice from the company is far more detailed, and involves resetting a potentially large number of access tokens across multiple technologies, including Microsoft Active Directory credentials, GIT credentials, web access tokens, and any single sign-on (SSO) secrets or tokens.
Krebs’s article has been updated to include an email from Sisense’s CISO that contains a list of 21 credentials or steps that must be taken in response to this breach. If you happen to use Sisense, make sure you check that out – though, by this point, you really should have received these recommendations through official channels.
The consequences have also reached third parties that rely on Sisense’s services, such as BigPanda. The company has a page dedicated to the incident.
Although the details surrounding this incident have not been confirmed, the remediations alone are enough to understand that an attacker managed to obtain shared secrets used by Sisense to access customer environments. The responsibility to prevent intruders from leveraging these access tokens now falls to their customers.
According to Krebs’s reporting, the attackers somehow got inside the company’s self-hosted code repository, found an access token for the company’s S3 buckets, and used this S3 access to copy terabytes of data from their systems. How the attackers managed to obtain this access is not yet known. Krebs notes that a PR representative from Sisense told him the company did not wish to comment on the story.
No matter how this actually happened, several steps could have prevented this type of incident. Rich Mogull has a post with some recommendations to protect secrets.
Nevertheless, it’s even better if you can integrate with someone else’s environment without a shared secret. This can be accomplished through Cross Account Access with a unique ExternalId. Houston Hopkins shared this recommendation on LinkedIn, but Tenchi’s Alexandre Sieira gave an in-depth talk on this very subject back in 2020, which is a great resource to understand this topic and why you need a unique ExternalId to prevent Confused Deputy attacks (at the 13-minute mark).
With Cross Account Access and proper privileges, SaaS providers avoid pushing the responsibility for rotating shared secrets onto their customers. In a breach like this, the SaaS provider would only have to worry about the reliability and security of their systems. As long they’re secure, customers don’t need to take any particular action, as the cloud provider itself is handling the authentication.
While SaaS providers must design their systems to implement cross-account access, everyone should at least be aware that a better solution exists. With enough demand, vendors that don’t improve their practices will likely begin to fall out of favor.
More commentary from Bankinfosecurity: “Sisense breach highlights rise in major supply chain attacks.”
JetBrains TeamCity vulnerability, silent patching, and vulnerability transparency in third-party risk management
This is a story from last month, but we wanted to have more space to cover it and to make sure the dust had fully settled before we did.
Rapid7 found a vulnerability in JetBrains TeamCity and reported it to the developers. In their communication with Rapid7, JetBrains said it wanted to release the patch without publicly announcing the vulnerability – a practice that Rapid7 deemed to be incompatible with its policies against silent patching. JetBrains broke off contact without letting Rapid7 know it intended to proceed in this way.
As soon as the patched version was published, Rapid7 confirmed the fix was working and disclosed the details of the breach along with their timeline. JetBrains later shared on their blog that they did keep Rapid7 out of the loop:
February 23 - 6:32 pm – The Rapid7 team confirmed our approach is against their policy, and they want to make a “coordinated disclosure”, which means publishing full technical details of the vulnerabilities once the fixed version is released. […]
At this point, we made a decision not to make a coordinated disclosure with Rapid7 as we strongly believe that publishing all technical details at the same time as releasing a fix allows anyone to immediately exploit the issue before all customers have had a chance to patch their servers.
After the patch went live, the timeline was not great for TeamCity users.
The patched TeamCity 2023.11.4 was released on March 4, and Rapid7 published all technical details on the same day despite not being warned about the patch. Shadowserver alerted on the next day that exploitation activity had already started on the day before – that is, March 4. By March 7, the vulnerability was added to CISA’s “Known Exploited Vulnerabilities Catalog.”
On March 19, Trend Micro published a full technical description of some of the ongoing attacks that install the Jasmin ransomware, as well as the SparkRAT backdoor and cryptocurrency miners. So, if you didn’t patch quickly, systems may have been overrun with any of these threats.
Vulnerability disclosure has always been controversial. History knows we did not get to bug bounties and coordinated vulnerability disclosures without some bumps along the road.
The fundamental issue is vendor accountability. If software developers are not liable for the damage that a vulnerability will cause, then the incentive to patch is, unsurprisingly, very low. The incentives are better today, as information gets around a lot faster.
“Silent patching” is the practice of fixing a vulnerability and not disclosing it – forever, if possible. While this is the norm for SaaS or web-based platforms when there’s no evidence of exploitation and no customer action is required, it used to be a rather common practice for on-premises software, too. Things are different now, with the CVE inclusion rules even allowing certain cloud vulnerabilities after an update in 2020, and we also have websites like Cloudvulndb.org to consolidate vulnerability data.
When silent patching is done to protect a vendor’s reputation, it is commonly understood to be very questionable behavior. Many researchers have understandably developed a hard stance against any sort of silent patching, as they believe this indicates an attempt to conceal a bug.
Silent patching is questionable even when the developer finds the issue. Not only because customers simply have a right to know, but also because being hit by a “silently patched” vulnerability could be very problematic for those who are operating under new regulations with incident disclosure requirements.
Still, making all technical details available on day 1 usually shortens the time until exploitation starts – even if only by a bit. This, of course, is undesirable.
While there is a case for negotiating a less technical disclosure, with some details being withheld until customers have a little time to patch, it’s important to understand that the benefits are limited. Security flaws can be quickly located by comparing the patched and vulnerable versions of the software. In other words, time is always short.
Whatever the case may be, coordinated disclosure is not optional. Trying to opt out is unlikely to work because whoever found the vulnerability will almost surely be able to find out if a fix was made available, too. To avoid this situation, your best bet is to improve your secure development processes to catch more vulnerabilities before others do. If someone does find a vulnerability, however, you should respect their requests.
Rapid7’s post on silent patching makes most of the arguments above, but there may be some space for a renewed discussion regarding what details could be held to help defenders more than attackers on day 1. Software vendors should remember the history of how we got vulnerability disclosure to become what it is today though, and that many researchers will be naturally suspicious of these suggestions – for good reason.
While transparency remains a key principle in vendor relations, those who can make a strong case to protect users do deserve to be heard. This is unfortunately rare, as many software vendors fail to understand that patching is already a form of disclosure.
It’s also unfortunate how often we see vulnerability disclosure issues today. SecurityWeek reported recently that Delinea, a privileged access management (PAM) solutions provider, was “scrambling” to patch a critical flaw after they ignored the researcher who found it, and Ars Technica also reported that Microsoft failed to properly inform customers that a vulnerability was already being exploited.
Third-party risk management requires us to work together, but situations like this keep reminding us of how difficult that is even for a decades-old topic like vulnerability disclosure.
领英推荐
Cisco Duo, Home Depot, MITRE: third-party breaches of the month
Our breaches round-up starts with MFA provider Duo, a Cisco subsidiary since 2018:
Cisco said one of the providers it uses to send multifactor authentication (MFA) messages was breached by a threat actor on April 1.?
In emails to customers, Cisco said the incident specifically affected Duo — a multifactor authentication company it acquired in 2018. The attacker breached the system of a telephony supplier that Duo uses to send MFA messages through texts and phone calls to its customers.
You can check the email sent by Cisco Duo and more coverage from Cybersecurity Dive. The attacker stole phone numbers and message metadata, opening a path for MFA fatigue or other forms of social engineering attacks. Note that this is a vendor breach for Cisco and a “fourth-party” incident for Duo customers.
Dropbox Sign eSignature suffered a breach of its production systems that resulted in the theft of API keys, OAuth tokens, and MFA data. Much like the Sisense breach, customers will have to reset passwords and rotate API keys.
Home Depot confirmed a data leak at a vendor that “inadvertently made public a small sample of Home Depot associates’ names, work email addresses, and user IDs.” Another leak hit World-Check, a screening database containing 5.3 million records for know-your-customer (KYC) checks. As per TechCrunch’s report on the breach, the hackers took the data from a third-party firm in Singapore that had access to the database.
The MITRE Corporation, a well-known name in the cybersecurity community for their work on the development of tools and standards, is now one of the organizations hit by the Ivanti zero-days that we mentioned last month. We may hear of more VPN breaches soon since Volexity documented a zero-day exploitation of a new vulnerability in Palo Alto Networks GlobalProtect. Palo Alto assigned it CVE-2024-3400, referring to the campaign as “MidnightEclipse.”
Federal contractor Acuity disclosed an unauthorized access to their GitHub repositories. The company says no sensitive data was stolen, but the State Department stated it was investigating the incident. The Department of Justice is investigating another government contractor, Greylock McKinnon Associates, after a breach stole data on 340,000 individuals.
Another consulting firm for the health sector also reported a breach affecting more than 1 million individuals. Berry, Dunn, McNeil & Parker is said to have traced the unauthorized access to a breach at their MSP, Reliable Networks, but the MSP is denying involvement.
Survey rewards platform SurveyLama suffered a data breach impacting 4.4 million users. Chipmaker Nexperia has been added to the list of manufacturers hit by cyberattacks, and it is believed – though not confirmed – that hackers stole chip designs and other trade secrets, possibly belonging to customers.
Security firm CertiK detected suspicious activity on Lightning-based exchange FixedFloat, with attackers moving over $3 million. FixedFloat blamed the issue on a vulnerability at a third-party vendor.
Ending this section we have a follow-up on Infosys McCamish Systems, which is reporting a $38 million loss following a security breach from last year. This is the breach that Bank of America and Fidelity have reported as third-party incidents in the past few months. The company currently estimates a total of 6.5 million individuals were impacted – few of which have ever heard of the firm, we could add.
New federal bills in the United States aim to end vendor lock-in and unify privacy regulations
Two new pieces of federal legislation that have been put forward could have significant consequences for vendors. One bill, by Ron Wyden, intends to “end federal dependence on insecure, proprietary software.” The “Secure and Interoperable Government Collaboration Technology Act” is a direct response to the recent Microsoft breaches and the CSRB report we discussed previously. As it’s focused on interoperability and standards to avoid vendor lock-in, it would:
- Require the National Institute of Standards and Technology (NIST) to identify a set of interoperable standards, requirements, and guidance for each of these collaboration technology features, based on a set of required collaboration technology features identified by the General Services Administration (GSA).
- Require that, to the fullest extent possible, the standards use end-to-end encryption and other technologies to protect U.S. government communications from foreign surveillance.
?- Require that collaboration technologies used by federal agencies enable those agencies to comply with federal record-keeping requirements.
With or without this bill, new regulations for federal acquisition in the United States are already taking shape, as government agencies established a framework, called part 40, that is expected to deal with cyber supply chain requirements.
Another bill has been proposed to establish a national data privacy standard. Data privacy in the US currently relies on a myriad of different regulations that were independently created by individual states. As digital services tend to be available to citizens all over the country, many would prefer a single rule for all states. The bill is receiving criticism, both due to its provisions and one politician who put it forward.
CISA has joined the Minimum Viable Secure Product Working Group and joined with others to announce and launch Protobom, a solution to open source software bill of materials (SBOMs). Meanwhile, the Department of Homeland Security established an AI Safety Board.
The US added four Chinese companies to an export blacklist for allegedly helping Chinese military programs. And, on the topic of China, the Wall Street Journal has more coverage related to the “Delete America” initiative, reporting that Chinese telecom carriers will be expected to phase out chips from Intel and AMD.
In Singapore, essential service providers could be expected to meet higher cyber-security standards due to a new law.
The last story in this section is from Russia, where the courts sentenced a former FSB officer to 9 years in prison. Grigory Tsaregorodtsev was found guilty of receiving a $1.7 million bribe from a cybercriminal gang to act as “roof” – an agent who interferes with investigations and protect the group. The verdict supports the common understanding that Russian cybercriminal groups stay afloat in no small part thanks to corrupt law enforcement agents.
UnitedHealth, XZ backdoor, and LockBit ransomware updates
This last section will go over some updates regarding the UnitedHealth/Change Healthcare hack and the XZ backdoor incident we covered last month.
The Change Healthcare incident is still escalating, as the UnitedHealth Group CEO testified at a Senate Finance Committee hearing on May 1:
UnitedHealth Group CEO Andrew Witty took fire from both sides of the aisle Wednesday during his testimony before the Senate Finance Committee on the cyberattack on Change Healthcare, a subsidiary of his company.
Senate Finance Chair Ron Wyden (D-Ore.) made it clear straight out of the gate that he blamed Witty’s leadership for the cyberattack, which caused widespread disruptions to the healthcare sector.
In the written testimony provided ahead of the hearing, Witty had already revealed the attackers managed to get inside using stolen credentials to a Citrix portal that didn’t require MFA. Attackers remained inside for nine days before deploying the ransomware.
During the hearing, he also estimated that maybe a third of US citizens were affected. The full hearing is available online.
UnitedHealth Group has published several updates on the incident. The media, meanwhile, picked up on the issue we also reported last month regarding the leaked data and the internal disagreements within the BlackCat ransomware gang and added some new information. According to Wired, a new ransomware group named RansomHub is threatening the company.
We also now know the cost of the cyberattack is expected to surpass US$ 1 billion.
As for the XZ backdoor, Russ Cox prepared a great timeline. It mentions some of the actions taken by Linux distributions which we didn’t cover previously.
Understanding the XZ story might be relevant moving forward, as similar attacks against other projects are now emerging – either because people realize what someone tried to do in the past or because new copycat attacks are being attempted against other projects. For example, someone attempted a similar trick against F-Droid in 2020 (F-Droid is an alternative app store for Android smartphones), and the Open Source Security Foundation published a blog post revealing yet another attempt and alerting maintainers of this issue.?
We have one more link with an update to a story that we covered previously. According to Trend Micro, LockBit ransomware activity has significantly decreased since law enforcement agencies disrupted a part of their infrastructure. We covered this operation in March.
This edition was already packed, but we still have three bonus links for you below. See you next month!
In this special edition of the Risky Business podcast Patrick Gray chats with former Facebook CSO Alex Stamos and founding CISA director Chris Krebs about sovereignty and technology.
China and Russia are doing their level best to yeet American tech from their supply chains – hardware, software and cloud services. They’ll be rebuilding these supply chains – for government systems, at least – from components that they have complete visibility into, and control over.
Costs associated with cyber attacks against the financial services sector are on the rise, the International Monetary Fund (IMF) has warned, with the number of incidents having more than doubled since the beginning of the pandemic. [...] The report also warned about the effects of financial firms’ increasing reliance on third-party IT service providers - a trend that’s accelerating with the emergence of AI. “Such external providers can improve operational resilience but also expose the financial industry to systemwide shocks.
Additional coverage of the IMF report is available from The Banker.
Google says it has patched a nasty loophole in the Android TV account security system, which would grant attackers with physical access to your device access to your entire Google account just by sideloading some apps. As 404 Media reports, the issue was originally brought to Google’s attention by US Sen. Ron Wyden (D-Ore.) as part of a “review of the privacy practices of streaming TV technology providers.”
Android TVs are linked to Google Accounts just like smartphones but are not expected to have a lock screen, creating a simple loophole in the authentication process.