The Nuclear Bomb On Your Doorstep

A Look At The Infosec Supply Chain

There’s nothing like a major cyber security incident to shake things up and remind us of all the dangers that lurk within the interconnectedness and interdependencies of modern infrastructure.  

The week after FireEye announced that it was a victim of a state sponsored breach we were all reminded of the sobering fact that enemies may be lurking within our networks, often for months and even years without detection. This is nothing new of course, but it takes an event like this to wake us up to the real and present dangers we face due to the sheer amount of trust we place in the dependencies we rely on daily.  

The supply chain risk is incredibly high, it’s here, and it’s hairy.

You are not immune to the nuclear fallout, and our systems are not designed to protect us from these deeply rooted connections and dependencies.   All it takes in one weak link in the chain, one vulnerability in your point of entry, and one determined attacker. 

We need to re-think how we operate, and re-architect our defensive postures to find the needle in the haystack.  

How is it that with all the cyber-security firms the US has, that virtually no one saw this coming?

Fun Fact: 425 of the US Fortune 500 use SolarWinds in 2020. 

Isn’t it ironic how the Zero Trust model touted by the very company that the US Treasury and thousands of other government agencies and organizations relied on to manage and protect their networks also was the source the breach?  Just two months prior to the breach discovery, and ongoing while a foreign adversary was in their midst happily pilfering, few things would embody the root of the problem more than this statement from their blog:

“The zero-trust model is critical for building a successful strategy to detect both internal and external cybersecurity threats. This approach requires setting and maintaining strict access controls and "no trust" as the default to all, including those already inside your network.”

Are we following our own advice? The very company which was ‘backdoored’ and in charge with securing and protecting many agencies inside the leader of the free world had:

-      One of their FTP passwords was previously 'leaked on GitHub in plaintext', see: https://www.theregister.com/2020/12/16/solarwinds_github_password/

-      Recommended disabling antivirus scanning in certain directories by their staff, see: https://thwack.solarwinds.com/t5/Announcements-Discussions/SolarWinds-AntiVirus-Exclusions/td-p/303243

These are two small examples that point to a fundamental problem in today’s networks and systems: We trust too much, even when we preach the opposite.  

We trust, because the opposite is too hard and time consuming to get right. 

We permit, because the consequence of not doing so results in too much short-term headache.

IOCs (Indicators of Compromise) are NOT enough.  They are a band-aid at best. A reactive patch that helps us discover if we have been on the receiving end of the next Russian Roulette bullet. Clever attackers will easily get around static IOCs by simply avoiding signatures and behaviors that have been seen or observed before.

While everyone is focusing on the event of the day regarding the FireEye and SolarWinds hack, the supply chain risk remains deeply rooted in the fabric of our lives and our devices.

Are we content to wait until the next press release describing a major security incident affecting a large portion of the Internet population, or do we have a specific risk mitigation plan to avoid being on the receiving end of such a discovery?

The infosec supply chain presents a daunting challenge with numerous avenues for attack. 

Let’s review some of the areas we don’t regularly think about when it comes to the infosec supply chain and our risk-based dependencies below.

Browser Extensions

Browser extensions are the plugins that we install in our browsers to extend their functionality or when the out-of-the-box behavior isn’t enough to support those added features we’d like to have. They can provide value, but they are not without significant risks. Browser extensions can be updated by their owners at any time and are basically mini-software programs living in our browsers or our employee’s which can be revealing everything that is present within the browsing environment including very sensitive and internal corporate details, systems, documents, and more.  They can also be sold at any time – there are entire companies that are built around this – they buy up smaller plugins in order to aggregate knowledge and intelligence about the people that use them.  

Some of this came to light most recently in 2019 when it was discovered that a network of millions of users was leaking their full browsing context to third parties:

https://arstechnica.com/information-technology/2019/07/dataspii-inside-the-debacle-that-dished-private-data-from-apple-tesla-blue-origin-and-4m-people/

Users behind many companies were affected, including: Apple, Symantec, FireEye, Palo Alto Networks, Trend Micro, Tesla, Blue Origin, and many more.

Don’t think for a second that this doesn’t still exist today.  It does. The problem hasn’t gone away, and your browser is still for sale.  

Virtual Private Networks (VPNs)

Third party VPNs (Virtual Private Networks) offer a useful service in that they provide some needed security protections such as encryption and anonymity.  However, not all networks are created equal and it can be very difficult, if not impossible to really determine which networks might actually be spying on your activity 24x7. 

Who really owns the providers that your employees use for VPN services whether they are company sponsored, or third party? 

Who has visibility to the log and track the traffic that transits those networks?

How would you know?

At the consumer level, several well known providers such as IPVanish, PureVPN, HideMyAss, and EarthVPN have all been caught lying about their ‘no logs policy’; although there is certainly a legitimate purpose in helping Law Enforcement, what other providers might be selling your usage data and activity out the back door without your knowledge?

See: https://restoreprivacy.com/ipvanish-provides-logs-to-authorities/

While VPNs provide some measure of security and anonymity, in other respects they may consolidate that ability into different hands which are susceptible to spying or compromise.

Who’s minding the networks your customers, your employees, and your partners use?

VPN providers certainly create a juicy target for nation states or would-be attackers in a way that could give them visibility into a wide range of users.  This information could potentially be purchased or acquired through more nefarious and illicit means. 

Take Citrix for instance… 

Fun Fact: Citrix is an enterprise virtualization and security software provider used by 490 of the US fortune 500 companies.  

In 2019, a report came out showing they had been hacked, and the attackers had been in place for at least 6 months - more likely several years - before they were discovered.

https://techcrunch.com/2019/04/30/citrix-internal-network-breach/

Shortly after Citrix initially disclosed the intrusion in March 2019, Resecurity, a high-end intelligence-driven cybersecurity company claimed it had evidence Iranian hackers were responsible and had been in Citrix’s network for years, stealing terabytes of data. They also presented evidence that it notified Citrix of the breach as early as Dec. 28, 2018, a claim Citrix initially denied but later acknowledged.

This is an all-too-common occurrence. How many times must we collectively fall victim to these attacks before realizing that it’s time to rethink our strategy? 

Mobility, BYOD, and Personal Devices

Your network policy may demand that all software be reviewed and paid for, but what goes on in your employee’s homes or on their personal devices may be another matter entirely.  

What are the full security implications and risks of unmanaged, personal mobile devices and laptops? 

How many of your employees have jail-broken their phones?

How many install mobile applications from outside the Google and Apple stores?

How many have pirated software that has been tagged with spyware and malware?

Although both the Google Play Store and Apple App Stores demand their own security measures and procedures, they are not immune to fraud and abuse.  In particular, Google is regularly scrubbing applications from its store due to new, frequent, third-party reports which uncover additional kinds of spying or intrusive tracking being employed within their app ecosystem. Google is battling an almost constant threat with malware-filled apps attempting to infiltrate its “stringent” security measures. One recent example involved over 100 apps containing malware and adware:

https://www.express.co.uk/life-style/science-technology/1308193/Android-app-warning-google-play-store-100-applications-delete

Where is the ultimate line of defense in such a scenario?

It goes without saying, but we cannot rely on publications, articles, and third-party discoveries to be our last line of defense.

The general lack of control even on “legitimate” software or devices is perhaps just as problematic since it introduces an additional set of other risks to deal with or consider.  The vast majority of employees use personal devices in the workplace with or without employer knowledge. These same devices are frequent victims of malware infection and data leakage.

Software Development Kits (SDKs) and Third-Party Libraries

Third-party SDKs and libraries, while useful in many ways, can also provide a desirable target for an attacker due mainly to their widespread use. These kits are usually designed to be fully configurable, and able to be updated remotely and dynamically which introduces the ability to add a vulnerability at any stage. 

A vulnerable period of impact would be if the SDK or library were sold to an aggregator or third-party company who may have ulterior motives than its typical use. They may want to leverage the data or add functionality that suits a different ultimate purpose. In addition, the SDK update vehicle itself might be compromised through an attack.  

Common types, not usually considered, are advertising mediation platforms, stats collection and analytics systems, attribution, cloud service, and other types of vendor SDKs and libraries that could be compromised, resulting in loss of revenue, reputation, or clients.

The average Android mobile app implements almost 16 different SDKs.

https://en.wikipedia.org/wiki/Software_development_kit

Both SDKs and third-party libraries can be unsafe because they are implemented within apps, but yet run separate code. 

An SDK dependency is a supply chain dependency.

Open Source Dependencies

 There are other dependencies to open source distributions as well. Even though a project or library is open source, it may still be equally vulnerable to an attacker, especially if the libraries provided come in precompiled format where little to no security protections are in place. 

Fun Fact: 95% of enterprises use open source code in their internal projects.

Who manages the open source platforms and how are they maintained and updated?

How would you detect if there was a problem or unwanted behavior associated with that software dependency?

In 2018, an open source project published and maintained by a single individual, Event-stream, a popular package manager for JavaScript developers, was hijacked through NPM.

A malicious user convinced the original developer that he could take over the project and maintain it properly when he didn’t have time for it. The new owner laid low for a time, and then inserted malicious code that was able to hijack certain Bitcoin wallets. That code could have been something completely different.

How big was the reach of this attack? The project code was downloaded almost 1.5 million times each week, and it used in more than 1,600 other packages that were, themselves, downloaded millions of times. The scope of the attack was staggering. 

With open source, trust is implied in several ways:

• Trust the Project
• Trust the Developers (which may change)
• Trust the Update Process
• Trust the Code
• Trust the Reviewers (Such as third parties)

Most enterprises have no idea what open source technologies their developers are using. 

Why are we willing to place so much trust on third parties and what are we trading for that trust?

SaaS companies and Software Providers

It would be next to impossible to function as a business without outsourcing some functions to external services and tools. Most companies will end up relying on many pieces of software platforms that are completely owned and maintained by third parties.  

Some common examples of these include:

• Messaging Systems (EG: Slack) – Who really has access to your internal business communications? Are they fully encrypted end-to-end and in the cloud? 
• Ticketing Systems – These platforms often store passwords, sensitive documents, internal materials, screenshots, and proprietary information. Who runs/owns/manages them? Who could hack in and how would it be detected? 
• NOC Software (EG: Nagios, Zabbix, Solar Winds, etc.). More than enough has been said on this topic elsewhere and doesn’t need rehashing. 
• Digital Conferencing Systems – Platforms like Webex, Zoom, GoToMeeting and others are used by virtually all enterprises.  Either one or several instances of these programs are used to support the daily needs of their staff in the meeting digitalization age. These programs are constantly updated and are not without their own faults such as Zoom, reported in 2019 and 2020. 
• Customer Management Platforms (Hubspot, Salesforce, etc.)
• Document Management Platforms (Docusign, Microsoft Teams, Sharepoint, Basecamp, etc.) - Platforms such as these offer a great mechanism for storing contextual and sensitive corporate information but they also serve as a central point for attack. It’s not uncommon to find contracts, IPs, passwords, tax documents, private emails, conversations, and many other similar things here.

One benefit to a remotely hosted in-the-cloud platform is that no code is running in-house. However, software providers do not have the same luxury. Regardless, each of these systems introduce another leg in the supply chain dependency where information could be leaked or extracted and used to gain a foothold.

In addition, not all of these platforms run strictly in the cloud. They present an additional place for data exfiltration, or worse – they may provide an update and code execution path within your network premise and trusted enclave. 

Routers, Switches, IOT, and Hardware

Hardware devices for the most part are “black boxes” because they have update mechanisms managed by third parties, they are closed source, and proprietary. These systems, should they be attacked, would be incredibly difficult to detect that they were compromised.

One of the easiest ways to compromise a device's security is to do so before it even reaches its intended recipient.

A company’s attack surface is the entire infrastructure they own and manage

AND

the devices used to access their systems

AND

the devices used to store proprietary or sensitive information.

Hardware devices are often overlooked due to the complexities involved in both attacking and defending.

Desktop devices, network infrastructure, IOTs, and everything in between could be vulnerable points of access and entry. 

Ironically, with the latest SolarWinds attack, Cisco, the world’s largest maker of networking equipment was also compromised.  They provide hardware and software for the Internet backbone and are central to corporate and government computer networks across the globe. In a sense, the supply chain of supply chains was also compromised in this incredibly sophisticated and nefarious attack. 

https://www.bloomberg.com/news/articles/2020-12-18/cisco-latest-victim-of-russian-cyber-attack-using-solarwinds

These types of compromises can happen to everything from power plants, power systems, voting machines, cash registers, ATMs, and even your front door lock.

Other important examples include recent discoveries of firmware vulnerabilities within Dell, HP, and Lenovo machines.

https://www.cyberscoop.com/firmware-eclypsium-equation-group/

In February, 2020 it was reported that a stealthy hacking technique that could make it possible for attackers to access different components inside PCs made by the likes of Dell, HP and Lenovo still exists - five years after researchers first warned of it!

Insert Your List Here

Supply chain attacks are not new, but they are increasing.  This article was not meant to be an exhaustive list of the dangers and potential weak links. In fact, it only begins to scratch the surface.  However, it should hopefully help to highlight some of the common potential threat vectors we face as administrators and Internet citizens. 

Not all dependencies are created equal.

Measuring the potential loss or business impact can be challenging but it needs to start somewhere. Clearly some avenues for danger are worse than others. A proactive security posture will review and assess these threat vectors with focus on total risk, total exposure, likelihood, and severity of impact.   

Let's Get On to the Good Stuff...

What Are Some Possible Solutions?

If there was a cure-all for the supply chain attack vector we certainly would have seen it implemented.  Supply chain attack vectors are a daunting problem and seemingly impossible to solve, especially since the possibilities are endless.  However, network and system administrators must begin to rethink how we participate in the Internet ecosystem and how our businesses may be negatively affected by the consequences of supply chain vulnerabilities and attacks.  Furthermore, measuring the potential impact and risk of those consequences should be front of mind. 

Following are some personal views on strategic initiatives that should warrant further review and development:

1.    We protect each other, not just ourselves.

Pen-Test all the companies you have dependencies on.  Put them through their paces; don’t just ask them for compliance to a certain set of written standards. 

Start with firmly understanding what your own trust assumptions really are and then move to those of your vendors and partners.

The bigger the potential impact of compromise, the more time should be spent on analysis of supply chain weaknesses further upstream. Spend the time to find the weaknesses in your partners before an attacker does.   Ask for permission first, but why not “Red Team” your supply chain dependencies, partners, and vendors? Red Teaming isn’t just about finding the holes in your own defense.  The goal is to improve that collective defense going forward.

2.    Incentivize problem discovery.

Most employees, staff, and people in general are not trained to locate, hunt for, or address problems.  Weaknesses, and soft spots are often seen as distractions and are not isolated soon enough. 

However, well crafted incentives tend to refocus our interests and re-align our priorities. Much like a bug bounty program, we should look for ways of making this concept front and center across the entire infosec supply chain and across the entire work force.  

With cost containment measures in full swing and Covid-19 survival mode hitting many companies, your supply chain dependencies may also be cutting costs everywhere they can. Resources that might have normally gone into greater internal security controls, could be increasingly limited. 

The SolarWinds incident might have been prevented with proactive Red Teaming from their customers. This exercise further helps to distinguish between the vendors that listen and are willing to change when problems are discovered and the partners that may not have their eye on the ball. 

Furthermore, if employees were trained to be constantly on the lookout for BAD recommendations by vendors such as SolarWinds to disable their Antivirus Scanning, that too should be flagged up to management for review, and be handsomely rewarded. 

Combine an internal Bug Bounty program for supply chain security deficiencies with the principles of Autonomy, Mastery, and Purpose for a much higher level of success and impact. 

See: https://youtu.be/u6XAPnuFjJc

• Give your employees autonomy and get out of their way to find security problems, solve challenges, and come up with new solutions.
• Challenge them with the ability to get better at the process and hunt. Give them the challenge of mastery to address infosec problem discovery and to come up with innovative solutions. 
• Give employees a purpose to better the process, better the company, and in the words of Steve Jobs, ‘put a ding in the Universe’.

3.    Maintain a baseline of what normal looks like.

 If a new pattern of network behavior emerges, it should be investigated. In order to know what “new” is, we must first understand what the baseline looks like. In the Solarwinds incident, seeing new domain callouts from established software may have been a sign that something wasn’t quite right.  If we observe new domain or IP endpoints from a router vendor, it may warrant investigation.

Instrumenting this is dynamic and non-trivial, especially with anything managed or controlled by a third party. 

Imagine if any one of the thousands of companies affected had a proper baseline and anomaly detection system in place to find this proactively. Any one of those partners working together with the vendor could have spotted this earlier and the entire ecosystem would have benefited.

In each of these cases, having the ability to determine what’s different/new/untrusted would yield a potential short list of items to review and address.

Do your baseline profiling at the app level for every update and every release. Do the same at the device level.  If something has changed from release to release, we should know why and whether it follows the same pattern of behavior and protocol.   

4.    Not if but when… have the necessary tools to find the damage.

It goes without saying that it’s just a matter of time before you will get hit. No agency, organization, or enterprise is immune. I learned this the hard way when helping to protect the FBI website at IBM in the early 2000’s from thousands of attacks per day from all over the globe. Fast forwarding a few years when we started tracking compromised devices worldwide at Team Cymru circa 2005, there was not a single BGP Autonomous System number that wasn’t affected by some type of infection, compromise, spam, or issue.

That was 15 years ago and the problem has only worsened substantially.  No one is immune.

As such, it’s important to have the necessary tools in place to find the needle in the haystack AFTER an event. This means collecting network flows, storing and indexing DNS query logs, tracking system access, and more.

5.    Develop a True Zero Trust Process and Mindset.

 Trust but verify is dead. Zero Trust and verify should be the new norm:

• Zero Trust IPs.
• Zero Trust Domains.
• Zero Trust Files. 
• Zero Trust Users. 

Zero Trust means Zero Trust. Stick to your guns and don’t let third parties get in the way of your policies.  Train your staff to question everything.

What’s trusted today may not be trusted tomorrow.

Are your security postures being enforced, or are people just checking boxes? Do you take shortcuts?  How do you measure and quantify risk?

Do we need a new breed of appliances to take us beyond the traditional trust model we have in place today?

Conclusion

Supply chain weaknesses are a complex topic with many facets. It’s not a problem that will be solved overnight, but we’ve reviewed a few ways in which we can continue to think about the problem and to increase the chances of finding problems further upstream and faster.

Attackers often remain undetected for years.  

See: https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-solarwinds-compromise-to-breach-organizations/

Defenders often rely on the “luck” of their peers to be protected.

Each external dependency comes with a nest of unknown dependencies. 

If you aren’t finding anything to deal with, maybe you’re not looking hard enough.

Twenty years ago a young college graduate with an inquisitive zero-trust mindset, little industry experience, and an appetite for problem solving, uncovered a massive security hole in the entire IBM hosting datacenter that he worked at.  While troubleshooting a connectivity issue behind some high-end network gateways, he reluctantly raised the issue to his team lead for confirmation and support.  As it turns out, an implementation of the vendor security recommendations was deeply flawed.  

That day turned into one of the most defining moments in my early career and I saw it was possible to contribute substantially even as a newbie. 

• The customer change management process was in place. 
• The numerous firewall administrators had reviewed and applied the thousands necessary firewall rules as requested for months.
• The network administrators made sure that the network and datacenter was fully operational as it hosted some of the worlds leading sites and customers at the time.
• The “normal” was when customers weren’t calling to complaining about connectivity issues.

Throughout all of this, our team had lost sight of the big picture: for months the network had been completely unprotected and vulnerable.

We may laugh and think that something like this could never happen to us, but the fundamental principles that led to the discovery and remediation of the issue couldn’t be more true today.  

It doesn’t take a rock star A-team to change our ways. We need a collective culture shift just as much now as we did 20 years ago. 

Curb your stone-throwing; realize how you are just as susceptible as the next guy, and let’s get to work helping each other!