"Trusted supplier" and cyber security? Stop barking up the wrong tree!

For the best part of the last two years I have been involved in discussions with European stakeholders on the topic of 5G supply chain security. Part of these discussions involved my peers from the cybersecurity industry, many of them involved politicians. Ideas have been floated, some of them made it into whitepapers, non-biding guidance (like the EU 5G Toolbox) and into country legal systems. After all these months one question continues to echo in the back of my head – aren’t we barking up the wrong tree? The tree called “trusted supplier”.

Control the risk where you can

Supply chain security is the bread and butter of any security executive. It just goes without saying, that the cybersecurity risk that your organization is running, depends on what and who has access to it. It should be also obvious to any security professional with experience worth their salt that in the ICT industry this chain of suppliers has the complexity of a fractal – your suppliers and service providers have their suppliers and service providers, too. For all practical means and purposes, the supply chain of telecommunications industry is infinitely complex and ever expanding. And still, surprisingly many of those who like to be searched by the handle of CISO or a cybersecurity expert, believe that they have what it takes to check, verify and differentiate between suppliers based on ex ante risk analysis.

Wrong, dear friends. All you effectively and realistically can do is to exercise a simple check aimed at removing the suppliers who have absolutely no clue on cyber security – i.e. because they demonstrate virtually no effort, commitment or resources dedicated to the matter. Other than that, you have absolutely no way of telling which of your tech suppliers does a better job controlling its supply chain and development process. You do not have the time, tools or resources to verify the evidence, let alone go yourself and check. All you can do is tell the boys from the men – but differentiating further, especially within Fortune 500 technology suppliers is simply not realistic, for vast majority of telecommunications operators, enterprises or government authorities out there. I’ll stand corrected if you prove me wrong – but before you start shooting, do honestly answer for yourself the question if you have actually seen the inside of your suppliers factories or development centres – and the facilities of their sub-suppliers? Did you check the outcome of your supplier’s risk assessment process, , for a company named SolarWinds Inc, which I’m sure you’ve work with? It’s time for a wake-up call – according to Extended Enterprise Risk Management Survey 2020 by Deloitte, only 20% of respondents say they can effectively monitor either all or even just the more critical subcontractors. If you gave not caught up by know, you will never do – the pace of technical and business progress resulting in ever increasing complexity of supply chains will simply leave you in the dust.

What can you do instead? As your practical ability to evaluate the risk over this domain managed by your suppliers is limited due to lack of visibility and influence, therefore you should control trust and minimize risk at the point where the less-trusted domain interfaces with yours. There are three ways your supplier can harm your network’s security:

-       By people or systems that can access your network

-       By vulnerabilities in the products they deliver

-       By failing to deliver what is necessary and when it’s required to operate the network.

That’s it!

A supplier – whether it’s the threat actor or is a victim of a malicious party itself - cannot and will not send fighter jets, bombers, rockets or troops, nor can he interfere with your 5G network by magic. It needs something – a piece of technology or a pair of hands that actually touch the network. And this is what we can and should control to get what we want, and shield ourselves from the unwanted. On a rainy day we do not try to make an assessment which cloud is the most likely source of our future misery, and engage in a voodoo ceremony to make this very cloud go away – instead, we tend to focus on having and deploying an umbrella. Don’t we?

People matter

Controlling people who have access to the network requires three things:

-       The people you grant permissions to need to be vetted ex-ante, through government vetting processes if required

-       The permissions you grant as the network operator must be explicit, not implicit and bound to specific individuals with time, scope and location constraints

-       Individual operational behavior must be continuously monitored during access to your network, systems and facilities. This extends to such service models, where a technical connection is established to the systems under supplier’s control.

Whereas the latter two tools are a widely accepted best practice, somehow many operators and governments stop short of requiring the personnel having access to the 5G networks they care so much about to have a national-level security clearance. It would certainly increase the workload on the authorities running the vetting process, I understand the pushback – but since 5G networks are positioned as a national security issue, it doesn’t make sense that the people who run and maintain them, are not.

The worst vulnerability is the one that you don’t know about

Let’s address the issue of product vulnerabilities. First, from an outcome perspective it does not matter whether they are intentional, or resulting from an honest mistake. What matters is if they exist, they can be used to obtain access to the network and used for various nefarious activities. Again, there are only three conditions under which a threat actor can harm your network by exploiting a vulnerability:

-       It existed in the product you originally purchased or got introduced by the means of an update

-       It is either a piece of functionality preprogrammed to act based on specific conditions, i.e. time or date or it can be activated from inside or outside of the network

-       you cannot detect nor react in time to completely mitigate or at the very least contain the consequences of exploiting the vulnerability.

It should be said that no amount of security testing will give you 100% guarantee that a product doesn’t contain vulnerabilities. It can however limit their number, or be used as a differentiating factor in making your procurement decisions. As a principle, for security-conscious operators or countries, no version of a critical network element or update should end up on a live network without security verification, and no network element should be exempt of a sampling security verification i.e. applicable to selected updates. In 5G such critical network elements would be a subset of the 5G core at least according to ETSI/3GPP specifications.

As previously mentioned vulnerabilities may be triggered by a predefined condition or via external communication. It is therefore essential that the operators have access to testbeds that can be used as “network twins” i.e. as staging deployments in DevOps terminology for verification on how changing network parameters’ configuration or intra-network traffic impacts the behavior of network elements. In addition that the control traffic to and from the network elements on the live network is at the very least monitored and filtered for the compliance with expected traffic profile. This would require operators to set up and maintain capabilities and facilities to run, compare, sanitize and test at least the binaries, evaluate new products and updates on these “network twins” testbed prior to their deployment in live network. It is effort and investment intensive, so operators and regulators should consider some types of joint ventures focused at security verification of mobile network products– similar to joint ventures that many have set up for radio access network design and operation. The good news, is that such a solution yields a great return of investment, SolarWinds-type hacks would have most likely been detected or even completely mitigated by a combination of these reasonable and effective means.

For smaller operators or those willing to accept more risk, the basic level of assurance provided by certification and assurance schemes such as EECC, GSMA NESAS as well as these currently under development by ENISA could be sufficient. Knowing that any certification needs to be understood as a supplier capability demonstration, rather than a guarantee of the product security throughout its lifetime. Certification does confirm the quality and capabilities of given version of the product, and it may be considered a strong security assurance if the operator would use exactly the same version of the product with no subsequent updates; this is unlikely to happen for 5G network elements as they are enhanced with growing functionality over time and tailored to network operators demands.

Security through obscurity is a thing of the past

The need for detection and containment of malicious actions is of paramount importance – no supplier should make it impossible for the network operator to understand interfaces specifications or the normal traffic profile nor should it decline operators’ requirements to access important security information. For 5G networks these could translate into the following security requirements that should be included in the tenders and possibly supported by country legislation:

1.     All 5G network elements s must report to a central log repository over a secure channel in near real time without using relays

2.     All 5G network elements must support installation of 3rd party network probes and end point agents e.g. EDRs.

3.     All network elements must provide an open API to retrieve the detailed inventory of hardware and software of the network elements,

4.     All network elements must report AAA events, security events, abnormal behavior, operational status (tasks, CPU/Memory, Process list, utilization)

5.     All network elements must provide detailed accounting of their network traffic matrix (protocols, ports, etc.)

6.     All network elements must provide a best practices hardening guide and secure deployment operational manual

7.     All network elements must only install software or firmware that is digitally signed by the supplier and installed with dedicated accounts with are the only accounts permitted to perform this operation, the install process should require multi-factor authentication

8.     All software, firmware binary artifacts must be binary equivalent i.e. the compilation of the product source code matches the binary artifacts delivered by the supplier.

9.     Security patches are separated from functional patches and limited in scope.

10.  Suppliers must disclose without undue delay any vulnerabilities or weaknesses identified in their products, provide guiding on to possible mitigations if exist and develop security patches to mitigate these vulnerabilities within reasonable time.   

The one supplier process that impacts the network security posture the most, and over the whole lifetime is good old security updates development and delivery. The vulnerabilities, if found, need to be patched, full stop. There is no excuse for a supplier to provide an effective security patches for products that haven’t been marked as end of life/usage, and there is no excuse for customer not to demand it. Product maintenance, end of support and end of life aspects of course apply here – as they may limit the feasibility for a patch to be installed on an unsupported device or software platform – but in principle, the service level agreement for the delivery of security updates needs not only be in place but also be enforceable.

What if the supplier goes out of business?

In the volatile geopolitical and economic environment of today the risk of a 5G supplier going out of business is not to be neglected. Such a risk has actually materialized in the past, with Nortel Networks then the #2 global telecom equipment supplier worth at its peak 250 Billon USD in 2000 going under eight years later in 2009. What can operators and regulators do to control this risk?

A supplier going out of business means it will neither be able to deliver, maintain, update or fix products he sold hardware or software which are used in your network. The ups and downs of picking suppliers betting on the future of this suppliers’ survival is just that – it’s simply a bet, as seen in the Nortel case. Assuming that best practice financial due diligence over the supplier’s ability to deliver is done, there are two further precautions that a network operator should take to not rely on the outcome of this bet:

-       make sure that you have secured sufficient stock of spare parts

-       make sure that the source code of the network software you use – together with tools and instructions for building it – is deposited in escrow, and the contract allow for operator access under specific circumstances 

Trusted supplier – the tree with no cat

Which brings me to the topic of “trusted supplier” – or “untrusted supplier” supplier exclusion, as a method of supply chain risk management. It is a common misconception that excluding suppliers is the best risk mitigation option. It certainly works if we exclude ALL the suppliers as then you can argue that your supply chain risk has been effectively mitigated, as a result of having no supply chain. However, if you wisely decide not to go down that path, in industries with few suppliers, supplier exclusion is simply a bad a risk mitigation strategy. This is exactly the case of 5G networks suppliers, where in the majority of real-world cases an operator would choose between at most five suppliers, only four of which being a realistic choice for non-green-field 5G rollouts. It is a common misconception that limiting the choice by publicly excluding suppliers from the pool decreases the risk – if the only choice one can make is to select two suppliers from only two options, there’s really no choice nor is there a competitive market that motivates suppliers to actually raise the bar in the form of adequate security requirements, furthermore there would be no incentive for those suppliers to provide better security or enhance their products with additional security features - unfortunately this has been precisely the curse of mobile networks security for decades.

As I’ve stated before – if your assessment, based on information received from a supplier, clearly shows that the supplier has no interest, resources or capability to deliver a secure product, then an exclusion may be in order. This is comparatively easy to filter out. But the effort required to further differentiate the risk between suppliers, who have pretty much built all the existing mobile networks for decades, can be better invested in the risk management measures that were discussed before, at a fraction of the sank costs resulting from suppliers exclusions and swaps. 

Supplier exclusion has its place though as a form of punishment, for those who continuously fail to deliver on promises or refuse to cooperate for security testing or timely vulnerability handling. Putting market share at stake can be a strong motivator, especially if supported by country legislation – however, punishment should never come before the crime, and that’s why it should be considered inappropriate and ineffective to exclude established suppliers based on a theoretical risk analysis.

The consequences of barking up the wrong tree

Time and again the cybersecurity (and privacy) industries are hit by popular ideas which address an agenda often commercial or political, but from the point of view of cybersecurity and privacy these so called improvements are nothing more, than enormous waste of time resulting in embarrassment. Consider the “Safe Harbour” fiasco. Recall the Privacy Shield. Remember multifactor authentication based on SMS codes or perimeter-based network defense. What all these ideas have in common is that they were never designed to provide privacy or security. They were prostheses. They “worked” for a while with a bit of luck, but mainly because so many of us decided to turn a blind eye to obvious design flaws and the mounting evidence pointing to these mechanism inefficiency.

The “Trusted supplier” concept is yet another wrong tree we should stop barking up. Not only it does not address a single cybersecurity risk, it actually takes modern cybersecurity thinking back in time. Quite a while back, at least to the times when the Trojans chose to believe that Epeius was a trusted supplier - and decided not to poke around his product. Three thousand years later we really should know better.