The Rambo Architecture and Third Party Risk

When I first started at Netflix, we’d sometimes describe our system as the Rambo Architecture. As John Ciancutti wrote - ”Each system has to be able to succeed, no matter what, even all on its own. We’re designing each distributed system to expect and tolerate failure from other systems on which it depends.” This thinking is really at the heart of disciplines like resilience engineering and chaos engineering.?

And if you zoom out, is the overall enterprise really any different? You’re relying on dozens, hundreds, or thousands of third parties to make your organization run. SaaS vendors, business process outsourcers, cloud service providers. And yet, when it comes to making our organizations resilient to the inevitable security failures in this ecosystem, we spend our time sending around questionnaires and squinting at third party risk ratings to try and predict where the problems will occur.?

I get why third party security risk management is how it is. We can easily track how many questionnaires we send out and review. Third party risk is an amazingly complex problem and it makes us feel good to be able to do something. Unfortunately, activity doesn’t equal progress or risk management.?

So what should you do? Ultimately, I look at third party risk management as a classic operational security challenge that benefits from a pretty straightforward approach - you need to aggregate and operationalize distributed context. No AI or blockchain needed.??

Aggregating and Operationalizing Distributed Context

In modern enterprises, many teams are involved in the selection and onboarding of third parties. Procurement, finance, legal, and IT are the usual suspects, though in modern organizations you’ll often see business teams (e.g. marketing, customer service) selecting vendors and procuring solutions directly.?

The context your security team needs to manage third party risk is distributed amongst these teams. You want to seek answers to questions such as:

• Who is the business owner of the third party relationship??
• What is the nature of the service the third party is providing??
• Do you have appropriate contractual terms and conditions in place with the third party? (e.g. security terms, data processing agreements, etc.)
• Is the third party in scope of any compliance or regulatory requirements you’re responsible for??
• What is the technical architecture of the relationship? (connectivity, API keys, data being shared/processed/stored)
• What’s the impact if a security issue or other interruption affects the vendor??

The nice thing is that the answers to these questions are inside your organization. When you’ve gathered this context and operationalized it, you know your key vendors and partners, who to contact when issues arise, where your systems, applications, and data are exposed, and whether you’re covered legally if and when something happens. And you don't want these answers buried in spreadsheets. To operationalize this data, it needs to be online and available to other systems (e.g. asset inventory, incident response). And - like anything inventory-related, it’s an ongoing process that you’ll need to enrich and improve over time.?

On the provider side, I do think it’s important and valuable for vendors and service providers to be transparent and public with their security programs. What compliance regimes (e.g. SOC2, ISO, HIPAA) are you aligned with/audited against, what security incidents have you dealt with, etc. Make this information available and accessible to customers and prospects and seek increasing transparency and automation over time.?

The next time you have the urge to send an email blast to your vendors asking “are you affected by SolarWinds, log4j, CircleCI, <vuln of the week>?” take a beat. When you want to send out your shiny new custom 300 question security survey to a non-critical vendor, hit the pause button.?

Can you confidently predict when and where security issues are going to occur in your own environment?? Probably not, even though you have full knowledge of the environments you’re defending. Yet we think we can take an outside-in look at our vendors programs’ and use that incomplete, unvalidated, and likely out of date information as a crystal ball??

I hope sometime in the not too distant future we look back at the current era of finger pointing and questionnaire swapping and recognize it’s little more than another example of security theater. Until then, there’s nothing stopping you from thinking resilience first and borrowing some tips from Rambo .?