Identity crisis in AWS and Azure

One of the most acclaimed IT security enabler of PaaS is undoubtedly the introduction of instance roles, by AWS, in June 2012. What might be thought of as an EC2 quality of life feature, so purely a IaaS treat, turned out to become the most secure way of managing secrets in a world of meshed PaaS applications. The concept was so useful that it was replicated in Azure as managed identities in September 2017, despite the huge re-engineering works this would entail in backends.

Core architecture security patterns

The number one principle sustaining instance roles and managed identities is that Cloud Providers handle customer's PaaS secrets lifecycle, from creation to distribution and revocation, through their own control plane, in a seamless way.

Seamlessness is not the only voodoo operating here, there is also an alignment between the Cloud Shared Responsibility Model and PaaS secrets: the segregation of duties is clearly delineated, there is no crossing over customer-provider tenant boundaries, except at authentication time, Just-In-Time.

If we take a quick look under the hood, the management of PaaS secrets is mostly an hypervisor affair. What needs to be authenticated, if not compute workloads? And what supports compute workloads, if not hypervisors?

From hypervisors to micro-VMs

So hypervisors make for ideal agents carrying secrets to the last mile of the complex secrets distribution chain, down to the final, receiving VM or micro-VM.

Today, everything PaaS is a micro-VM. Even the most transient Lambda function is implemented as a transient micro-VM. This micro-VM revolution was made possible by another outstanding AWS security innovation: firecracker.

Firecracker is the real enabler of pervasive PaaS managed secrets, because firecraked onboarded, right from its beginning, a lightweight API called IMDS. This API is in charge of exposing secrets from the hypervisor to workloads running in micro-VMs. The concept and even the IMDS name were retained by Microsoft for exposing secrets in their own Azure micro-VMs.

So, when the number of micro-VM exploded in provider backends and when micro-VMs replaced a growing number of legacy, poorly isolated multi-tenant VMs (and cumbersome dedicated VMs, for critical workloads), instance roles and managed identities became readily available across an increasing range of PaaS.

The Capitol One breach

Despite a few warnings (read the warning bell I wrote in Via Nebulosa in 2019), things were looking bright for these managed secrets in PaaS.

Until a severe incident impacted a notable AWS customer: Capital One.

In September 2019, I analyzed the Capital One beach in a detailed article. I surmised a potential fix (resorting to UNIX sockets for protecting IMDS). Two months later, AWS turned out to implement another similar, less intrusive way to protect IMDS. In fact their solution is not just a single fix, but a set of measures. This set as a whole is now referred to as IMDSv2, and you are probably aware that it took AWS 4 years to decommission v1, despite constant pressure from the IT security community, with Scott Piper on the front line.

The lesson that we should have learnt from this incident is what I explained in my warning bell article: there is a fundamental contradiction between the concept of a managed identity (or an instance role) and its implementation as a vehicle. Here, the vehicle is a secret. Why should a workload be entrusted with a secret to empower it interacting with PaaS? Isn't identity the DNA of a PaaS workload?

Couldn't the workload interact directly without authentication, on a need to know basis, based on its DNA, whithout further need for empowerment? Why do we need IMDS?

Engineering is the reason. Getting rid of secrets and their distribution through vehicles is an almost impossible technical task.

So, if secrets have to be carried to micro-VMs, providers have better triple check that they are bound not only to the micro-VM, but to the underlying workload.

Identity crisis in 2023

Thunder stroke again in March 2023, when Karl Fosaaen from NetSPI made an amazing discovery in the heart of Azure function apps: some Managed Identities credentials could be decrypted and exfiltrated from environment variables.

The security foundations of Managed Identities, that we thought were acknowledged once and for all during the Capital One intrusion, were proven to be broken once again.

In Capital One, the session tokens of instance roles could be exploited by "remote controlling" the micro-VM from an adversarial environment.

Here, the persistent secrets themselves could be not only retrieved, but also be used at will from anywhere outside of the trusted, customer tenant.

Conclusion

Cloud security architecture is an almost limitless source of innovation and is definitely a great business enabler in PaaS. Instances roles and managed identities are, for me, the best example I can think of.

But what is the point of designing great patterns if one can't abide to them?

Fortunately, the feature disclosed by NetSPI didn't have any customer impact.

When will be the next identity crisis? It can/should/MUST be prevented.

Confidential computing is a promising long term option for replacing vehicles by a sui generis, authentless workload identity. More of that in a later article, when confidential computing becomes production grade…

Further reading

Mistaken identity, by Karl Fosaaen: https://www.netspi.com/blog/technical/cloud-penetration-testing/mistaken-identity-azure-function-apps/

Greatness and fragility of managed secrets, viaNebulosa: https://www.dhirubhai.net/pulse/demystifying-paas-security-part-3-christophe-parisel/

Analysis of the Capital One incident, viaNebulosa https://www.dhirubhai.net/pulse/aftermath-capital-one-incident-aws-christophe-parisel/