Under the surface of Azure Container Apps

Azure Container Apps is a multi-tenant, serverless compute solution for containers. It is based on Kubernetes, but since Microsoft manages all cluster components, end-users have no access over the Control Plane, including the kube API:

A good way to scout what end-users have access to is to remote connect to a container through an SSH or RDP session. But it is not a supported option in a serverless environment: end-users are not supposed to log in a container, they are not given a password or an authentication key.

So a typical workaround is to run a web shell into a live container: A web shell is an application like any other, except that it answers to system (UNIX or Windows) commands through an HTML form. The following command is run to spin up a tiny python web server (with CGI-BIN module) when the container is deployed in Containers Apps:

The application itself is a simple Python script called shell.py, which generates the HTML form on-the-fly when the script is called from the cgi-bin directory.

When all this is in place, you get a plain but operational command line interface. Let's look at what you get when you type-in 'id' and hit the submit button:

As stated in Azure documentation, the user runs in unprivileged mode. The GID is a bit unexpected, but I couldn't exploit SUID or SGID bits to escalate privileges, it seems to show that Linux namespaces are properly enforced.

So what can we do from that point on? Out of the several very tempting options (probing Microsoft's managed VNet, using a trampoline container to escape to the micro-VM...), there is only one which is perfectly harmless to go: see what we can grab inside our local container.

What's the point probing a container that we built and own? Well... When a docker image gets instantiated, it does more that bringing up the resources we planned in the Dockerfile. It adds additional, guest-mounted resources. What is mounted is highly dependent on the local implementation: thus, unlike what is commonly thought, containers are not 100% platform agnostic.

In the case of ACA, the container you run on your local workstation is not exactly the same as the container that Microsoft runs on its managed platform: the Cloud provider adds stuff through agency of "their" guest:

As just said, the local implementation rests on Microsoft's shoulders. Let's see what mount points they have come up with by issuing a mount command:

One of the mount point (highlighted in orange) is a kubernetes.io service account. That's very unexpected, because as we said in introduction, the control plane is fully managed by Microsoft and we have no reasons whatsoever to be given a service account: a service account needs a kubernetes API to connect to, and we are not granted such a luxury in Container Apps' operating model.

The service seems to be unintentionally exposed in user land. It reminds us of the importance that the kube API be protected by a firewall!

Let's try to figure out whether this service account is shared (multi-tenant) or dedicated by dumping its namespace:

Remember that Kubernetes namespace names are unique within a cluster, right? To me, if a namespace is statically called 'k8se-apps' with a trailing s, chances are it is holding either multi-tenant applications or a Microsoft managed cluster wide application.

Finally, for the service account to readily exploitable, we need its token, the public root certificate of the Kubernetes API and connectivity to the API server. Getting access to the token and the certificate was a simple matter, as shown in the example below:

After checking with Microsoft, this token is perfectly harmless because no permissions are attached to the service account: to our relief, the Cloud Security Shared Responsibility Model is perfectly honored with regards our finding.

As for API connectivity, one would need to scan Microsoft's network in search for the API, a move I was not ready to make.

Sticking to my initial posture (to grab whatever I could within the boundaries of "my" container), I tried to gather intelligence from a potential candidate IP address in /etc/resolv.conf and other files mounted by Microsoft, to no avail.

Conclusion

Container Apps is quickly becoming one of my favorite Azure services: this is what prompted me to known more about it and to perform some investigations in the first place.

I hope you've enjoyed the little dive which purpose is to show how one can investigate PaaS compute "from the inside". We've discovered a harmless, but unexpected feature along the way.

Microsoft informed me that the unexpected service account would be removed in a next update of Container Apps: this is a sound practice given that small rocks can always become mountains if left unattended for long enough.

For much more advanced techniques and actual exploits, I invite you to read the fantastic report #unit42 published last week: it's called "Digging inside Azure functions": https://unit42.paloaltonetworks.com/azure-serverless-functions-security/