Immutability 101

In our world of Continuous Delivery, building immutable infrastructures sounds like an unattainable goal. But immutability has so many benefits in terms of security and quality of service that security architects shouldn't give up on it so fast.

Cloud environments, especially the public Cloud PaaS, offer the best opportunities for making the dream come true. In this context we should reconsider what must be immutable and how to make it so.

Immutability - how?

How immutability is to be enforced depends on your Cloud security posture and the capabilities of your Cloud provider. My recommended stand to solve this is to leverage native, in-band controls.

For me in-band control is the only current way to ensure lightweight and continuous immutability (among other security measures). I explained this in two previous articles: one build-oriented, the other run-oriented.

Immutability - what?

To make a viable, production-ready solution, only a specific set of sensitive properties must be immutable, not the entire deployment obviously. The scope under consideration lies first and foremost in the Control Plane of your Cloud deployments and must also embark the entities acting directly upon it (devOps pipelines, emergency troubleshooters), which I will call 'authoritative sources'.

But which Control Plane exactly? There are several nested planes into the PaaS model, here I mean the top level one, the PaaS Control Plane. At this level, you have to identify and set or freeze the items that you need to shield against risk scenarios (eg: an execution error, or an automation key compromission). This might sound daunting unless one thinks in terms of integration process.

The major advantage of following this process comes from the fact that, unlike on-premises, the number of PaaS integration options and degrees of freedom are extremely limited.

The services integration process

All activities where sources interact with the PaaS Control Plane and fire a state change in your PaaS architecture fall under the services integration process.

There are two sides to PaaS integration:

services composition, an action performed to mix, match and instantiate one or more services from the Provider PaaS catalog,

services chaining, an action performed to instantiate a service in the PaaS catalog (the 'master' service) which depends on other services ('ancillary' services).

Service composition

Composition is by far the most common PaaS activity. What needs to be done here is to identify and monitor the parameters that an authoritative source has to set in the Control Plane of each service independently from one another (ie vertically), and how they reference one another (ie horizontally). The authoritative source sets the nominal, streamlined process in line with the use cases of her business line. Other behaviors should be considered as outliers and can be excluded from the monitoring activity; they can always be added later if necessary.

For example, an Azure Service Bus may be configured with VNET integration, meaning that the bus will reference a VNET service in its definition. It may also reference the private endpoint of an Event Grid service for the processing of a dead letter queue: in this scenario you have a composition of three PaaS, so you have three sets of vertical parameters to collect and monitor (or enforce) as well as three sets of horizontal parameters (Service Bus->VNET, Service Bus -> Event Grid, VNET -> Event Grid endpoint).

Service chaining

There is a difference of philosophy between Cloud providers when we speak about service chaining:

• Azure Resource Manager handles ancillary services pretty much the same way as it handles service composition: as long as the authoritative source has proper Azure Roles (RBAC) in each service, the master service and all its dependencies will be instantiated. The only difference with service composition is that reference to ancillary services must be explicitly accounted for. An extreme example is Service Fabric; it uses a minimum of one Azure Load Balancer, a number of Storage Accounts, VM Scale Sets, a VNet and an optional Key Vault.
• AWS handles service dependencies like Azure in some cases (eg: the way ECS references Application Load Balancers), or by using service linked roles in other cases (eg: the way ECS manipulates CloudWatch). In this latter situation, the authoritative source will not perform the operation directly on ancillary services. She will pass a role to the master service that will make the subsequent updates to the dependent services on her behalf based on a trust relationship between master and ancillary services. The authoritative source may or may not have the permissions herself, but the passed role has.

The drawback with the former way is that it requires more configuration efforts on both Tenant and Provider sides since more explicit parameters must be monitored. The drawback with the latter is that implicit parameters are not well documented for the Tenant (this is understandable since policies are managed by the Provider) and prone to change without notice that could break your controls.

It is important to be aware of both paradigms for proper management of transitions, i.e. to ensure that the resulting state is acceptable in terms of immutability with regards to the risk scenario(s) you plan to mitigate. You want to avoid a contamination from false negatives that would render the whole scheme useless.

Final word

There's more to say on the matter, I plan to write a sequel of this article in the next months. Hopefully this article will provide you with sufficient information to get started and realize that we are all getting closer to immutability than we could imagine!