You're Killin' Me, AWS!

Part I: The Load Balancer Club Sandwich

I really do like you Amazon Web Services (AWS) , but sometimes I get a little frustrated by amazing existing services or promising new ones that are just lacking one or two critical features. While there are typically workarounds that have been documented by AWS Professional Services (which I appreciate), not everything can or should be fixed with duct tape or Lambda functions.

As a co-author for SANS Institute SANS Cloud Security 's SEC549: Cloud Security Architecture, I get the opportunity to research a variety of different cloud topics and services. As part of this research, I have been keeping abreast of some of the new services announced at re:Invent. Stumbling across the presentations for Amazon VPC Lattice and AWS Verified Access this year, at first glance, I was super excited about the possibilities only to have this excitement tempered by some crucial missing features and odd design decisions (see Part II and Part III). I was hoping that these services would also offer alternatives to some of the design elements of our transit gateway hub and spoke lab environment that could be improved. But, alas, it wasn't meant to be. At least, not yet.

One issue I have with the hub and spoke model in AWS, is that if I want to centralize ingress with native services, I have to set up what is called the "Load Balancer Sandwich" (see elastic load balancer (ELB) Sandwich, https://aws.amazon.com/blogs/networking-and-content-delivery/design-your-firewall-deployment-for-internet-ingress-traffic-flows/). However, when doing this in a hub and spoke environment, there is a glaring deficiency that is not always highlighted well in the documentation. The internal load balancer or network interfaces in the spoke networks that you are targeting have to support static IP addresses. This is fine if you can use a network load balancer (NLB) instead of an application load balancer (ALB) (only NLBs support static IP addresses), but what if your target is a Lambda function (only ALBs can target Lambda functions)? Well, you have two documented approaches (that I am aware of) unless you want to go with non-native solutions that support FQDN or hostname targeting.

The first option, and really the only one that would make sense to run in a production environment, is to create what I have decided to call the "Load Balancer Club Sandwich". This is where you have an ALB or NLB in your centralized, ingress VPC. That load balancer targets the static IPs associated with an NLB in your spoke workload VPC, which then targets an ALB, and finally, that ALB targets the Lambda function (see note at the top of the reference for solution 2 in next paragraph for details on this model).

What is going on here? Why am I paying $45 for something I should be able to consume for $15 or at the least $30? Oh right, it is all that extra bread! Now, NLBs were not always able to target ALBs and so the previous solution used Lambda to dynamically update the IP address targets of your ingress ALB/NLB (see https://aws.amazon.com/blogs/networking-and-content-delivery/using-aws-lambda-to-enable-static-ip-addresses-for-application-load-balancers/). While at least we had a workaround, it is recommended you run the Lambda function every minute (we decided to use EventBridge to capture the IP change instead) to check DNS to see if the IP has changed and update the target groups. Whether you run it on a schedule, leverage EventBridge, or both, there is no way that your load balancer targets in your ingress VPC remain healthy the entire time and requests can and will get dropped during the change. How can any organization be comfortable with a solution that they know is going to drop some requests when the ALBs IP addresses change?

While the "Club Sandwich" model with an extra slice of NLB is now available and does not suffer the same deficiencies, I share the old model to highlight the use of workarounds which are helpful in the short term, but not helpful if they delay the real solution. That being said, I still don't know why I have to add yet another load balancer to my hub and spoke design to get this to work. There are more than just Lambda functions that require, or work better with, an ALB than an NLB. If AWS can change NLBs to support static IP addresses, why not ALBs as well?

While there may be some explanation for why ALBs can't have static IP addresses (technically, I can't imagine what that would be), there are other solutions that could be implemented to solve the problem and some of these would also solve other problems that I will address in future articles. My favorite solution to the problem would be to allow FQDNs or hostnames as targets. This would allow so much more flexibility in the architecture and design of hub and spoke networks in AWS. There are other solutions that support this so I would like to know why this is not possible with ALBs? What makes this technically infeasible? (ideally, ALBs should be able to support static IP addresses and FQDN targets cause, why not?)

Another option would be to allow NLBs (and why not ALBs too) to target ALBs in other VPCs. To make this happen, you could support using a load balancer ARN in the target configuration or require that the load balancer or load balancer subnet get shared (via the Resource Access Manager (RAM)) with the load balancer or load balancer subnet where the ingress load balancer target groups are being configured. This is not supported today as NLBs can currently only target ALBs in the same VPC.

A third option would be to allow ALBs to target VPC Lattice services, but I will discuss this more in Part II.

I am sure there are other solutions, but what I would like to see is something that doesn't require any workarounds and doesn't make me set up a totally redundant proxy to get everything working consistently. For a single, secured connection, I now have 4 or more TLS sessions being negotiated between the different components. This amount of proxying seems excessive. I would much rather remove that middle slice of bread; it just adds to the carbs and the cost.