Load Balancers and How to Build Them (Part 1)

Load balancers - Yes, you might have heard this very common term in today's world of open source and cloud development. What do we do when we need to scale our applications? We put them in front of a Load Balancer ??. But how does this actually work? What does a Load Balancer do? Can we build one for ourself?

If you have not subscribed to this newsletter, subscribe it now - System Design Newsletter by Shrey Batra

What are Load Balancers?

Very simply saying, let's imagine multiple instances of your backend service in your cloud. You need to distribute the incoming requests from clients (or frontend) to your backend services, so we use a Load Balancer which is a "single point of entry" to our system, and distributes the requests to each running instance at the back.

What do you mean by Single Point of Entry?

Well, when you say you have 100 running instances of your application, does the client or frontend need to learn 100 different IP addresses and then take intelligent decisions where to send the request to? No right? Therefore a single point of entry (let's say a DNS - example.com) sends all requests to a single service (the load balancer) which in turns forwards the request to each running instance.

Yes, sometimes client's learn these 100 IPs and it's call Client Side Load Balancing but that is out of scope of this article.

Distribution Algorithms

Now, how do you think a Load Balancer distributes the requests to the instances behind it? One would say Round Robin where we distribute equally in a circular fashion going over each instance available. I would argue, that Round Robin might not be ideal for "equal distribution" of work.

When do we have only 1 or 2 types of APIs in any backend service? There are typically 10s if not 50s of APIs in any backend microservice, each having its own business logic and performance benchmark.

A simple GET API might have 30ms latency whereas a POST create API can have 200ms latency if you involve external system call, some validations, etc.

Therefore, we need to check the "work weights" of each API call and then intelligently take the decision in Load Balancer.

State in Load Balancer and Performance Considerations

As mentioned above, we need to capture state of each API in our Load Balancer. Another use case could be to maintain avg response time of each instance and forward the request to fast servers. Yet another could be to maintain least number of active requests per server.

As and when we increase our state maintenance, we increase the network lag overhead of each API call. Each single operation over here means we need to spend more time in Load Balancer before our requests can reach the application instances. Hence, using Redis/cache/Key Value based databases are used on these layers to speed up performance..!

How and where can Load Balancers be used?

Anywhere where you have multiple instances running and need a single point of entry, we use a Load Balancer -- scaling our backend services, scaling our databases, etc.

Yes, we can also put a Load Balancer in front of databases when we have replication in our databases -- multiple servers which could handle our database queries. This is normal and mostly already there, written and deployed by database developers themselves..! We don't need anything extra in this.

Single Point of Failure - Scaling Load Balancers

Yes, Load Balancers can easily become single point of failures for our system, as they are the single point of entry to the applications behind it. How can we improve it?

Simple, partitioning and replication to our rescue ??

So, let's say our Load Balancer is down, how do we keep our application live? We add replicas -- secondary load balancers which take up duty when primary goes down.!

What happens when we have too much scale and single Load Balancer cannot keep up with it? We can answer this ourself -- we scale our application and put a Load Balancer in front.

Similarly, we can scale our Load Balancer application - multiple load balancers each having it's own set of application instances to forward request to. When we talk about Mirror's like eu1.example.com, eu2.example.com, these are mirrors of load balancers distributing your requests..! As these have static IPs / DNS, it is easy for client's to find multiple load balancers and interact with them.

How to Build Load Balancers?

Seeing the above things, how hard is it to build a Load Balancer, probably a simple poc one? All we need is a Server, which has a pool of backend instances already known, and forward the request to the backend server?

We will see sample code where we can build a simple Load Balancer in like 15 min..!!

If you don't wanna miss out, subscribe to my newsletter - System Design Newsletter by Shrey Batra ??

Conclusion

There is much more to discuss over here, one such example of advance load distribution was done by Twitter which I have written in a separate post..!

If you like this content, I urge you to please click the Share button and repost..!! ??

You likes, comments and feedback are highly appreciated..! ??