How do you implement horizontal scaling in your system?

Above all, when architecting your application, choose components that allow for horizontal scalability. Replacing vertically scaling building blocks (like a relational database) with horizontally scaling building blocks later can be expensive. Of course, not all components in the architecture may need to scale horizontally, so consider thinking through where bottlenecks would arise as your system scales up.

These metrics could be trickier to obtain than it seems on the surface. There will be a lot of things you can measure and improve. A lot of things will be inversely related and you have to take into consideration of business value. For example, if the fastest, end to end performance is the goal, a more monolithic architecture will be faster due to less inter process communication vs a micro service architecture. Another sin is premature optimization in system architecture, the cost usually comes in the form of maintenance and bugs, every abstraction you make multiples the cost down the road. And Horizontal scalability is all about decoupling and adding abstraction layers.

Choosing a load balancer should most definitely not be the 2nd step. First step should have been gathering non functional requirements with business. And then agree on KPI to measure which would be 2nd step. 3rd step is a design that flexible enough to accommodate these measuring steps, no more and no less. 4th step would be choosing implementations that fulfill accommodate these designs. LB may or may not be part of this. This is way too specific of a task at this stage.

Haha what? oK AI has lost it at this point. Replication is not usually a topic under horizontal performance. It is for data protection though.

A bold experiment our team did was to recompose Kubernetes in what we call as "Scale-out architecture" driven by the observation that scaling limits for the number of nodes a K8s cluster was based on the capacity of etcd to handle object and event reads/writes. In a nutshell, we partitioned the responsibilities of managing the nodes and managing pods to different clusters (etcd stores) and wired them up with Kubernetes scheduler changes. This way of partitioning responsibilities, in theory, allows us to horizontally scale K8s clusters itself. For details, please see: https://sched.co/11Npw

To protect data from loss in a failure or catastrophe, You should replicate data across regions. Replicating data allows for secondary DBs to be used for READ operations, resulting in faster response times. It's important to consider the disadvantages. When a WRITE request is made to the primary DB, there may be a delay in response until the data is replicated. This delay occurs because the primary DB must replicate the data before responding. With asynchronous replication, the primary DB responds immediately after the action is performed, even though replication may still be in progress. If a READ request is made to a secondary DB after a WRITE operation, there is a possibility of receiving incorrect data due to incomplete replication.

Also not a major step in horizontal scaling as horizontal scaling is mostly about decoupled architecture so that you can independently run, measure and provision on demand.

Flamegraphs are a very useful tool to analyze code and look for areas to optimize that can yield the biggest bang for the buck.

It depends on use cases. Simple, stateless applications can scale well horizontally but it is not as easy with more complex, stateful applications. IMO, a well thought out combination of horizontal + vertical scaling is an approach to consider. Recently, Amazon Prime Video team published a blog detailing how they leveraged monolith architecture because cost and overhead of distributing load across micro-services exceeded the benefits of horizontal scalability in their case. Looking closely, they recomposed the modular implementation blocks from micro-service architecture, added an orchestration system within the monolith to horizontally scale detectors within the monolith (vertically scaling the monolith), then cloning the detector service.