Low-latency software architecture is a design approach that aims to minimize the delay between the input and output of a system. It is especially important for applications that require real-time responsiveness, such as online gaming, streaming, trading, or IoT. In this article, you will learn some principles and practices that can help you design a low-latency software architecture.
The first step to design a low-latency software architecture is to define your latency requirements and measure your current performance. Latency can be measured at different levels, such as network, application, database, or end-user. You should identify the sources and types of latency in your system, such as network latency, processing latency, or queuing latency. You should also set realistic and achievable latency goals and SLAs for your system, based on your business needs and user expectations.
Systems that are latency sensitive design can vary significantly based on the latency requirements. The latencies can range from ~100 uS e.g., Stock Exchange execution to ~100 mS e.g., real time fraud detection in payments to ~1 second which is typically high performing web systems. The latency requirements should be defined in SLA measured over percentile e.g. < 100 ms for 90th percentile & < 500 ms for 99th percentile. By having concrete latency requirements we can design the software architecture that will meet these requirements.
Apart from the requirement, we need to be aware of the near ideal latency, realistic latency achievable. But realistic latency may not be defined in isolation. There will be many parameters affecting it. How much processing power, and memory you can throw at your service to run this software, how efficient you want your system to be, which can also be influenced by its cost and the availability of the resources not just to your service and team but also in the industry and market. Remember that we can't be providing the 100% perfect SLAs, we need to find a balance between how fast we can ship something, which is useful to the client/users at this moment, without much compromise to the user experience and cost.
The second step to design a low-latency software architecture is to select the appropriate technologies and tools that can support your latency requirements. For example, you should choose a programming language that has low overhead and high performance, such as C++, Rust, or Go. You should also use a framework or a library that can handle concurrent and asynchronous requests, such as Node.js, Reactor, or Akka. Moreover, you should use a data store that can provide fast and consistent access to your data, such as Redis, Cassandra, or MongoDB.
A common technique in low latency architectures is to pre-compute results wherever possible and retain it in primary memory so that there is no I/O or computation required at request time. Results may be retained in the serving process's memory or across a process boundary in a service such as Memcached or Redis. An example of the former approach is indexing in single instance database systems; this approach provides the maximum latency benefit. The latter approach of using an off process cache, exhausts some latency budget for the benefit of being reusable across multiple service processes; this approach is commonly used in web serving clusters.
Based on the requirement the right architecture is to be selected e.g., for very low latency requirements most systems are event driven and written grounds up with C, C++ or GoLang that gives control over how operating systems handles memory, IO etc. rather than a garbage collected language like JVM languages. For latency sensitive web systems any of the web technology can be chosen however it is software architecture that defines how it will perform.
The third step to design a low-latency software architecture is to optimize your network and communication between your components. You should reduce the number and size of your network calls, by using techniques such as batching, caching, compression, or aggregation. You should also use protocols and formats that can reduce the overhead and latency of your network communication, such as HTTP/2, WebSocket, or Protobuf. Furthermore, you should use techniques such as load balancing, CDN, or edge computing to distribute your workload and reduce the physical distance between your servers and clients.
Network & communication is critical to latency management. For low latency application the network IO should be eliminated or limited. For web interactions WebSockets is a good option that provides open socket channel and can enable low latency experience on web browsers e.g. Stock Trading Terminal. Another key approach is to manage the HTTP protocol e.g. by setting NODELAY to ensure the latency is not introduced by optimisation done by Nagle algorithm.
The fourth step to design a low-latency software architecture is to design your system for concurrency and scalability. You should use a microservices or serverless architecture that can decouple your components and allow them to scale independently and dynamically. You should also use a message broker or a stream processor that can enable asynchronous and event-driven communication between your components, such as Kafka, RabbitMQ, or AWS Lambda. Additionally, you should use a reactive or an actor model that can handle concurrent and non-blocking operations, such as RxJava, Vert.x, or Erlang.
Concurrent design must be accompanied by also splitting computation into parallelizable units where possible. Long running transactions by their very nature will be high on latency. In order to improve latency, transactions may be broken down into workflows where tight consistency requirements may be relaxed in favour of an eventually consistent data model. Of course this may not be applicable in all domains e.g. financial transactions.
Microservices have become default for many organizations.. but quite often they try to implement strong consistency model. with distributed services and designing for strong consistency compromises on availability, latency & scalability.
So many projects I evaluated as a consultant were very tightly coupled. Everything had to happen in order before any results were published. Moving towards an architecture that is multithreaded and loosely coupled is a great way to scale. One example I can think of is Google BigQuery. You can query data that is still enqueued to be placed in the table, so you don't have to wait for it to be fully processed before you query it. Very cool!
The fifth step to design a low-latency software architecture is to monitor and troubleshoot your latency issues. You should use tools and metrics that can help you track and analyze your latency performance, such as latency histograms, percentiles, or heat maps. You should also use tools and techniques that can help you identify and isolate the root causes of your latency problems, such as tracing, profiling, or logging. Moreover, you should use tools and practices that can help you test and improve your latency performance, such as load testing, benchmarking, or A/B testing.
Observability is key to measuring & monitoring the latency and ensuring the system requirements are met. Some steps that can help - Log each system latency inner wall to wall - Log latency of each network separated component in the stack - Use correlation keys and distributed tracing to track the performance end 2 end in a complex system - Beware of heisenberg principle
The sixth step to design a low-latency software architecture is to apply design patterns and best practices that can enhance your system's performance and reliability. Circuit breaker, bulkhead, CQRS, event sourcing, and backpressure are all common patterns that can help you achieve this goal. Circuit breaker prevents cascading failures and timeouts by breaking the connection between faulty components, while bulkhead isolates and limits the resources allocated to each component. CQRS separates the read and write operations of your system, reducing contention and complexity of data access. Event sourcing captures and stores state changes as a sequence of events for fast replication and recovery. Finally, backpressure controls the flow of data between components to prevent overload and congestion.
An oft-overlooked possibility is to offload computation to clients so that more compute cores and memory are employed overall. For example, in AI/ML based applications, inference may be offloaded to clients which have local access to the latest version of the model. This will cut latency by eliminating round-trips to the server. Cutting down network round-trips almost always provides significant latency benefits especially for end-user facing applications.
If min. latency is required among services, consider the use of Webflux( Springboot). If min. latency is required among client and server then think upon using sse( server sent events), in case consumer is browser based. It will minimize the latency.
Architecture is more important than tools. Define specialized processing units. Define the communication protocol and interface. Define the on demand scale up. Define the on hotspot relocation. Identify the critical process. Introduce competitions in implementation. Implement isolation to prevent communication interference.