The Key to Breakthrough Performance

In the modern programming world, performance and scalability are always crucial factors for building robust and reliable applications. Nearly all programming languages are constantly evolving to meet this demand. While current models we use provide high performance, they still have certain limitations.

A few years ago, most systems deployed their services using the "blocking I/O model." This model gradually became inadequate for scalability when system load increased (1 thread - 1 socket), leading to a rise in the number of sockets and subsequently OS threads, sometimes reaching the OS's maximum thread limit.

Fortunately, several non-blocking I/O models emerged in the market to address heavy I/O tasks. These models use fewer threads to handle multiple socket connections (1 thread - n sockets). This effectively solved the load issues at the time.

Over the years, as the number of users increased significantly, and more products and services emerged, with more data, some drawbacks of the "non-blocking I/O" model started to surface. When handling requests involving database interactions or cross-service calls began to slow down, it led to I/O threads being blocked. To prevent these I/O threads from being affected, the model evolved further to handle asynchronous requests using separate threads, termed “non-blocking + async.” This model is still in use by us for building high-load systems.

However, “non-blocking + async” still has its limitations, particularly in elasticity when handling asynchronous processing. For instance, we have a thread pool with a max thread count of 1000, used for handling asynchronous requests. Initially, we set the thread count to 50, but during peak system loads, the number of threads could expand to 500-600 threads, and these threads are not reclaimed when no longer in use. Additionally, if there's a database issue or cross-service call timeout, the threads could expand to the maximum limit of 1000. Once the system stabilizes, the thread pool remains at 1000 threads without being reclaimed. This raises the need for a scalable solution where threads can be automatically reclaimed and freed when not in use.

We researched and found a solution using "Virtual Threads." In the traditional thread model, creating a thread object maps 1-to-1 with an OS thread. However, with "virtual threads," it maps n virtual threads to m platform threads to m OS threads. This relationship allows the creation of virtually unlimited threads. For example, there could be 1000 virtual threads - 200 platform threads - 200 OS threads. Virtual threads are managed by the application and can be reclaimed when not in use. Although the 200 platform threads and 200 OS threads cannot be reclaimed, this model is still significantly better.

The development of models generally aims to overcome drawbacks that impact growth and expansion. By understanding the limitations and business requirements, we can easily decide on the suitable model and strategize effectively for system expansion.

The illustration is in Java, but this mindset can be applied to finding suitable languages and technologies.

#Java

#JavaVirtualThread

#JavaMultithreading

#ReactorPattern

#EventDriven

#wecommit100xshare