Some reasoning over using threads based web servers

Well Known that widely-used programming languages supports threading model (multi-threading), such as Java, C++, Python, C#, etc.

The threading model or so named "thread-per-request" model is a traditional approach to handling web requests in synchronous programming models where each incoming request to the WEB server is handled by a dedicated thread. Just this thread is responsible for complete processing the request and composing response. Two main threading models are most famous currently: Platform Threading Model and Virtual Threading Model.

This article has a goal to show out how using threading models in Web servers and evaluating them functional possibilities. Here the Web servers on Java platform will be evaluating only by using Ubuntu 20.04 system.

Platform Threading Model

On a thread based system the web server creates a new thread as soon as a request receiving. So all the work for this request is handled by that thread. Such thread is named often as platform or real thread due to its implemented as a thin wrapper around an operating system (OS) thread. Every thread exists until his work will not be finished, independently it's in waiting state or uses processor resource.

This means, that the maximum of available threads on a computer determines how many concurrent users can send request to the web server simultaneously.

Let say, Tavr – average processing time of request and RQS – requests per second.

Therefore RSS = 1/ Tavr is average responses per second.

If RQS < RSS then all is OK and server works quite calmly, the number of threads is not growing. The minimum count of threads are enough to be processing every request.

But a dangerous situation could occur in case of RQS > RSS. In this case the number of threads will growing and can reaching critical state when machine resources are overhead.

The number of threads during some time can be evaluated by using simple formula:

Nt = No + (RQS - RSS)*t

where Nt – current number of threads

No – initial number of threads

t - observation time

Thus, for instance, if No=10 (initial threads pool) and (RQS – RSS) = 10 Req/Sec then in 10 min (600 sec) the count of requests processing threads will be already above 6000!

Sure, above mentioned formula is abstract and idealized representation of real situation but anyway it gives some insights about existing problem concerning usage of threading model.

Take into account also that every thread is in fact a lightweight process and naturally as many processes are working in one machine then more slowly they working because of competition for machine resources.

Note also than the maximum count of threads in any machine is limited. In practical terms, this limit is influenced by the operating system and the available system resources, such as memory and CPU cores. So, the only way to rectify such situation is scaling (horizontal or vertical).

Let’s to do some experiment and use for instance a simple HTTPS server build on Java platform.

The certificate for HTTPS server can be generated like described in article. Sure, of course. we can use JKS certificate because many Java developers are still used it. But notice that PFX format certificate (PKCS12 KeyStore) is recommended to use now in Java (since JDK 17.x) instead of deprecated JKS KeyStore. Therefore we will use the PFX format, given that in the current test we are using JDK 19.x.

Start the server and evaluate the benchmark.

To evaluate the performance, we will be used autocannon tools to benchmark test our HTTPS server (results of test shown below):

HTTPS Server receives quite big of client requests during 10 sec. Active threads count evaluates every 1 sec.

Above depicted test results shows that during 10 sec HTTPS server was attacked with speed 2076.9 Req/Sec in average by using 500 connections (users). The average respond latency was 240 ms. Note that about 60 requests was finished with errors (not correctly answered). The requests service threads count was growing quite speedy and reach up to 2252 threads in 10 sec.

Anyway, let’s try to highlight some widespread advantages and disadvantages the threading model.

Advantages:

• Improved performance: Threading model (in contrast of single-threaded) can help increase the overall performance of an server, especially on systems with multiple processors or cores.
• Responsiveness: Threading model can prevent server blocks (unresponsive state) by running long-running requests in separate threads, ensuring the server remains responsive.
• Better resource utilization: Threading model allows better utilization of system resources by keeping the CPU busy while waiting for I/O operations to complete.
• Parallelism: Threading model, by its nature, enables parallelism, which can lead to significant performance improvements in server.

Disadvantages:

• Complexity: Threading model adds complexity to the program, making it more difficult to design, implement, and debug.
• Synchronization overhead: To avoid data corruption, access to shared resources must be synchronized , which can result in additional overhead and reduced performance.
• Context switching: Context switching between threads consumes CPU time and system resources, which can lead to performance degradation.
• Resource Overhead: Since each thread actively uses system resources (RAM and CPU) then the creation of large number of threads can lead to excessive resource consumption and potential system instability.
• Hardware limitation : The performance of threading model are limited by the number of available resources (cores, RAM, threads,..) in the system.

Virtual Threading Model

Oracle company introduced recently so named virtual threads in Java (preview feature since 19.x). This functionality eliminates some of the disadvantages associated with traditional threads based servers. Virtual threads are an abstraction layer between operating-system processes and application-level concurrency which is ideal for high-concurrency scenarios without significant resource overhead. They can lead to practically unbounded resources usage in highly concurrent web servers where thread creation is a bottleneck.

Above mentioned resource contains also the WEB server script (ServerV.java) that used virtual threading model.

So we can start similarly the server by using the following command:

Note that java 19.x virtual threads is in preview state for now. Thus we should use –enable-preview parameter in start command to be used this functionality.

To see the performance, we will be used again autocannon tools to benchmark test our HTTPS server (results of test shown below):

During test no any real thread (OS threads) is used for processing requests and preparing responses.

Above depicted test results shows that during 10 sec HTTPS server has been servicing 500 connections (users) and receiving about 560.9 Req/Sec in average. Throughput was reaching to 636 KB/sec in average. The average respond latency was 1334.5 ms. Note also that no any requests was finished with errors.

Let’s try to highlight some advantages and disadvantages of virtual threading model in java.

Advantages:

• Scalability: JVTs can scale to millions of threads without significant overhead, making them ideal for applications that require handling a large number of concurrent operations, such as network requests or database queries.
• Memory Efficiency: Virtual threads are much lighter than platform threads, requiring significantly less memory. This reduction in memory usage helps prevent OutOfMemoryError exceptions that can occur with traditional threads when creating a large number of threads.
• Simplified Concurrency Model: JVTs simplify concurrent programming by abstracting away the complexities of thread management. This can lead to cleaner, more maintainable code, especially for developers new to concurrent programming.
• Compatibility with Platform Threads: Virtual threads are designed to be compatible with platform threads, allowing developers to migrate existing applications to use virtual threads without significant refactoring.

Disadvantages:

• Performance: While virtual threads excel in I/O-bound tasks, they may not provide significant benefits for CPU-bound tasks. Platform threads, which are more suited to CPU-bound work, can still be more efficient for these types of tasks.
• Garbage Collection Impact: Creating and destroying virtual threads can lead to more frequent garbage collection activity, which can impact performance.
• Debugging: Some monitoring and debugging tools may not fully support virtual threads yet, potentially making it more challenging to diagnose and troubleshoot issues in applications that heavily use them.
• Complexity: The "thread-per-request" model that virtual threads promote can introduce complexity and potential pitfalls, such as synchronization issues, race conditions, and deadlocks.

Notice that although Java Virtual Threads offer significant advantages in terms of scalability, memory efficiency, and simplification of concurrent programming models, however, they also introduce challenges, particularly in terms of performance for CPU-bound tasks, the impact on garbage collection, and the need for updated debugging support. Developers should carefully consider these factors when deciding to adopt virtual threads in their applications.

Short comparison

Both threading models tests averaged results are depicted in table below.

According tests the performance of Virtual Threading Model is unfortunately weak enough but anyway this model has one big benefit - it gives unbounded count of threads and therefore doesn't lead to critical resources consumption and system instability.

Summary

In conclusion, can be say the above mentioned doesn't mean Platform Threading Model shouldn't be used at all. We tried to mark only some problematic places while using such model. Anyway note that threading model in WEB servers development offer significant advantages in terms of performance, responsiveness, and resource utilization. However, it also introduces complexity and potential issues related to synchronization, concurrency and resources usage.

Take into account also that Virtual Threading Model are a big change under the hood, and they are intentionally easy to apply to an existing codebase. Popular WEB Servers like Tomcat already allow for virtual threads (since 10.1.x). GlassFish server (since 7.0.13) is also ready to be supporting virtual threads. Jetty Web Server is also supporting Virtual Threading model since 10.0.x. Applications running on these servers will get scalability gains without any changes to the code, which could have enormous implications for large-scale applications.

So, developers need to carefully consider the trade-offs and use both threading model judiciously to achieve the desired results.