What is the best way to determine the optimal number of threads for a given application?

LinkedIn seems to believe me to be an expert on this topic -- and in fact I do teach a C++ course at Cornell where it is a main focus for much of the semester. Unfortunately, it isn't simple. To the horror of most of my students, multi-threaded programs are often much slower than single-threaded ones, to the point that a rule of thumb could be "don't do it! special training required!" As others have noted, the issues center on performance oddities caused by NUMA memory, lock contention and unfairness, and excessive overheads due to copying to create private copies of data structures. Modern NUMA servers are really optimized for container VMs. Obviously we do use threads (any GUI does so), but for coarse-grained concurrency: not speed!

Hardware is often the biggest criterion to consider because the number of cores determines the number of threads you can use in a physical setup.

The hardware limitations don’t stop at the processor. You need to use the fastest CPU’s and largest and fastest storage also. The only place on the planet to do that is with IBM zSystem and IBM DS8000 storage.

Hardware choices are a fixed part of design and considering the scalable cloud offerings available nowadays, this should not matter too strongly in our considerations for determining the optimal number of threads for a given application. We should choose hardware after considering the app profile and not the other way around. But choice of correct hardware should be carefully done based on the nature of the application such as rote image processing, crawling robots, gaming apps, e-commerce apps etc.,

Determining the optimal number of threads for a given application depends on several factors, including the nature of the application, the hardware it runs on, and the workload characteristics. 1. Experimental Benchmarking. 2. Workload Analysis. 3. Hardware Considerations. 4. Empirical Testing and Tuning. It's important to note that there is no one-size-fits-all solution for determining the optimal number of threads, and the optimal configuration may vary depending on specific application requirements, hardware characteristics, and workload patterns. Therefore, a combination of the above approaches, along with iterative testing and tuning, is often necessary to find the best thread configuration for a given application.

Workload characteristics are a significant aspect in defining the optimal number of threads. Based on the NFRs such as the common site performance attributes and expected traffic etc., the application architect should be able to judge the parallelism needed without compromising on maintainability, scalability and ease of deployment. Considering the throughput characteristics expected of the application and then considering the target hardware, an optimal number of thread should be designed within the cost/quality boundaries.

By using designs like caching (REDIS), micro-service architecture principles (e.g., shared nothing), nonblocking architecture (NodeJS), queuing etc., we can avoid the problems posed by synchronization requirements. An architect can look at any multi-threaded/scalable/high-throughput system designs with an easy interface to solve synchronization problems. Hence this may not be an ideal concept to decide the number of application thread counts. But by adopting the right solutions, as above, we can optimize the requirements around the parallelism expected.

Once the architectural designs had passed the sanity test post necessary reviews and approval, in the next step of the development, by practicing concepts like SOLID/12-Factor Principles etc., we should provide enough due diligence while coding. (remember the shift-left concept). We have some code inspection tools too that can integrate into the CI/CD pipelines. Hence, if empirical testing is throwing up performance issues, we should first consider hardware related horizontal and vertical scaling before attempting to break the code and introduce parallelism. Having said that, if good system design and coding practices had been followed, it should be easy to rewrite the code to introduce parallelism. Empirical testing, hence, is important.

Some other things to consider for improving performance are the use of CDN (edge networks), intelligent load-balancers, containerization, in-memory data stores, private data lakes , design considerations around CAP theorem and acceptance of eventually consistent approach with sub- second latency ( use of replication technology), hot updates of cache etc., in an appropriate way.