How to solve OutOfMemoryError: Unable to create new native threads

There are 9 types of java.lang.OutOfMemoryError, each signaling a unique memory-related issue within Java applications. Among these, ‘java.lang.OutOfMemoryError: Unable to create new native threads’ stands out as an interesting error, as it demands a totally different diagnosis approach than other types of OutOfMemoryError. In this post, we’ll delve into the root causes behind this error, explore potential solutions, and discuss effective diagnostic methods to troubleshoot this problem. Let’s equip ourselves with the knowledge and tools to conquer this common adversary.

Here’s a video summary of the article:

JVM Memory Regions

To better understand OutOfMemoryError, we first need to understand different JVM Memory regions. Here is a video clip that gives a good introduction about different JVM memory regions. But in nutshell, JVM has following memory regions:

1. Young Generation: Newly created application objects are stored in this region.
2. Old Generation: Application objects that are living for longer duration are promoted from the Young Generation to the Old Generation. Basically this region holds long lived objects.
3. Metaspace: Class definitions, method definitions and other metadata that are required to execute your program are stored in the Metaspace region. This region was added in Java 8. Before that metadata definitions were stored in the PermGen. Since Java 8, PermGen was replaced by Metaspace.
4. Threads: Each application thread requires a thread stack. Space allocated for thread stacks, which contain method call information and local variables are stored in this region.
5. Code Cache: Memory areas where compiled native code (machine code) of methods is stored for efficient execution are stored in this region.
6. Direct Buffer: ByteBuffer objects are used by modern framework (i.e. Spring WebClient) for efficient I/O operations. They are stored in this region.
7. GC (Garbage Collection): Memory required for automatic garbage collection to work is stored in this region.?
8. JNI (Java Native Interface): Memory for interacting with native libraries and code written in other languages are stored in this region.
9. misc: There are areas specific to certain JVM implementations or configurations, such as the internal JVM structures or reserved memory spaces, they are classified as ‘misc’ regions.

What is ‘java.lang.OutOfMemoryError: Unable to create new native threads’?

When a large number of threads are created in the native memory region than the available RAM capacity, JVM will throw ‘java.lang.OutOfMemoryError: Unable to create new native threads’.

What causes ‘java.lang.OutOfMemoryError: Unable to create new native threads’?

‘java.lang.OutOfMemoryError: Unable to create new native threads’ is triggered by the JVM under following circumstances:

1. Thread Leak due to Buggy Code: Due to the bug in the code, application can inadvertently create a lot of new threads, it can lead to buildup of unused threads in memory, eventually exhausting the available native memory, resulting in OutOfMemoryError.
2. Lack of RAM capacity: When there is a lack of RAM capacity in the container/device in which the application is running.?
3. More processes in Memory: When other processes are running on the container/device, it leaves less room for the threads to be created in the native memory.?
4. Kernel Limit: By default, Kernel sets a limit on the number of threads each process can create. When the application creates more threads than the allowed kernel limit.

Solutions for ‘OutOfMemoryError: Unable to create new native threads’

Following are the potential solutions to fix this error:

1. Fix Thread Leak: Analyze the thread dump of your application and identify the leaking threads. Instrument fix to ensure that threads are properly terminated after it completed executing its tasks.
2. Increase RAM capacity: Try to run your application on a container/device which has larger RAM capacity.?
3. Reduce other processes: Terminate (or move) other?processes that are running on the container/device, so that there is more room for the java application to create new threads.
4. Reduce thread stack size: When you reduce the thread’s stack size (by using -Xss JVM argument), your application can create a number of threads within the same amount of memory. However, be cautious when you pursue this option, as reducing thread stack size can result in StackOverflowError.
5. Change Kernel setting per process thread limit: By default, Kernel sets a limit on the number of threads each process can create. If OutOfMemoryError is happening because of this limit, then you can increase this limit by using ‘limit -u’ command.

Sample Program that generates ‘OutOfMemoryError: Unable to create new native threads’

To better understand ‘java.lang.OutOfMemoryError: Unable to create new native threads’, let’s try to simulate it. Let’s leverage BuggyApp,? a simple open-source chaos engineering project. BuggyApp can generate various sorts of performance problems such as Memory Leak, Thread Leak, Deadlock, multiple BLOCKED threads, … Below is the program from the BuggyApp project that can simulate ‘java.lang.OutOfMemoryError: Unable to create new native threads’ when executed.

public class ThreadLeakDemo {

   public static void start() {
      
      while (true) {
         
         try {
         
            Thread.sleep(100);
         } catch (Exception e) {            
         }
         
            new ForeverThread().start();
      }
   }
}

public class ForeverThread extends Thread {

   @Override
   public void run() {
         
         try {
            // sleep for 2 hours
            Thread.sleep(120 * 60 * 1000);
         } catch (Exception e) {}
   }
}
        

The above program has a ‘ThreadLeakDemo#start()’ method. Within the ‘start()’ method, there is an infinite ‘while (true)’ loop that keeps launching ‘ForeverThread’ every 100 milliseconds. If you notice the code of the ‘ForeverThread’, it puts the thread to sleep for 2 hours. It means the application is going to be flooded with ‘ForeverThread’ which will terminate only after 2 hours. Once all the ‘ForeverThread’ size exceeds the native memory limit, JVM will throw ‘java.lang.OutOfMemoryError: Unable to create new native threads’.?

How to troubleshoot ‘OutOfMemoryError: Unable to create new native threads’?

It’s a two-step process:

1. Capture Thread Dump: You need to capture thread dump from the application, right before JVM throws OutOfMemoryError. In this post 9 options to capture the thread dump are discussed. You might choose the option that fits your needs. My favorite option is to use the ‘jstack’ tool which is present in the JDK. ‘jstack’ is an effective command line tool to capture thread dumps. ‘jstack’ tool is present in the JDK_HOME\bin folder. Here is the command that you need to issue to capture thread dump:

jstack -l  <pid> > <file-path>        

where, pid: is the Process Id of the application, whose thread dump should be captured file-path: is the file path where thread dump will be written to.

Example

jstack -l 37320 > /opt/tmp/threadDump.txt        

As per the example thread dump of the process Id 37320 would be generated in /opt/tmp/threadDump.txt file.

What is Thread Dump? Thread Dump is a snapshot of your application memory. It contains information about all the threads that are running in the application. It will carry the following information about each thread: Thread Name, Thread Id, Thread Priority, Thread State, code execution path (most important), locks it’s holding, lock for which it is waiting,… They provide valuable insights helping developers identify and resolve thread-related issues.

2. Analyze Thread Dump: Once thread dump is captured, you need to use tools like? fastThread to analyze the dumps.

How to analyze Thread Dump?

In this section let’s discuss how to analyze thread dump using the fastThread tool.

fastThread is available in two modes: 1. Cloud: You can upload the dump to the fastThread cloud and see the results. 2. On-Prem: You can register here and get the HeapHero installed on your local machine & then do the analysis.? Note: If you are working for a large enterprise, you should prefer on-prem installation of the tool instead of using the cloud edition, because thread dump can tend to contain sensitive information (IP addresses, 3rd party libraries/frameworks that your application uses, …) and don’t want the dump to be analyzed in external locations.

To the fastThread tool, upload your thread dump. Tool will analyze the dump and generate a report instantly. Here is the thread dump analysis report generated by the tool for the above program. Let’s review the excerpts from the report.

From the above screen shot you can see that fastThread reporting that problem has been detected in the application. It specifically points out that there are 1891 threads in the application. They all have identical stack traces, and they all are in the ‘sleep()’ method. It also provides a hyperlink to view those threads’ stack trace. When we click on that hyperlink, we get to see the following screenshot:

This part of the report clearly shows the stack trace of the ‘ForeverThread’ which is sleeping (i.e., they are in the ‘sleep()’ method) for a long time. Stack trace shows the code execution path of the thread. Equipped with this information, one would know the exact code path where the thread leak happened in the application, which will facilitate them to instrument a fix.

Conclusion

In this post, we’ve covered a range of topics, from understanding JVM memory regions to diagnosing and resolving ‘java.lang.OutOfMemoryError: Unable to create new native threads’. We hope you’ve found the information useful and insightful. But our conversation doesn’t end here. Your experiences and insights are invaluable to us and to your fellow readers. We encourage you to share your encounters with ‘java.lang.OutOfMemoryError: Unable to create new native threads’ in the comments below. Whether it’s a unique solution you’ve discovered, a best practice you swear by, or even just a personal anecdote, your contributions can enrich the learning experience for everyone.