How to Troubleshoot StackOverflowError on ServiceNow MID server

Ever encountered a StackOverflowError on your ServiceNow MID Server? ?? In this post, we’ll dive into the art of simulating and tackling this pesky issue head-on. Get ready to uncover the nuances of troubleshooting JVM stackOverflowError problems in your ServiceNow MID server environment, ensuring top-notch stability and performance for your applications. Let’s jump in!

What can cause StackOverflowError in ServiceNow?

First let’s discuss what can cause the StackOverflowError in the servers:?

1. Infinite Recursion: StackOverflowError commonly occurs when a method recursively calls itself without a proper termination condition, causing the call stack to overflow. In ServiceNow, this can happen if there’s a recursive function that doesn’t have a base case or termination condition.
2. Deeply Nested Method Calls: StackOverflowError can also occur due to deeply nested method calls, where the call stack grows too large to accommodate additional method invocations. This can happen if there’s a chain of method calls that recursively call each other or if there’s a deeply nested loop.
3. Insufficient Stack Size: The default stack size allocated to a thread might not be sufficient to handle the depth of method invocations required by the application. This can happen if the application has complex processing logic or if it deals with large data sets.
4. Excessive Memory Usage: If the application consumes too much memory, it can leave insufficient space for the call stack, leading to StackOverflowError. This can happen if there’s a memory leak or if the application allocates a large amount of memory for data structures or objects.
5. External Libraries or Frameworks: StackOverflowError can also be caused by issues in external libraries or frameworks used by the application. This can happen if there’s a bug or a limitation in the library or framework that leads to excessive method invocations or recursion.

Simulating StackOverflowError

Below Java program simulates StackOverflowError when it’s launched:

public class StackOverflow {
    
    private static void delay(long millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt(); 
            System.out.println("Thread was interrupted, failed to complete operation");
        }
    }

    
    private static void recursiveCall(int count) {
        delay(10); 
        System.out.println("Call number: " + count);
        recursiveCall(count + 1);
    }

    public static void main(String[] args) {
        try {
            recursiveCall(1);
        } catch (StackOverflowError error) {
            System.out.println("StackOverflowError: The stack limit has been reached.");
        }
    }
}
        

The ‘StackOverflow’ class in Java is designed to mimic the conditions that lead to a stack overflow situation. It includes a delay method that pauses execution, to simulate processing time within each recursive call. The core method of this program is: ‘recursiveCall()’, it calls itself endlessly, incrementing a count with each call to track the recursion depth. This recursive method lacks a base case, which is essential to stop the recursion before the thread’s stack memory is exhausted. As ‘recursiveCall()’ continues to call itself, it consumes more stack space, eventually triggering a ‘StackOverflowError’. In the ‘main()’ method, recursion is started and a catch block is in place to handle the resulting error. When the ‘StackOverflowError’ is caught, the program outputs a message indicating that the maximum stack limit has been reached.

StackOverflowError in ServiceNow MID server

Now let’s try to simulate this StackOverflowError in the ServiceNow MID Server environment. Let’s create a JAR (Java Archive) file from the above program by issuing below command:

jar cf StackOverflow.jar StackOverflow.class        

Once a JAR file is created, let’s upload and run this program in the ServiceNow MID Server as documented in the MID Server setup guide. This guide provides a detailed walkthrough on how to run a custom Java application in the ServiceNow MID Server infrastructure. It walks through the following steps:

1. Creating a ServiceNow application
2. Installing MID Server in AWS EC2 instance
3. Configuring MID Server
4. Installing Java application with in MID Server
5. Running Java application from MID server

We strongly encourage you to check out the guide if you are not sure on how to run custom Java applications in ServiceNow MID server infrastructure.

StackOverflowError Diagnostics in ServiceNow using yCrash

yCrash is a light-weight monitoring tool crafted to detect performance bottlenecks and deliver actionable recommendations within the ServiceNow environment. Infact, ServiceNow organizations itself internally uses yCrash to troubleshoot their performance challenges.

When the stack overflow situation arose on ServiceNow’s MID Server, yCrash actively monitored the micro-metrics of the ServiceNow environment. It promptly recognized the StackOverflow issue and presented detailed reports on the dashboard, aiding in swift resolution.

Below is the excerpt from the yCrash’s root cause analysis report:

The report highlighted a concern that one of the thread’s stack traces exceeded 400 lines, which is unusually long. Long stack traces in threads can be indicative of complex or deeply recursive operations and might lead to a StackOverflowError. Not only that, the yCrash tool also points out the exact code path which is causing the StackOverflowError. Refer to the screenshot below:

The above screenshot shows the details about the looping thread which is causing StackOverflowError. You can notice the name of the thread (i.e., ‘main’), its priority (i.e., ‘5’), its state (i.e., ‘TIMED_WAITING’), and other details reported. More importantly, it shows the stack trace of the thread. In the stack trace you can observe the ‘recursiveCall()’ method to be called repeatedly, which is the root cause of the problem.?

Equipped with information, developers can easily isolate the problem and instrument a fix in their code to terminate the infinitely looping thread. Once the necessary corrections are made, the thread’s execution will be normalized, and the risk of a StackOverflowError will be mitigated, ensuring the stability and performance of the application. To see the real yCrash report for this simulation, you can click here.

If you need to troubleshoot performance issues in your ServiceNow deployment using yCrash, feel free to sign up here to start using the free cloud-based tier. Alternatively, if your security requirements as a large enterprise prevent you from sending data to the cloud, you can register here to access the on-premises installation of yCrash.?

Conclusion

In wrapping up, mastering the art of handling StackOverflowError incidents on your ServiceNow MID Server is pivotal for ensuring uninterrupted service and optimal performance. By simulating and effectively troubleshooting these issues, you not only resolve immediate challenges but also enhance the resilience of your ServiceNow MID Server. Remember to employ the right tools, stay proactive in monitoring, and continuously refine your troubleshooting techniques to stay ahead of potential issues. With these insights and strategies at your disposal, you’re empowered to navigate and conquer any StackOverflowError that comes your way. Happy troubleshooting!