Don’t lose your mind over slow code check your performance sanity

Did you ever run into a problem that the IT application you were testing behaved like a couch potato? Getting a slow IT application off the couch might be a frustrating and challenging task, and it might require you to spend a lot of hours in stressful war rooms trying to find out what is causing the problem.

In the end, someone finally figures out that one of the reasons for the problem was a poorly designed method or class, creating all those issues. In this article, I will be sharing my thoughts and beliefs on how we, as performance engineers, can shift left and help improve the performance of the code in the earlier stages of the development life cycle. Please read along and share your thought in the comments!

Testing Environments

To be able to test the performance of an application reliably, you are going to need a reliable and stable test environment. In some organizations, it might be challenging to get your hands on one, and the upkeep costs of running such a test environment might also be pretty high.

Luckily with the move to DevOps and infrastructure as code, it is becoming so much easier to get a suitable test environment that resembles a production-like environment. Just being able to request a test environment on the fly, entirely or partly automatic is super helpful.

This level of automation gives teams the ability to manage environments without even breaking a sweat, cost-effectively. On top of that, with mature enough automation around test data management, teams could load these environments up with a complete data set to accurately simulate the production data.

I understand that for most companies, some of this still sounds like science fiction but achieving a reliable and stable test environment in a non-DevOps world is certainly possible. It might just be harder and more time consuming to get a suitable test environment for every application you are testing.

It is also imperative that teams gain insights into the resource utilization of their test environments, thus giving them the ability to validate if the resource utilization on the virtual machine and the bare-metal host are behaving within a certain degree of normalcy. If your organization has its hardware onsite, gaining these insights is just a matter of getting access to the right monitoring tool. But within a cloud environment, being able to see the complete picture might be much harder.

A reliable and stable testing environment does not mean that this has to be a direct copy of production. As long as it approximately resembles production-like statistics, you should be able to rely on the test environment.

Below is an example of two tests that I executed against the same code base. On the left, you can view the test I performed within a stable test environment and to the right, the results of the same test within an unstable test environment. Both the test environments arguably were production-like, but only the left test environment was able to output stable test results consistently that resembled the performance that we measured in production and was trustworthy enough to use for testing.

Achieving a trust-worthy test environment will open the door to do more effective end-to-end performance testing. But having the ability to create trustworthy test environments in a quick and automated way makes it also possible for us to shift left and help out developers to design and build what I consider to be performance sanity checks.

Don’t lose your sanity!

So what is a performance sanity check? Well, it is a test to quickly evaluate if the code on a unit level has the expected performance behaviour within a reliable testing environment.

It is then possible to answer two primary questions with these sanity checks - Does the performance overstep the set thresholds? How much regression is there between the old method and the newly written method?

If these performance sanity checks pass, then we know that the performance of the code meets our expectations. On the other hand, if they fail, then we know that the performance has deteriorated. Once the developer realizes that the sanity check has failed, it will prompt them to start an investigation to determine the root cause of the slow down.

Unit Testing

Almost every seasoned development team is already using unit tests to validate the functionality of a piece of code. So if it is possible to run functional unit tests to verify if the business logic is working correctly, then we could also put stress on the business logic and check the performance against a reliable test environment.

By introducing performance sanity checks within your unit test framework, you can raise awareness earlier around the performance impact of a code change. This way, a lot of potential performance problems will be found earlier in the development life cycle.

Early action will then mitigate the risk of finding easily preventable performance issues in the later stages of the development life cycle, whereby solving these issues can become an expensive endeavour.

By cutting away the low hanging fruit before it becomes a relevant problem with a performance sanity check, you are helping your team to free up resources to tackle more complicated functional and non-functional issues.

Therefore, the entire concept of incorporating performance sanity checks into your unit tests falls entirely within the philosophy of test-driven development and all the benefits that brings to the table.

Defining The Line Between Sanity and Insanity  

Within a performance sanity check, it is mandatory to define limits as they are going to determine whether the test is going to pass or not. That is why your team needs to identify appropriate thresholds based upon the non-functional requirements established by the business.

E.g. while testing the search functionality of an application, the non-functional requirement is set to a maximum duration of 3 seconds. If two units make up the search functionality, then each unit shouldn’t exceed an average duration of 1.5 seconds, totalling a maximum duration of 3 seconds. By logically dividing the maximum duration over all the units, we guarantee that no component exceeds the defined non-functional requirements.

The core idea behind setting these thresholds is to validate if the expected behaviour occurs. If we have a reliable production-like environment where we can confirm such a pattern is conforming to the non-functional requirements, then that’s desirable. But if this isn’t possible, then the thresholds must be adjusted to the expected performance within the non-production-like test environment.

Therefore, thresholds should be identified by the development team with the help of a performance engineer; when possible, that the configuration of thresholds is performed intelligently. With a sanity check, it is not only possible to validate if the non-functional requirements are met, but also to check how much regression occurs after a code change.

By showing developers if a code change impacts the performance adversely or not, it can provide helpful insights into the performance. This way, developers are motivated to write more efficient code and become performance-oriented.

Preventing a Catch-22

It is possible to get a catch-22 situation when implementing sanity checks. Hence, these checks will be rendered useless and can become an unnecessary overhead. Some reasons that might cause such a situation are testing the code in an unreliable test environment or basing your threshold on absurd non-functional requirements, ultimately creating a spiral towards a catch-22 scenario. Due to this, it no longer makes sense to validate the performance on the unit level or even on the system level.

Prevention comes down to having the ability to stress your code against a production-like test environment. If this is not possible, then your threshold should be tweaked to reflect the non-functional requirements in a reliable, non-production-like test environment. Achieving the latter is by far harder but not impossible. On top of all of that, it is super important that the business defines realistic non-functional requirements for us to use.

Implementing Performance Sanity Checks

To begin adding performance sanity checks to your unit test framework will differ per programming language, but all programming languages are going to require some sort of structure. That enables the encapsulation of a method by a profiler to capture as many performance metrics and system resource metrics as possible.

With metrics collected by the profiler, we are then able to make an automated decision if the executed method showed any signs of regression or if it did not match the predefined thresholds.

In the Gif above I am showing an example of this concept within Python where I have written a simple method and have attached a neat looking decorator to this method called ”profile” that when activated will collect as many performance statistics as possible.

It then allows us to write a unit test enabling the “profile” decorator, and test some performance aspects of the method that is under examination. Afterwards, we validate if we have seen the expected results and pass or fail the unit test, respectively. We are then able to generate insightful graphs with for example a BI tool to obtain a better understanding of the performance behaviour of the method. 

Because I found the idea of creating a framework around this topic fascinating, I decided to dedicate some of my spare time in developing an open-source Python Framework, that would put this entire concept into practice for the Python programming language. I have code-named my new little hobby project as QuickPotato, and I am planning to share the source code of it on GitHub as soon as it is mature enough.

Conclusion 

Knowing and understanding the pitfalls of performance sanity checks is critical to be able to experience any of the benefits they offer. Incorporating performance sanity checks within your unit test framework and knowing when to use them, is not hard to achieve but striking a balance by defining the right threshold in the correct test environment is the tricky part of implementing this solution.

For some companies, this is going to be easy because of their move to DevOps, but for others, it might be a difficult journey. I believe that within the right circumstances, every company can start adding performance sanity checks to their unit test framework.