How to select a Performance Testing Tool
Performance Testing tools are becoming more fractured and competitive as LoadRunner continues to cede ground to more agile and cheaper alternatives. These are my thoughts on selecting a Performance Testing tool.
Be Tool Agnostic by Default
Everyone has personal preferences. For example, you may be familiar with a particular toolset or programming language or have an aversion to using a GUI for script design. That shouldn’t be the only reason to decide which tool is used, especially if you work in an organisation with a pre-existing investment in a tool and the skills to use it. The load testing tool may be just one element in an integrated stack of technologies.
It's important to remember that the tool is just that—a tool. It will not dictate the quality of your testing or outcomes as much as the individual or team using it. In summary, use what is there unless there is a clear benefit to switching or the in-place tool doesn’t have the required functionality.
Understand the Features of the System Under Test
Make sure you understand the basic details of the System you are going to test - particularly the protocols used to drive interactions and the scale of the load testing that will be required. For example, if you are testing a REST API over a HTTPS connection, all load testing tools will cater to you and the script creation process should be relatively straightforward. However, if we are using Web Sockets, MQ, testing Voice over IP, or any other “edge case” scenario, you may find that your options are more limited.
Regarding the scale of the load testing, if you are testing massively high throughout systems, you may find limitations in some cases in terms of how much you can easily ramp up. This generally is more related to the load generation element, which is a separate concern. Still, some cloud providers may have limits on this - for example Azure Load Testing only allows us 49 Load Generators to run in a single test instance with a recommendation of 250 users max per generator - so running more than 12k users would require a special request.
Also, consider how network access works. Some solutions may rely on cloud infrastructure to send traffic over public networks, which may not work if you are behind a firewall or VPN.
Cost
Cost is always a factor. Why would you pay good money if a free tool is good enough to do the job? There needs to be something tangibly better about the outcomes with a paid product. You could argue that time is also a cost, and if you do lots of load testing, having a paid version with on-hand support may save you money in the medium to long term if it makes testing much easier for your team.
Breakdown of Major Available Load Testing Tools
*This list is not exhaustive and based on my research and personal opinions.
LoadRunner
??Notes: This is a behemoth of the previous generation. It has not really improved or changed much in the last 15 years, but it is probably still the best out there. The price makes it hard to justify unless you have a technical need for its extensive protocol support or you are a huge organization.
?Plus Points: Extensive Protocol Support, Extremely Scalable, Generally easy to use with an option to use the GUI or several different programming languages.
??Downsides: It is expensive and fewer people have experience and skills with this tool.
??Price: $$$
??Ease of Use Scores:
?Recording: 9
?Scripting: 9
?Scenario Creation: 9
?Results Analysis: 8
Apache JMeter
??Notes: Open source, pure Java implemented Load Test Solution that enjoys extensive community support and “plug-in” development.
?Plus Points: It is extensively used and actively developed, Flexible, extendable, and Scalable. It also has various managed options, such as Azure Load Testing and Blazemeter, which can improve the toolset.
??Downsides: It is not great for results analysis unless paired with another tool such as Grafana. It requires a GUI for script and scenario generation, which is not brilliant. Protocol support out of the box is lacking.
??Price: $ (Free)
??Ease of Use Scores:
?Recording: 4
?Scripting: 8
?Scenario Creation: 7
?Results Analysis: 2 (all of these can be improved with plug-ins and/or service wrappers)
Locust
??Notes: Open Source Python based Load Testing tool. Pure code driven, popular with developers and integrates well into CI/CD.
?Plus Points: Pure code with no GUI required for scripting, is a Python module and can be used with other modules and the core Python easily, The scenario and results UI is decent, Now integrates with Azure Load Test.
??Downsides: Requires some Python skills. It is relatively new and the surrounding plug-ins/community behind JMeter currently.
??Price: $ (Free)
??Ease of Use Scores:
?Recording: 3 (relies on using a different tool for this)
?Scripting: 7
?Scenario Creation: 6
?Results Analysis: 5
K6
??Notes: A Grafana developed load testing tool. Script writing in JavaScript.
?Plus Points: Has a free local version and a cloud-based offering, In-built support for WebSockets, Integrates nicely with the Grafana stack, Can be integrated with other Dev Tools, Cloud version is easy to set up and use, Scripts are 100% transferrable between local and cloud set-ups.
领英推荐
??Downsides: JavaScript has some limitations (writing to file for example) - Extension is possible but requires you to rebuild binaries.
??Price: $ (Free for local set-up)
??Ease of Use Scores:
?Recording: 3 (relies on using a different tool for this)
?Scripting: 6
?Scenario Creation: 6
?Results Analysis: 4, 6 or 9 - for local, cloud and fully integrated into your Grafana stack respectively.
Neoload
??Notes: Tricentis Neoload is an enterprise solution that fills that gap between free open source offerings and the more expensive LoadRunner.
?Plus Points: Excellent Protocol support, Will include expert support, GUI based scripting for less confident coders, Good integration into CI/CD stacks.
??Downsides: Costs money, GUI feels a little claustrophobic, especially on small screens.
??Price: $$
??Ease of Use Scores:
?Recording: 6
?Scripting: 8
?Scenario Creation: 8
?Results Analysis: 7
LoadView
??Notes: Cloud-based load testing tool for websites and APIs. It uses real browsers to simulate user interactions and provides detailed performance insights.
?Plus Points: Real browser-based testing, Supports various protocols including HTTP/S and WebSockets, Flexible deployment options (cloud, on-premises, hybrid).
??Downsides: Can be expensive for extensive testing and only suits browser-based apps.
??Price: $$
??Ease of Use Scores:
?Recording: 7
?Scripting: 8
?Scenario Creation: 8
?Results Analysis: 8
WebLoad
??Notes: Enterprise-grade load testing tool by RadView, known for its robust scripting and extensive protocol support.
?Plus Points: Powerful correlation engine for script creation, Supports JavaScript and Java for custom logic.
??Downsides: Can be complex for beginners, Higher cost for enterprise features.
??Price: $$
??Ease of Use Scores:
?Recording: 8
?Scripting: 9
?Scenario Creation: 8
?Results Analysis: 8
Gatling
??Notes: Open-source load testing tool designed for DevOps and CI/CD, focusing on HTTP protocol.
?Plus Points: High performance and scalability, Excellent support for HTTP protocol, Integrates well with CI/CD pipelines.
??Downsides: Limited protocol support beyond HTTP, Requires coding skills (Scala or JavaScript).
??Price: $ (Free)
??Ease of Use Scores:
?Recording: 3 (requires coding)
?Scripting: 7
?Scenario Creation: 7
?Results Analysis: 6