What I will miss about Visual Studio Load Test

With Microsoft's announcement that they will be discontinuing Visual Studio's load test features after Visual Studio 2019, I wanted to take a moment to acknowledge what I am going to miss about the tool-set.

I suspect many of you never used it, which is a shame because it had some of the best features out of any tool-set on the market. Some of the other (still popular) load test tool vendors could learn a thing or two from what Microsoft achieved in the little tool sitting on the back of a giant IDE.

Error logging during load tests

When an error occurs during a load test it's important to identify what happened. Most of the time it turns out to be a test data or environment issue, but now and again an important issue occurs which has the potential to impact the business.

When an error occurs during a multi-user load test it can be difficult to pinpoint what caused the issue. I struggle with this in JMeter. It's easy to log a specific request which failed, but more often than not the failure occurred during a prior request.

Visual Studio can be configured so that when a failure occurs every request and response is logged for the full end to end transaction. It also logs the value of user defined and run-time variables at each step. It presents this in the same UI you see when you do a single script playback, which makes identifying exactly what the error was and when it occurred easy.

I haven't seen another tool do this quite so well. It provided me with a sense of confidence that I understood exactly what my test is doing and the reason behind every failure.

Storing results in a standard DBMS

Most load testing tools store load test results in either:

• A flat file (e.g. CSV or XML)
• A bespoke database format which can only be viewed in the tool itself

The latter prevents you from connecting with and analysing the results with a third party tool, and the former often requires you to do some kind of parsing or at least data migration in order to compare multiple test runs against each other. Not to mention that working with very large flat files is a nightmare.

Visual Studio stores load test results directly in SQL Server. The significance of this is that you can connect to it however you want to pull the raw data in real-time. I connect Tableau Desktop to my results database to easily visualise and analyse multiple test runs, even while a test is still running. The table structure is a little daunting at first, but at least I have the option to access it using an external tool.

I'd like to see more load testing tools adopt this approach. There's no good reason for hiding the raw results in a bespoke database. As a load testing tool vendor it's important to acknowledge there are other tools out there specifically designed for data analysis and visualisation that can do the job better than the ones built into our load testing tools. Let us use those by avoiding bespoke database or results file formats.

Ethical licensing model

Relative to the quality of the tool Visual Studio was cost effective. In previous years you paid a one off fee for the Visual Studio Ultimate license. Now you pay a yearly subscription for the Enterprise license. Either way - you pay one fee per user which gives you unlimited test runs with unlimited virtual users.

To me, that's fair. They provided a tool which you paid for, and once you had it then all of its features and power were at your fingertips without any additional cost.

The licensing model of "pay more money the more virtual users you want to simulate" is outdated. Vendors are essentially providing stunted software which requires paying large fees in order to unlock the potential already packed up but locked away. Looking back, maybe this was an early incarnation of the "in app purchase"? It doesn't make sense to me, whether the tool simulates 10 users or 1,000,000 the tool vendor is not providing me anything additional. Unless a vendor is providing the infrastructure to run large tests (which makes sense to charge by the hardware used) then I don't think it makes sense to charge more for the more virtual users you want to simulate.

Extensibility

Because Visual Studio is, well, an IDE rather than a load test tool, it comes with the full power of .NET. Other load test tools also provide the ability to extend them programmatically, but with Visual Studio it seems like everything is made available.

In a past life I worked with a team of extremely talented performance experts who built a framework to make performance testing assets more re-usable and to make very complex scripting work simpler. They did a proof of concept looking at extending a number of load testing tool-sets and settled on Visual Studio for a reason; because of the level of access Microsoft provided to modify the way tests were executed and orchestrated.

Summary

It is a shame that Microsoft are pulling the tool-set from the market. Visual Studio was my go-to tool when open source was not an option. But more than that, Visual Studio did some things really well that the other tool vendors could learn from.

In the interest of continually improving our industry, I hope that other tools out there adopt some of the best bits of Visual Studio so that we do not lose these things in the past.