What to do when your A.I. is just not very smart?

Artificial Intelligence is all the rage these days, in particular, ChatGPT.?But what happens when you trust your A.I. responses and you wind up with more "A" and less "I" in the responses??Looking at how these A.I. engines are trained, what if you fed it garbage data from across the internet as truth and expected it to generate accurate responses?

To test against this hypothesis, I asked ChatGPT a question about a topic that I know quite a bit about, Performance testing and, in particular, OpenText LoadRunner.??I have been answering questions about the use of this tool online since 2001, amassing almost 12000 answers to questions in that time.??I have also been using the tool continuously since version 4.51 in April 1996 -?just shy of 27 years.??I also picked LoadRunner because of the length that it has been available in the market.?This time on the market would produce the largest number of public questions and data to train ChatGPT responses.

I asked, "Why is Microfocus LoadRunner a bad tool?"?I disagree with the premise of the question, but I thought it worthwhile to examine how ChatGPT has been trained and whether a mass of poor information will overwhelm accurate information.

The Responses

How did ChatGPT respond???I include each of its points in response and a counterpoint based on my years of experience using the tool.

High Cost: LoadRunner is a commercial tool, and its licensing costs can be relatively expensive compared to some open-source or lower-cost commercial performance testing tools.

All commercial tools are expensive to purchase versus open-source ones.?ChatGPT fails to note that this objection is the first objection of people who have no sign-off authority for any tool.??It is also a foundation religious argument in the software universe, going back as far as Microsoft's first commercial tool where Bill and Paul would not share their source code for BASIC for the Mits Altair:

https://en.wikipedia.org/wiki/An_Open_Letter_to_Hobbyists

An oft-ignored item is that LoadRunner was for many years sold at the list price, without a discount, reflecting the actual value delivered by users of the software.?There was even a time when Mercury offered a full refund if the customer did not improve the performance of the tested software by a significant percentage.??This reflects how much value has been delivered using LoadRunner over the years, from version two until the current release.

Complexity: LoadRunner can be complex to set up and use, and may require specialized skills and expertise to configure and manage effectively.

There are a couple of pieces here to pull apart.?First, to install the product, you run "SETUP."??No shell scripts.?No package managers (unless you want to install a LINUX load generator).?It is a Windows app consisting of a few programs, VUGEN for the development of virtual user assets, A CONTROLLER to run a test and to communicate with the load generators, and ANALYSIS to bring the test results and monitor data together in the end for analysis. Installation of MS Office requires more choices.

The skills required to install, configure and use the tools are no different than the skill required for any other tool in this category.?You need to understand networking.??You need to have a command of application architecture.?The open-source tools require an even higher level of skills to use and configure as a tool to achieve the same level of utility as LoadRunner out of the box.?Open source multi-user performance tools are unitaskers - they throw load.??Monitoring and analyzing results requires using additional tools to integrate monitor data with the response time data from the open-source load throwers.?Often an additional tool is also necessary for reporting.

Refs:

https://admhelp.microfocus.com/lr/en/2022-2022-r2/help/PDFs/LoadRunner\_Installation\_Guide.pdf

Limited scripting options: Some users have reported that LoadRunner's scripting options can be limited, making it difficult to customize scripts to fit their specific testing needs.

All of LoadRunner's ``Scripting Options'' are general-purpose programming languages.??The default for the product is 'C,'?hardly an exotic scripting language.?In addition to C, you can build test code in C++, C#, Java, JavaScript, and Visual Basic.?Not all languages are available for all interface types - as an Example, Winsock is a C-only virtual user item.?Plug-ins for Visual Studio and Eclipse allow developers to work in their natural environments (if you use developers to produce test assets).?Junit and NUnit tests are supported.

Programming skills are a foundation skill class item for all performance testing tools.?The only popular programming language not supported for building test code is Python.?Python is horribly resource inefficient when compared to the options already available

Refs:

Resource-intensive: LoadRunner can be resource-intensive, which means that it may require significant computing power and memory to run, making it difficult to use on lower-end machines.

LoadRunner's default language for all virtual users is 'C.'?This results in the most resource-efficient virtual user.?No virtual machine is required for a guest operating system (Java, .Net).?No interpreter is required to be loaded (JavaScript).?It's difficult to understand how it could become more efficient in the production of load on systems. I have watched 5000 users for Winsock be hosted in a single 8GB windows host with resources to spare, and fewer than a half a dozen truclient users on the same hardware. Winsock is C, running without a GUI. Truclient runs a full Chrome Browser per virtual user.

Training is critical for any performance testing tool.?An untrained, unmentored person with a performance testing tool available is as dangerous as a two-year-old with a machine gun. Performance Testing Tools are a weapons-grade class of software designed to "take out" systems under test to allow Performance Engineers and Architects to understand how and where systems fail under load.?Having audited many users over the years, a piece of test code built by a new user without any training or mentoring can easily have one or two orders of magnitude more resource requirements than a comparable script from a skilled user of the tool.?Yes, that is potentially a 100MB piece of code versus a highly efficient 1 M.B. version that has the same conversation with a server.?This need for education and mentoring of new users is also true of other commercial and open-source solutions for software performance testing.

Lack of flexibility: LoadRunner is a Windows-based tool, which can limit its flexibility when it comes to running tests on different platforms or integrating with other tools.

Load Generators, the component that generates load, are available for Windows & LINUX.??Docker images are available.?Cloud images are available for Windows and LINUX.??There are APIs that LoadRunner supports that are only available on Windows.??Can the counterargument be made that any tool that does not run on windows is insufficient in its ability to generate load for these APIs???That would be an equally absurd argument. Through version 6 of the tool, both UNIX-Native and Windows-Native controllers were available.?People stopped purchasing the UNIX native controllers as a more expensive option.?All features and functionality were integrated into the Windows controller.

And what of DevWeb??It runs on Windows, Mac, & LINUX for the production of test assets.?Hardly Windows-Only.?

This "integration with other tools'" commentary is interesting.??This common argument often comes up in discussions where open-source tools are currently in use.?Open-source multi-user performance testing tools are unitaskers, requiring integration with third-party tools to match the out-of-the-box capabilities of LoadRunner.?This is true not only for LoadRunner but also for every other commercial multi-user performance testing tool that monitors resources and analyzes results within the toolset.?To address another common argument, all current market tools for multi-user performance testing, both open-source and commercial, integrate with DevOps pipelines.

One potential issue with LoadRunner is its complexity and the steep learning curve it may present to new users.?LoadRunner is a comprehensive tool that offers a wide range of features and capabilities, but this can also make it challenging to use and configure effectively.

Management groups across information technology would like to think that social promotion will result in an effective performance tester, engineer, or architect.?I witnessed an onsite engagement manager for a classical big 4 consultating firm tell an automated tester to go home and watch LoadRunner videos, for tomorrow they would be leading performance efforts. This assumption on non-domain-specific and highly tranferable skills is far from the truth.?

No matter which tool is elected in this space, open-source or commercial, there is a foundation set of skills on the part of the user that are high indicators of success.??In addition to the foundation skills, a period of training and mentoring is required to achieve the highest possible value from the return on the use of the tools.?Fewer than ten percent of Performance Testers in the market today have been through a training or mentoring program.??This unwillingness to train personnel in a new field would not be considered acceptable behavior for other operational requirements areas, such as security, but it is common for software performance.??If your organization constantly questions the value of your performance testing team, this paragraph may provide insight into the source of your issues.??

For greater insight on key foundation skills for performance personnel, consider the following book on hiring Performance Professionals,

Conclusions

It is clear from the responses that Software Vendors need to find a way to provide accurate education models to ChatGPT and its peers.?Without this education, there will be a lot of poor decisions based upon the information presented by these artificial intelligence aggregators as a source of truth.?These systems are more a measure of the mass of training posts related to a topic.?OpenText, Tricentis, Akamai, Apache Foundation, Blazemeter, ..., you have your work cut out to ensure that your tool capabilities are reflected accurately by ChatGPT and its peers.

While I selected LoadRunner for this line of inquiry, the responses are equally problematic for other tools. I chose LoadRunner because of the availability of the most extensive training set of questions and answers on a particular tool. Be wary!

Artificial Intelligence-driven natural language responses may eventually reach the level of reasoned analysis in selecting performance testing tools, it is not here today.?If you are going to decide on tools and usage of tools in this area, you should call upon a human expert.?Per the note above on learning curves, ensure your expert has been through a training and mentoring program, or their analysis will likely be just as flawed as that of ChatGPT.