Using Artificial Intelligence in Test Automation

I was in Barcelona last week at Gartner Symposium, speaking with business leaders from all over Europe about how they are constantly pressured to deliver better customer experiences in order to grow, or even just survive.

Artificial intelligence is high on everybody’s agenda, because it promises an infinitely scalable, low cost and super-precise digital workforce. And while a lot of what’s being said is science fiction, there are ways we can leverage AI today that can have a tremendous impact.

In a world where technology disruptions happen almost on a quarterly basis, everyone from German car manufacturers to pan-European insurance companies and Swedish paper mills have long since declared that they are at the core, IT-driven enterprises. They embrace their own digital transformation and the results are clearly visible downstream; the recently published World Quality Report 2018 shows that we have reached a tipping point, with 99% of all 1.700 interviewed CIOs saying that they use DevOps.

This is all great news, of course, but it creates a massive pressure on quality assurance efforts, which currently accounts for more than 25% of all IT-spending. Especially test automation is under fire, since the true value of DevOps can really only be attained with automation.

A strong demand for better customer experiences is a main driver behind digital transformation, which requires agile and DevOps methodologies, which in turn creates a tremendous pressure on test automation.

Test automation is an 8 billion dollar a year market, but it’s been mainly powered by technology and vendor thinking from the last century. Automation products are stale, outdated and horribly complicated, requiring the use of programming languages such as VBScript and Java. And the people who are tasked with automating tests are not programmers. This gap between users and technology has become a major roadblock to digital transformation and threatens to derail the whole process.

I was on stage Tuesday talking to a room-full of about 150 tech executives about how we solve hard problems like that in test automation and robotic process automation (RPA). We do it by delivering a uniquely visual and intelligent platform where business users can automate any process on a computer screen without knowing anything about programming or indeed the technologies they are automating.

Selfie! I swear the guy with his back turned was just finding a seat and wasn’t leaving, and besides this was before I even got to the stage…

As a side note, if you have never seen LEAPWORK before, I’ve included a brief example at the bottom of this page and of course, there is much more on our website.

Understanding the AI Hype Cycle

By far the most frequently asked question I’ve had this week is: How can we use AI? To answer that, let me first take a step back and share what Gartner has to say in the Hype Cycle for Artificial Intelligence 2018 report, something I whole-heartedly agree with:

“AI is overhyped as a socioeconomic phenomenon. The media, governments, corporations and individuals each have an opinion about AI, mostly based on vague ideas of what it really is. This Hype Cycle views AI as a pervasive paradigm and an umbrella term for many innovations at the different stages of value creation. The traffic jam at the Peak of Inflated Expectations is increasing, as early implementers grow in numbers, but production implementations remain scarce.”

Even if you have never heard of the Hype Cycle model before, you probably won’t be too surprised to learn that AI is considered to be right at the “Peak of Inflated Expectations”. Sounds about right, doesn’t it?

Here is the model in all its glory:

Separating Fact from Fiction

I had the absolute pleasure of visiting the MareNostrum supercomputer this past week in Barcelona. It’s one of the 30 fastest supercomputers in the world, but perhaps more importantly, it’s partly built using highly experimental, next-generation technology. This incredible thing, about the size of a half basketball court, is placed in a glass box inside a beautiful old deconsecrated church, and when you walk through the heavy wooden doors to the inner sanctum, somber choir chanting plays through hidden speakers. The site manager there never gets tired of the look on his guests’ faces.

For his 2017 bestselling thriller “Origin”, author Dan Brown chose this place as the home of Winston, an incredibly powerful AI that plays a central role in the plot. I can see why -- this is without a doubt what real AI looks like.

But the truth is, there is no such thing as an independently thinking, general artificial intelligence computer program. It doesn’t exist. It hasn’t been invented and may never be. We humans have simply not come up with a single good idea on how to possibly build it. A general artificial intelligence such as Winston is entirely science fiction.

The great news, however, is that we humans have in fact invented some very clever statistical data analysis methods, and we call those artificial intelligence algorithms. Many of them are machine learning (ML) algorithms that can build clustering and predictive models out of even small amounts of data. Other algorithms mimic human decision-making processes, such as the way humans interact with software. For instance, image recognition algorithms mimic human visual cognitive processes by recognizing buttons and other things on a screen. Algorithms like these can be incredibly powerful when applied in the right ways.

All these technologies don’t need a supercomputer to run; most of them can be hosted on a cloud server or even on your own laptop.

I’ll quote Gartner once more, this being Symposium week and all:

"The key to AI success is narrow AI -- narrow use cases with clearly defined benefits."

But what does that mean, exactly? Think of it like this: There is no general artificial intelligence, so we can’t teach an AI how to perform test automation by itself, but we can use AI technologies to take most of the repetitive, boring and error-prone work away from us humans and bring us data insights that would otherwise be impossible to get to.

Like finding and clicking on the right buttons on the screen, or automatically flagging anomalies in reporting data.

Practical Applications of Narrow AI in Test Automation

At LEAPWORK, we invest heavily in using artificial intelligence to drive productivity and robustness when designing, maintaining and running test automation and RPA flows. Here are some real-life examples of how we do that today and where we’re going tomorrow:

• Intelligent capturing of screen elements. We use sophisticated, pre-trained intelligent algorithms to create “strategies” when designing software robots, for finding things like buttons and fields in desktop, web and SAP applications. In the very near future, these algorithms will combine even more analysis methods, including ML-based visual recognition, in what we call “smart recorders”.
• Self-healing flows. When automation flows run in LEAPWORK, the above-mentioned intelligent strategies are employed in smart ways to for instance appropriately wait for screen or page changes, so that time is not wasted needlessly during execution. We’ve already begun expanding this to include unique, cross-strategy learning and autonomous error handling, to make things even more robust to changes.
• Intelligent text recognition. We use trained neural networks both inside LEAPWORK but also with partners such as ABBYY to recognize text and numbers from screen pixels. This works incredibly well in virtualized environments such as Citrix, but also in graphical-heavy applications such as tv broadcasting software, 3D movie producing packages, computer games and much more.
• Natural language processing and sentiment analysis. LEAPWORK integrates directly with leading AI cloud services such as IBM Watson, Google TensorFlow and Microsoft Azure ML to bring textual processing into the automation mix. This can for instance be used to perform sentiment analysis of screen messages and from that automatically choose responses to given scenarios.
• Cross-technology locators. One of our big investments is in cutting-edge intelligent, cross-technology strategies for finding screen elements. So for instance, using visual recognition to find a certain button and then automatically map that to how that button was rendered on-screen in a web browser, desktop application or SAP GUI.
• Anomaly detection. Another of our big investments is in automatic anomaly detection both in reporting data but also in the individual building blocks inside LEAPWORK, so that dynamic and static parts of applications are automatically recognized and reconciled with the self-healing algorithms.

There are many more examples, including a lot we’re not quite ready to talk about yet. The thing is, individually these examples are very narrowly focused and maybe don’t seem all that magical, but together they add up to something incredible because users are now able to wield the power of artificial intelligence without technical training.

More than just productivity increases, this puts the ownership of process automation in the right hands, regardless of whether it’s in IT-based test automation or cross-departmental RPA.

The bottom line is: If we stop dreaming about general artificial intelligence that could autonomously test applications or perform robotic process automation, and instead focus on applying narrow AI today, we’re on our way to change the world.

Final Thoughts

To round off this post, I promised a brief description of our platform. Well, here we are. The LEAPWORK Automation Platform gives business users the ability to visually design and maintain software robots without having to write a single line of code.

It works just like drawing on a whiteboard:

Even if you’ve never seen LEAPWORK before, you can almost certainly see what it does at a glance, and we can get you up and running with our fast-adoption program, “Seeing is Believing” in less than a week.

Our customers are globally recognized brands such as Mercedes-Benz, Arcelor-Mittal, BNP Paribas, BAE Systems, Electronic Arts, Sheraton, and many more.

Get in touch and I’ll connect you with the right person!