Being a Silver Winner in the Visual AI Rockstar Hackathon - What test design I used?

On January 23rd, Applitools announced the winners of the Visual AI Rockstar Hackathon Winners. And yes, I am one of them, Hurraaaaaay! Thousands of top engineers in top tech companies from all around the world have participated and a 100 people have made it to the winners list. 

I love how Applitools’s Head of Growth Marketing Raja Rao has described the winners of the best and most exciting QA automation hackathon out there:

We think the people who participated and won the Hackathon are not just some of the top QA engineers but also trailblazers who are leading the way in pushing the QA industry forward. Congrats to all of them!

You can read the full blog post here: if you are interested  in understanding more about the hackathon idea, app, judging, and results.

Here is the summary of the results:

“Part of our test design included building on-page functionality that required thoughtful test engineering. Hackathon winners needed to cover relevant test cases as well as ensure page functionality. Generally speaking, people who scored the highest points wrote a lot of code and spent a lot of time in the traditional approach. Even with economic coding, the winners wrote many lines of code to validate a large number of on-page changes, such as sorting a table.

Unfortunately, we also found that many participants struggled writing proper tests in the traditional approach.  Some struggled with test design, some with the validation of a given page structure, and some struggled with other technical limitations of the code-based approach.

While participants either struggled or succeeded with traditional test code, pretty much every participant excelled at using Visual AI. Many of them succeeded on their first try! We found this to be very gratifying.”

What test design I adopted?

Truth to be told, it is painful to try to imitate a powerful AI agent like Applitools. Even if you are successful to achieve the best test coverage, you are most likely to fail covering an element existence in the code at least. Here is my test design which covered:

• Login Page UI Elements Test
• Data-Driven Test
• Table Sort Test
• Canvas Chart Test

For the Login Page UI Elements Test I tried to code a visual check which includes:

• Elements existence
• Position
• Size
• Value
• Weight (for text)
• Color (for text. For a logo or an image this is just impossible in the traditional way. If the element exists and has the same tag but different in color, there is no way to check that with locators)
• Font family

And more!

Here is the GitHub page for the traditional test code:

In this blog post, I show a statistical comparison  in terms of time, number of lines of code, and effort between the use of Applitools and the traditional method.

Beside the huge amount of time and effort one would lose using the traditional method, the coverage can’t and won’t be 100%. Also the test code most likely won’t be reusable as the test code depends heavily on the page code.

I would love to read about the experience of the top winners and learn about their test design and code. I believe that Applitools presents not only the future of QA but also the chance of building a community.