Leveraging AI (Large Language Models) to write scripted automated tests

Recently, I was asked by a colleague how I use AI or Large Language Models (LLMs) to write Playwright tests.

Here’s a summary of my approach:

1) Generate Playwright Test Skeletons

• Ask multiple LLMs the same generic question to generate a basic Playwright test structure.

Some examples of Large Language Models:

OpenAI ChatGPT: https://chatgpt.com/        

Google Gemini: https://gemini.google.com/app        

Perplexity AI: https://www.perplexity.ai/        

Microsoft Bing Copilot: https://www.bing.com/chat        

• Example prompt:

Create 10 Playwright TypeScript tests for https://www.example.com.        

• Consolidate the ideas from different LLMs to create a comprehensive test suite.

The Large Language Model (LLM) chatbot provides a foundational test structure, which can be refined and expanded based on specific requirements. To ensure comprehensive coverage, I often run the same query through multiple LLMs and then consolidate the generated ideas into a robust test suite.

2) Request Specific Syntax for Custom Functions or Assertions

• Use LLMs to obtain the exact syntax for specific Playwright commands, eliminating the need to memorize everything.
• Example prompts:

i) What is an alternative Playwright TypeScript wait syntax if await page.waitForTimeout(3000); fails?        

ii) List typical web page element assertion syntaxes for Playwright TypeScript.        

iii) Write a custom function in Playwright TypeScript script to handle multiple overlays that appear on the Webpage in different Web page components? (all overlay objects share the same class name 'overlay').        

Test automation frameworks offer a wide range of commands, but memorizing all the possible syntaxes can be challenging. LLMs become invaluable in this context by providing exact syntax or crafting custom functions tailored to specific scenarios.

The specific example queries/prompts mentioned above allow me to bypass documentation searches and get directly to coding, reducing time spent on manual lookups.

3) Debug and Troubleshoot Failing Tests

• When encountering errors in a Playwright test, provide the LLM with the test script and error message to get debugging suggestions.
• Please remember to use delimiters such as triple quotes """, to separate different parts of the prompts, making it easier for the LLM to distinguish and address each section.
• Example prompt:

This Playwright test script encountered with error as follows. Please help to fix the error.

 (A) Playwright test script 
"""
        // Comparison using Regular Expression
        await expect(page.locator("[class*='countAndSortByArticles']", {timeout: 10000})).toContainText(/^[1-9] Articles$/);
"""

(B) Error:
"""
[chromium] ? loggedInHomePageSearch.spec.ts:238:9 ? Home page visual check using logged in user ? Search using valid article description with at least 3 characters should render auto-suggestion drop down options on "Home page for logged in user
Error: Timed out 5000ms waiting for expect(locator).toContainText(expected)
Locator: locator('[class*=\'countAndSortByArticles\']')
Expected pattern: /^[1-9] Articles$/
Received string:  " 5 Articles Sort by: Name (ascending)·················································································································
Name (descending)·····························································································
Name (ascending)·································································································
"""        

4. Convert Cypress Tests to Playwright

• Copy existing Cypress test scripts and use LLMs to convert them into Playwright TypeScript tests.
• Example prompt:

Convert this Cypress JavaScript test script to a Playwright Typescript test script.

"""
 describe('Webpage header visual check for anonymous user', () => {
  beforeEach(() => {
    cy.viewport(1920, 1080);
    cy.visit(Cypress.env('baseUrl'));

    const bannerClosed = "OptanonAlertBoxClosed";
    const bannerClosedValue = "2023-12-22T04:41:54.468Z";
    Cypress.on("window:before:load", (window) => {
      window.document.cookie = `${bannerClosed}=${bannerClosedValue}`;
    });
  });

  it('should display Landing page with default header text', () => {
    cy.contains('Landing Page').should('be.visible');
    cy.contains('Products').should('be.visible');
    cy.contains('Support').should('be.visible');
    cy.contains('Login').should('be.visible');
  });

  it('click on "Login" redirects to federation login URL', () => {
    cy.contains('Login').click({ force: true });
    cy.url().should('include', '/app/login');
  });

  // Additional tests follow...
});
"""        

Migrating tests from one framework to another is a common task in test automation. Using LLMs, I can quickly convert scripts written in one framework to scripts compatible with another framework.

#playwright #llm #ai #automatedtest #scriptedtest #autotestgeneration