How to automate the testing of the canvas elements using Cypress or Playwright

Hi QA fellows!

Intro

During the end-to-end automated UI testing sometimes we all face a situation when we can not locate or interact with the elements on the page using the, as they are rendered on a canvas element.

Canvas elements, often used for rendering dynamic graphics, charts, games, or animations, present unique challenges in test automation. Unlike DOM elements with accessible attributes and text content, canvas elements are bitmaps, making their content invisible to most automated testing tools.

It can be quite a non-trivial task to test and assert these elements - like maps, graphs, or dashboards made with pure JS on a canvas, so we do not have any locators in the DOM.

Indeed there are some approaches we can use to accomplish such a task.

Before diving into testing, we need to identify the purpose of the canvas element. If we are testing:

• The rendering of specific visual content (e.g., charts, maps, or shapes)?
• The interaction logic (e.g., drag-and-drop)?
• The dynamic updates (e.g., animations)?

Based on this we will build our strategy.

Possible solutions

1. Image comparison

The Playwright framework supports visual comparison out of the box. Cypress will need to use the cypress-visual-regression plugin.

It is as simple as you can see on these code snippets

This approach is good for simply comparing the webpage's visual appearance with the previously saved image template. But using this approach we lose the actual interaction with the canvas. It could be clicking on the points on the map, or toggling the dashboard options. We just can not test the functionality with only the visual regression testing approach.

2. Interaction with the canvas using the coordinates on the canvas element

This approach can be more challenging and will require us to write more code, to help us with it.

It is worth noting that first, we will need to wait for the canvas element to load and be rendered completely on the page.

Both frameworks have robust waiting mechanisms to make tests stable.

Let's see what we can do...

First, we have to create some helpers to leverage the abstraction of the repeatable steps from the tests.

I will focus first on the Playwright implementation followed by the Cypress mirroring.

Just for the sake of convenience, I created a helper function to visualize our clicks on the page.

This function takes the element and the coordinates as arguments and creates a small red dot on the page located on these coordinates, so we can see where exactly we will click.

as you can see from the code snippet we wait for the page loadState to become 'networkidle' on line 10. This will ensure that we do not have any not-finished requests/responses.

On the map it looks like this:

Having this we can encounter the required coordinates of the point to interact with a couple of shots.

This function is actually performing a click on the provided coordinates:

After that, we can assert that our click produced the expected outcome

The whole Playwright test snippet looks like this:

With Cypress the solution looks pretty similar. We can create a custom command and use it within our test:

And the test snippet:

Again - we wait for the DOM content to be loaded and only after that we interact with the canvas.

There is also a good article about testing the canvas elements, where you can find the Cypress and Selenium examples.

To be honest, there are situations where even clicking by coordinates we can not assert with traditional methods the change of state of the application, because all is happening within the canvas.

So the best approach could be combining two previously described methods as they will complement each other. We can click on the coordinates and assert the result of the action looks exactly like the previously taken template image.

I believe combining these two approaches will help you with testing these tricky elements and will make your life easier!

Conclusion

Testing canvas elements requires a blend of creative strategies and robust tooling. Playwright’s flexibility, combined with its advanced APIs, makes it well-suited for tackling these challenges. By employing techniques like image comparisons, and user interaction testing, you can ensure the reliability and quality of your canvas-based applications.

I appreciate any ideas on the topics and would like your feedback. Feel free to leave comments with your thoughts!