Compare visual regression testing tools: Diffy vs SauceLabs

SauceLabs is a very established company in the browser testing industry. After acquiring Screener, Saucelabs started offering visual regression testing as part of its product portfolio.

The main technical difference between SauceLabs and Diffy is that SauceLabs requires a testing framework, where you insert “capture screenshot” statements to build a set of screenshots for comparison.

We built a Diffy-like testing scenario in Cyress that visits multiple pages and takes screenshots to compare both tools.

Browsers

SauceLabs is designed to support multiple browsers / OS, while Diffy uses only the latest Chrome.

?

Screenshots quality

We took screenshots from a website, https://www.adaptive.co.uk/, with a floating header and saw that the header is duplicated multiple times.?

So SauceLabs does take screenshots from the entire page, but it works like taking screenshots with a set viewport and then stitching them together. In contrast, Diffy opens a page in a tall browser window to grab the whole page content.

?

Tools to modify screenshots

If you use SauceLabs, you need to implement your stabilization scripts. Here's an example we did for hiding the cookies popup. In the same way, you would need to add a delay, scroll the page, and perform other actions as required.

// Modify the page before taking screenshots.
cy.get('body').then(($body) => {
   if ($body.find('#CybotCookiebotDialogBodyButtonDecline').length) {
       cy.get('#CybotCookiebotDialogBodyButtonDecline').click();
   }
});        

On Diffy's side, we allow you to apply these modifications in the UI, but you can also inject custom JavaScript.

Ignore elements

SauceLabs has an interesting feature that ignores certain regions on the page. While Diffy has a way to mask elements (draw a green rectangle on top of the element so its content is invisible for the screenshots), in SauceLabs, you can specify the coordinates of the area that we should skip when we do a comparison. See the documentation page. ?

Baseline

SauceLabs allows you to compare any set of screenshots with the Baselines. There are some rules about how baseline screenshots are selected during comparison. There are overrides and different browsers / OS combinations. Check documentation for details.?

At Diffy, you can compare any set of screenshots with any other. You can take them from different environments. There is also a Baseline set, but it is more like a “special” set of screenshots rather than the only one you can compare against.

?

Review process

Diffy allows you to approve/mark individual changes as bugs, and in SauceLab, you approve/reject the whole screenshot. SauceLabs categorizes the changes by visual, style, dimension, and others. It also analyzes the HTML of the pages.

Conclusion

SauceLabs is very framework-dependent, so you must implement all the manipulations on the web pages yourself to implement things like delay, masking, mocking content, and other things that Diffy offers out of the box.?