What are the main benefits of using automated tools for WCAG testing and reporting?

Automated tools are useful for devs to learn what areas they need to pay attention to so that they can build up their knowledge and start on the right foot next time. Typical examples are the "alt" attribute for images, the "for" attribute for labels/inputs and colour contrast. This way, devs can discover easy fixes that can be replicated and often can be built into helpers so that all controls of the same type will be correct. But automated accessibility tools are in no way a reliable quality gate for accessibility. For example: how do you discover keyboard traps? Does the tab order make sense? Is the copy understandable? Is the semantic tag used (eg a tab, a list) appropriate for that type of content? A human tester is irreplaceable.

When we are focused on designing for cognition, this is one example where automated testing is very valuable and giving the developers an opportunity to focus on the higher impact areas to begin with and then they can move on to more manual testing. Using automation in many ways is improving the accessibility, and usability of the overall process.

How many accessibility testers does it take to test an entire site? 10? 100? If you have a web site with more than 20 pages, there really is no other way to use automation for quantitative tests. Manual testing should still be used for qualitative tests, but automation can cover thousands of pages easily in a coffee break. The idea though, should be to find non-compliance situations and uncover problems on pages that no one remembers existed. The same can be said about documents linked to websites. Another significant benefit of automation is to add it in development stages to prevent accessibility problems to go into production in the first place.

Consistent and reliable results for a limited scenarios; for others, there is a need for additional manual testing to find and uncover other accessibility issues!

I consider automated accessibility testing tools like "smoke tests". Yes, useful. No, not sufficient. The idea that accessibility is met if the tool reports a green means that the tester just wants to finish quickly (tick box exercise) but is is not really empathising with a user with different needs, different technologies or different devices. The trouble is that even one failure can make the site unusable for certain people (eg a keyboard trap), so it's important to test in the same conditions as users of assistive technologies.

While automated tools are a good start for accessibility evaluation (helps save time, cost and provide a sense of direction), they only cover a proportion of WCAG and are unable to check every aspect of accessibility. Manual testing and user testing/feedback is highly recommended.

Some tools give you hundreds of thousands of errors with a big table that offers pagination and search and wishes you good luck. Others are smarter to group problems together, per page and per error type, prioritize them in different ways to allow you to move much faster in fixing the problems and take big steps forwards towards a more accessible site/app. So, choose your automated tool wisely and try before you buy if a trial is offered.

Everybody likes a dashboard that shows you the summary status (hopefully all green) of test results. But what is the functional coverage of those tests? In my opinion automated tools, while useful, don't offer full coverage and stakeholders should not be led into a false sense of security by being shown a green automated accessibility test result.

I agree and additionally, this user experience includes the process of fixing those issues, so customers do not encounter those within our products. They provide a good starting point, which empowers users to uncover and test for additional accessibility issues!

Although automated tools for WCAG testing could offer efficient and scalable means of identifying accessibility issues across websites, the downside of relying solely on them is their inability to detect all accessibility issues, particularly those related to subjective criteria such as usability and user experience. Automated tools may also produce false positives or miss nuanced accessibility barriers that require human evaluation.