AI vs humans for accessibility advice

This article is going to be a bit technical, so I'll start with the summary. AI still isn't as good as a human with accessibility experience or an assistive technology user, but it is getting better.

A recent article suggested programmers are choosing ChatGPT over Stack Overflow to get answers to programming questions. Ironically, ChatGPT is scraping Stack Overflow for the answers, but has a more user-friendly interface (debatable, but in this context conversation works well).

Steve Faulkner, Scott O'Hara, Adrian Roselli and others have written about how poor the results are when you ask ChatGPT about accessibility.

I decided to check in on what kind of accessibility guidance ChatGPT would give for this prompt:

"Explain how to use aria to create an accessible button from a div"

Note: making buttons from divs isn't an optimal approach for accessibility, but it is common and somewhat complex, making it a great test case.

ChatGPT 3.5 says:

"This is particularly useful when you want to create custom-styled buttons or when you need to make non-button elements behave like buttons for better accessibility." It gives me a code sample that has the correct role="button", a tabindex="0" and an aria-label="submit form". The label overrides the visible button label of "Submit" - an uncessary use of aria-lable that makes it harder for anyone using voice commands to select the button. It adds onclick and onkeydown functions to handle keyboard interactions (not ideal).

I'd give that a C grade.

ChatGPT 4 helpfully gives a step-by-step process for writing the code, includes the correct role, tabindex and skips the aria-label (good!). It explains aria-pressed is optional and when to use it. It also writes better JavaScript to support keyboard interactions - creating event handlers for the enter key and spacebar. It gives me some visual styles to make the div look like a button (including a pointer cursor), but no focus or hover states.

Let's give that a B grade.

Using ChatGPT in Bing precise mode, I get the role, tabindex, aria-pressed (no explanation about it or that it should toggle) and it says I need JavaScript event handles for click and keydown but doesn't give an example and there's no CSS either. It gives me links to three helpful button accessibility specific articles and one way too generic ARIA article.

Another C grade.

Now, I wanted to explore using ChatGPT to actually learn about accessible buttons and not just spit out code. I found two helpful prompts (linked in the references below).

Using Rory Preddy?? 's accessibility prompt with ChatGPT 4, first it tells me that making buttons from divs is not a good idea and why. It gives me a list of things I need: role, tabindex, aria-label (with an explanation of when I don't need one), keyboard event handling and the optional aria-pressed. It shows me a code snippet with HTML and JavaScript event handlers.

Not that much more useful than a straight up prompt. It is pretty accurate, but not ideal for learning. Grade B.

Using Ethan Mollick 's tutuor prompt with ChatGPT 4, I was asked four questions for context, but the output wasn't specific to my use case anyways (a modal with a save and cancel button) and I didn't get any JavaScript code. It also unnecessarily duplicated the button text in an aria-label. On the plus side, it gave me some lovely analogies for learning:

"Imagine a library with no labels on the books. You might know it's a library, but finding a specific book would be difficult. Accessible buttons are like putting labels on those books, making it clear what each one represents and how to interact with it."

"When you use a div as a button, it's like having a door with no handle. People can see it, but they might not know how to use it." This was a missed opportunity to point out that not everyone can see.

I'll give this a B Grade as well.

In summary, human guidance via blogs, consulting services, courses, user testing, etc. is still the most reliable and user-friendly way to learn how to build accessible digital products.

P.S. Fable does this via Upskill

References:

• Inside the Decline of Stack Exchange
• ShatGPT
• Setting expectations for asking ChatGPT web accessibility questions
• No, ‘AI’ Will Not Fix Accessibility
• ChatGPT prompt for accessibility
• Tutor prompt