How can you test game accessibility for diverse players?

An extraordinary example of innovative user research is Microsoft's Games Accessibility Test Service- a center that recruited "game-loving persons with disability" to conduct unskilled playthrough evaluations. The center morphed into a regularly scheduled stable of semi-skilled, disabled a11y multimedia consultants. There we mixed personal experience with the Xbox Accessibility Guidelines to create better consumer products. Whether it be surveys, playthrough, or MGATS, using disabled persons grants invaluable insights. But you HAVE to provide extra flexibility, empathy, and appropriate budget for items like standing desks. A failure of Microsoft's was rigid 8 hour test shifts that don't accommodate all medical conditions- and ZERO budget.

Craft empathetic personas representing a spectrum of abilities within your player community. Tailor testing scenarios and feedback approaches to resonate with these personas. This compassionate approach ensures a thoughtful exploration of accessibility, striving to embrace and address potential barriers with care and understanding.

The principles contained within the XAG's (Xbox Accessibility Guidelines) don't just apply to Microsoft-only products, but all console games, regardless of manufacturer. They are Open Source and all Devs should bookmark that website. However, they are not infallible and can even be problematic. Accessibility is a crossdisciplinary venture requiring intelligence in all forms and has so many gray areas, that misunderstanding the nascency of the ever-developing Guidelines presents a major risk to the uninitiated. A skilled consultation often reveals novel bugs unique to that specific scenario that wasn't covered by a Guideline (yet). You should have a NAG, "Not A Guideline" type of bug category and empower testers to report those there!

One of my favorite Accessibility Tools to use is the CCA- or Color Contrast Analyzer. It is a simple tool to help us identify issues for users whom are sight affected. Over time, after evaluating many color schemes, I've recognized a trend that I want to share. Based on my experience many light blues, light greens, light purples and particularly light grey tones are nearly ALWAYS a WCAG Distinguishable 1.4.3 Color Contrast violation. When you see those, know they are trouble- be ready to CCA test the scheme and document any bugs. But remember- getting it right the first time around is cheaper than remediation. In the limited time you have with developers, this is a priority to educate them on, but is easy to both instruct and retain.

Company Forums may benefit big studio Devs, but indie Devs may not have that infrastructure. When searching for a11y feedback I had success drilling down on Reddit's specialized "subreddits" dedicated to specific disabilities. I obtained good data and even made offline friends. But most platforms come with a tradeoff- and dealing with an extremist political echo chamber like Reddit yields limited results. In this case, the self-appointed activist trolls looking for a fight will consistently inflame & hijack every single conversation over their weaponized phraseology standards. I now rely on Twitter/X as it allows surveys, represents a more universal sample group, no cranky mods and (unlike Reddit) will not censor political Free Speech.

A great Accessibility Expert can be worth a million bucks (literally) to an organization. Having the social and technical skillset to discern & mitigate multidisciplinary risk is essential in an era where a single mistake can cost Billions of dollars overnight (see Bud Light, Pelaton). The only problem? There are no "Accessibility Experts". The dirty secret is that a11y is beset by opportunists, overnight instant rice CPACC "experts", and fake evangelists. I've spent 15 years in this space and only encountered 4 people capable of: testing, educating, AND inspiring alike. We're still professionalizing- but look for a job candidate personally afflicted by a serious disability, with "techy background", and certs for your team's best result.

In accessibility, a Test Pass or Test Checklist should never carry the same weight as in standard game/software testing. Unfortunately, an overreliance of test checklists exists that is a vestigial carryover of developing this offshoot of software testing. That is because almost HALF of all issues identified by senior experts would not have been covered by a WCAG, XAG, or other guideline. For example, an evaluator may notice two clickable input fields are too close to click easily, or something is cognitively confusing but not covered by a specific WCAG. Another example is CAPTCHAs which are viciously inaccessible- yet not any violation. Thus, when building an evaluation process, keep in mind 40% of violations aren't pass identifiable.

Accessibility is still very much a growing field. We are VERY much in our infancy, and one day will look back at our current "expertise" as the Jurassic Period, or even the Wild West. Too often new testers or program managers look to a handful of senior personalities that have established dominance by dint of merely being around longer, or have a bigger platform with more followers- and seek to emulate them and parrot talking points. Given the many kinks still working out, and the nascent state which the field is in, that may not ultimately be the best technique or politic to hang your hat on. Evaluate every single scenario for nuanced, multidisciplinary metadata and optimize- don't rely on dinosaurs or soon-to-be outdated junk science!