Considerations on building accessible puzzle games for Android

This article describes an exploration into building accessible puzzle games for Android. The games are freely available from the Google Play Store at Grid Games, and the code for the games is publicly available. Feedback is invited around how the games can be improved.

Any Vincent van Gogh paintings shown in this article are included based on the non-commercial usage permission stated at Use and Permissions of Collection Images, with credit to the Van Gogh Museum, Amsterdam (Vincent van Gogh Foundation).

Introduction

A few months back I came across an inaccessible sample app which took the form of a matching puzzle game. The app got me thinking how an accessible version of the puzzle game might be built using the same UI technology. After all, I’d say that inaccessible samples have historically been a real problem in that not all devs can leverage the sample. What’s more, whatever dodgy code has led to the samples’ inaccessibility ends up getting copied into who knows how many shipping products.

The end result of my exploration into an accessible matching game for Windows desktop is detailed at How a Card Matching game is compliant with international accessibility standards.

After publishing that article, I built an accompanying app based on a traditional sliding squares puzzle, and made both apps available at the Microsoft Store. And having done all that, I solicited feedback from an expert in accessible gaming in the hope of learning whether the games really had potential or not. After all, I’ve learned from experience that what I think is a good idea, often isn’t really a good idea in practice. The gamer kindly provided some thoughts on my games, and I reacted to that feedback as best I could. Details on that are in the later section “Responding to feedback”.

Given that overall I found the feedback on the Windows games encouraging, I then wondered about how I might build accessible versions of my games for mobile devices. Xamarin Forms provides some support for accessible apps, so I set to work on a Xamarin Android app which hosts both my games. I’m not set up to build iOS apps at the moment, but hopefully it’d be little work to create an iOS version from my Xamarin app given how little Android platform code I’d had to write.

This article describes my considerations around building an accessible Android app, both from a player perspective and dev perspective. The app code is public at MobileGridGames.

Important: The code is only of interest from the perspective of building accessible UI. The code does not demonstrate at all best practices around building a Xamarin Forms app.

Please forgive any inaccuracies this article may contain. I’m not an expert in some aspects of what it took to build the app, but as Vincent van Gogh said: “I am always doing what I cannot do yet, in order to learn how to do it.”

Figure 1: The Pairs game with eight upturned pictures showing paintings and drawings by Vincent van Gogh. TalkBack is highlighting a square showing a painting of a tree, and the displayed TalkBack announcement is: “Matched The Pink Peach Tree. Per Van Gogh: ploughed lilac field, a reed fence, two pink peach trees against a glorious blue and white sky”.

Figure 2: The Squares game with 15 jumbled squares in a grid showing the Van Gogh painting, “The Pink Peach Tree”. Each square also shows a number from 1 to 15. TalkBack is highlighting a square showing the number 12, and the displayed TalkBack announcement is: “12, Row 3, Column 2”.

Figure 3: The Squares game with 15 darkened ordered squares in a grid showing the Van Gogh painting, “The Pink Peach Tree”. In the foreground is a bright window titled “Congratulations”, and TalkBack is announcing its contained text of “You won the game in 174 moves. Would you like to play another game?”

Responding to feedback

This section describes some of the feedback I received on the Windows desktop versions of the games, along with the steps I took to try to respond to that feedback.

I’d really like to thank Graham at Triple Tap Tech, who gave his time to try out the games and provide me with this feedback. That feedback was a true learning experience for me, and is very much appreciated.

Configurable input key

Both games reacted to a press of the Space key as a trigger for very common action. However, in practice, it was easy to unintentionally interact with a touch pad while working with the Space key, and that led to unexpected results. As such, I added an option to have the trigger for the common action be an Enter key press. This was a reminder for me to keep in mind the physical layout of the devices on which the games will be run.

Provide instructions on how to play the games

While I’d provided a link in the games to an app ReadMe in github, that was hardly helpful of me. No-one’s going to want to play a game when it’s a real mystery on what you’re meant to do. So I added some in-app help content to the games, and when building the Android versions I put more thought into how I might provide help to players when they first play the games.

Prefix an accessible name with a status

This one was particularly interesting to me, as I’ve always been a real stickler about accessible names not incorporating status. I’ve felt the name should not change as status changes, and status should be conveyed through some other accessible property. But this feedback said that when navigating to a card in the Matching game, if the card had already been matched, the screen reader announcement should start with “Matched”. Given that that change would improve the efficiency when playing the game, I did make the change in both the Windows and Android versions of the game.

Screen reader doesn’t navigate to all the UI

This was a real surprise to me. I’d believed I’d verified the screen reader navigation experience, but the player had used a different screen reader from the one I’d been using, and it seems that the screen readers didn’t leverage exactly the same programmatic data from the game UI. I eventually discovered what had led to the problem, and it related to the TabIndex properties that had been set on some non-focusable controls in the UI. I’ve documented ad nauseam the importance of checking the programmatic order of WinForms UI as exposed through the UI Automation (UIA) API, but if you want to really support all your customers, you also need to check that the TabIndex properties set on controls (even those that don’t get keyboard focus) matches the logical order. And while doing that, it’s worth keeping in mind that Visual Studio may set the TabIndex property on your controls for you.

Once I’d verified that the UIA order of the elements and all their TabIndex properties matched the logical order of the UI, both the screen reader I was testing with and the screen reader the player was using could navigate through all the UI as required.

Audio

I was told that the Matching game would be improved by adding audio which could be played when two upturned cards match and also when the cards don’t match. So I added options for that, and also accounted for this feedback when building the mobile versions of the games. If future feedback tells me the audio played should be configurable, I’ll update the apps accordingly.

Include more shortcuts

While the apps had some function key shortcuts, some common actions didn’t have shortcuts, and even those shortcuts that did exist weren’t obvious. So I added more shortcuts and made sure they were documented in the help content.

Screen reader announces an inappropriate control type

This was the one piece of feedback that I couldn’t do anything about. From an implementation perspective, the squares in the games are Buttons. That’s very deliberate, as Buttons provide a lot of accessibility by default. But from a game player’s perspective, the squares are squares, not Buttons. So the feedback reasonably asked for screen readers to announce the squares as squares, and not Buttons.

While UWP XAML provides a very handy LocalizedControlType property (which I once discussed at UWP XAML: Setting the localized control type on a custom control), that property is not readily available for WinForms apps. I spent a while trying to figure out how I might achieve the same result, but came to the conclusion that it’s just not practical. I posted a related question at How can a custom UIA LocalizedControlType string be set on a DataGridView cell? and at the time of writing this, the question’s not had a response.

Playing the game

This section describes some considerations around making the games enjoyable for as many players as possible.

Aim of the games

Squares Game

The aim of this game is to rearrange a collection of fifteen squares such that they appear in order from right to left and top to bottom in a 4-by-4 grid. One spot in the grid is always empty, and only the squares adjacent to the empty spot can be moved. Players can configure whether numbers should be shown on the squares, and whether a picture selected by the player should be shown across all the squares.

Game tips:

• Don't use a swipe gesture to move a square. Support for swipe gestures has yet to be added to the game. So tap on a square to move it, or use one of the other forms of supported input as detailed later in this article.
• Don't tap on the empty square. Instead tap on the squares next to the empty square.
• Practice moving numbered squares around before changing the setting to have a picture shown on the squares.

Figure 4: The Squares game with 15 unordered squares, each square showing a number in a large font from 1 to 15. TalkBack’s highlight is shown around the empty square, and its announcement of “Moved 1 up.” is shown near the bottom of the image.

Figure 5: The Squares game with all squares ordered to reveal a picture of the planet Jupiter. Each square also shows a number from 1 to 15.

Pairs Game

The aim of this game is to turn up cards shown in a 4-by-4 grid, and find matching pairs of cards. If two cards are turned up and they don’t match, then the next tap in the game will turn the unmatched cards back down. If two upturned cards do match, they remain turned up for the rest of the game. Players can select whether audio is to be played when two upturned cards don’t match, and when they do match.

The default set of pictures shown on the cards are photos I took around Lancashire and Yorkshire not too long ago.

Figure 6: The Pairs game showing eight matched pairs of default pictures.

Players can also select a set of their own pictures to appear on the upturned cards. In that case, accessible names must also be provided for the pictures, and optionally accessible descriptions. To select your own pictures, select an image on your device which lives in a folder that contains only eight pictures. In addition, the folder must contain a file called “MatchingGamePictureDetails.txt”, which is a text file with one line of text per picture, and each line contains the name of the picture file followed by the picture’s accessible name and description, with these three datapoints separated by a tab. This is the same file format as that created when playing the game in the Windows version of the app.

As a sample set of pictures, here’s a link to eight photos I took around Manchester recently, along with the accompanying MatchingGamePictureDetails.txt file, Manchester pictures. If you download the files in that folder to a new folder on your device, you can have them shown in the Pairs game. And after that, use whatever other pictures you and those around you would enjoy!

Figure 7: The Pairs Settings page listing custom picture files and associated accessible names and descriptions. The TalkBack announcement is shown near the bottom of the image.

Figure 8: The Pairs game showing eight matched pairs of custom pictures, with the TalkBack highlight around one square. The TalkBack announcement is shown near the bottom of the image, and includes the accessible name and description of the highlighted square.

TalkBack Screen Reader

The games provide the typical experience when navigating through, and interacting with, the contents of the UI. The games also provide some custom announcements in response to specific action in the game. For example, when attempting to move a square in the Squares game when the square cannot move, TalkBack will say: “A move is not possible from here.” And when turning up two matching cards in the Pairs game, TalkBack will say: “That’s a match.”

When the games are first run, a window appears which encourages players to read the help content before playing the game. The help content is reached through the app menu, but unfortunately today the menu button has a name of “OK”. I’d say this is really confusing, but at least the welcome message can help players by stating that the menu is called “OK”. This is far from ideal, but it’s probably better than not saying anything about the problem menu button at all. As I write this, this is the discussion I’m aware of on the subject of the menu button being called the “OK” button, How can I change the accessible name of the "OK" hamburger button in an app built from the Xamarin AppShell template?

I noticed during development of the Pairs game, TalkBack sometimes didn’t announce the row and column of the square beneath my finger as it should. I’ve not yet figured out why TalkBack does that, and if this is something that impacts other players, I’ll investigate further.

Figure 9: The Pairs game with 10 upturned squares showing 5 pairs of planets. TalkBack’s highlight is shown around a square showing Mercury, and the TalkBack announcement is shown near the bottom of the image, “That’s a match.”

Voice Access

Important: Voice Access only works today in the Squares game if numbers are shown on the squares. This will be fixed at the earliest opportunity.

To tap a square in either game in the app, speak the number of the square. In the Squares game the number of the square is the number shown on the square, and in the Pairs game the numbers of the squares run from left to right and top to bottom in the grid.

Given that the menu button has a name of “OK”, speak “OK” to have the menu shown. The Settings button shown at the top of each game can only be accessed through the Voice Access “Show numbers” feature.

And a tip for devs considering Voice Access support, if elements have names such as “1”, “10”, “11”, then speech of “one” is ambiguous across all elements whose name starts with “1”. To account for that, the Labels in the Pairs came have Text such as “one”, “ten”, and “eleven”.

Figure 10: The Pairs game with 11 upturned squares showing 5 pairs of planets and a single picture of Earth, which is shown on the square in the 11th position in the grid. The Voice Access feature is showing the following text at the top of the image, “Tapped eleven”.

Switch Access

To tap a square in either game, scan to the square of interest, and press the switch.

Figure 11: The Pairs game with 12 upturned squares showing 5 pairs of planets and single pictures of Earth and of Venus. The Switch Access menu button is shown at the top of the image, and the Switch Access highlight is shown around one of the squares in the game.

Font Size

The Squares game provides an in-app setting for showing larger numbers on the squares. Both games also respond to the Android “Font size” setting, such that text anywhere in the apps can be shown larger.

Figure 12: The Pairs Settings window with its contents being shown with a large font. The list of custom pictures of Planets selected is clipped near the bottom of the window and can be scrolled into view.

Magnification

Both games support the Android Magnification feature.

Figure 13: The Pairs game shown magnified, such that at this magnification, only 4 of the 16 grid squares can fit into the view at any time. The 4 squares shown in the view in this image contain pictures of planets.

Dark Theme

While the games do not respond to a change in theme made outside the app, the app does provide its own option to turn on a dark theme.

Technical considerations

This section describes some of the technical considerations around building a great experience for as many players as possible.

Important: Delivering a great accessible experience starts by using whatever standard controls are being provided by the UI framework used to build the app. As such, the app uses standard Xamarin Forms controls such as Buttons, CheckBoxes, and CollectionViews. The next step is to make sure that the order in which the UI is defined matches the logical order of the UI, and so deliver the required navigation experience regardless of how players are interacting with your app.

TalkBack Screen Reader

By using standard controls and a XAML layout definition that follows the logical order in the UI, I was well on the way to delivering a usable TalkBack experience. TalkBack will navigate through the UI in the expected manner, and generally make some useful announcements.

In order to deliver a more usable experience however, I took two additional actions.

My first action is to set the Xamarin AutomationProperties properties where necessary. The AutomationProperty.Name property sets custom accessible names, which is particularly handy when a name needs to change to reflect the visuals of a card. For example, the Name might be “Face down” before a card is turned up, and then changes to reflect what’s shown on the upturned card.

The code up at Grid Games forms code shows all the ways in which the AutomationProperties are used, and really, use of those simple properties can make the difference between shipping an unusable product and a usable product.

The second TalkBack-related action taken by the games is to have custom announcements made. This is implemented using Android platform code shown in DependencyService.cs. It’s easy to go overboard with custom announcements, so this is another place where I rely on player feedback to help me learn which announcements would be helpful in practice and which would not.

Voice Access and Switch Access

When I first started working on the app, I assumed that the first version would support keyboard use. The simplest way to support keyboard use seemed to be through responding to a change of Selection in the CollectionView. So I took that approach to support both keyboard use and touch use, despite the suggestion at Handling Item Taps that a tap gesture might be a more straightforward way to support touch input.

However, after some testing, it seemed that the tap gesture support would be required to support Voice Access, and as such I replaced my SelectionChanged approach with the tap gesture approach. However, yet more testing then indicated that the SelectionChanged support would be required to support Switch Access.

The end result is that I had the games react to both SelectionChanged events and tap gestures, and as far as I know, the games react as required now to all of touch input (with and without TalkBack on), Voice Access, and Switch Access.

There was one interesting challenge when adding support for Voice Access to the Pairs game. The squares in the game don’t show text, and typically in that situation support for Voice Access could be achieved by setting an AutomationProperties.Name matching what players are to say in order to interact with the square. However, in order to support TalkBack, I’d already set the AutomationProperties.Name property to be something appropriate for the card, and that did not match what was required for Voice Access. As such, I added a Label for each square, whose Text property was “one” to “sixteen” and which was visually obscured by other card visuals. This approach seemed to result in a working experience for players using either Voice Access or TalkBack.

Font Size

By using standard controls in the app, I had to do nothing specific to have the text shown in the app change size as the Android Font Size setting is changed. However, it’s crucially important to design and implement a UI layout that can handle large text, otherwise all sorts of clipping can render the app unusable.

The most impactful change I made to account for this was to host a lot of the UI in ScrollView controls. This means that as the contents of a page becomes too large to be shown on the screen, players can scroll the content to bring all of it into view. I ended up using ScrollViews in many places, and with some other specific rearranging of controls to avoid clipping, I believe all the content is now accessible with the largest Android Font Size setting.

Magnification

Due to my use of standard controls, the games support the Android Magnification feature by default. Nice!

Dark Theme

Many players prefer a less bright feel to the visuals in a game, and so the games support an in-app setting to select a dark theme. Perhaps in the future the game will automatically respond to turning on an Android dark theme setting outside of the app, but not today.

Implementing a dark theme in the app was relatively straightforward, but having done that, I found it particularly interesting to consider exactly what colours should be used for the dark theme. For example, I’d used a very dark green for some visuals, but then realized that my Buttons’ hit area was practically invisible against the black surroundings. Similarly, there was very low contrast between a menu item’s selection feedback and its surroundings.

So with WCAG 1.4.11 “Non-text Contrast” in mind, I tweaked some colours to provide some better contrast when the dark theme was active, without increasing the brightness of the UI too much overall. At some point I’ll probably continue making similar changes, for example, to show the hit target on the font size Picker.

Figure 14: The Squares Settings window shown in a dark theme, with a black window background. The buttons in the window show white text on a dark green background, with a white border.

What do accessibility checkers have to say about the app?

When preparing the release of an app at the Google Play Store, the developer site automatically does the very handy thing of highlighting what it feels might be an accessibility issue in the UI. In the latest release of the app, it highlights the following three potential issues.

1. Low Contrast. The tool highlights in a few places text which has a low contrast against its background. For example, some dark-on-light text has a contrast of 4.37:1, and some light-on-dark has a really worrying contrast of 3.04:1. In both cases, the text should have a minimum contrast of 4.5:1 against its background. But as far as I can tell, the app’s showing the default contrast based on the template I used in Visual Studio to create the app. I was really hoping not to have to change these default colours shown in the app’s Light Theme, but given the results, I expect in a later release I’ll override the default colours with my own higher contrast colours.
2. Questionable use of android:contentDescription. The tool detects that the Squares Settings window shows a number size Picker which has an android:contentDescription, and that description might get announced by a screen reader rather than the content of the Picker. I’m not aware of anything I’ve done which would have influenced the description, so perhaps this is the default result when using a Picker. When I navigate to the Picker, TalkBack does announce the current selection, followed by: “Button, Number size for Number size”. That seems a little unusual, but at least all the important information is announced, so I’m not planning on changing the app to account for what the tool’s highlighted here.
3. Touch targets are too small. The tool points out that the hit targets for the CheckBoxes shown in the app are too small for reliable interaction. Again, I don’t think I’ve done anything which would lead to this, as it’s the default size of Xamarin CheckBoxes, and so I won’t be changing the app in response to this.

I also pointed the Accessibility Insights for Android (AIAndroid) tool to the games when they were running in the Visual Studio Android Emulator. The only issue that AIAndroid highlighted was related to the size of the hit targets in the CheckBoxes shown in the app, as was highlighted by the Google accessibility checking tool.

Figure 15: Accessibility Insights for Android reporting two failures in the Squares Settings page, relating to touch input target size.

Figure 16: Accessibility Insights for Android reporting no failures in the Squares game main page.

All in all, the only issue reported by the accessibility tools that I’m planning on reacting to relates to increasing the contrast of the text that’s currently shown using the default colours from the Xamarin VS template.

Summary

This exploration has been an interesting learning experience for me. I’m really hoping that there’s potential for these Android games to be fun activities for players using a variety of forms of input at their devices, and also that the code can help other devs consider how their own products might be enjoyed by more people.

There are many, many improvements that can be made to these games’ experiences and code, but this seems like a reasonable way to get the ball rolling. Who knows, if I do get positive feedback, maybe I should explore updating the Xamarin app and build an iOS version.

The games can be downloaded from the Google Play Store at Grid Games, and the code’s up at MobileGridGames. Please do let me know how I can make the games more usable and enjoyable for you or someone you know.

I know when building an app, sometimes the subject of accessibility can seem daunting. There just seems so much to consider, and sometimes technology doesn’t seem to support you in all the ways you need it to. But often through a collection of straightforward and very practical small steps similar to those taken by these grid games, it really is possible to build experiences that will support many more players.

As Vincent van Gogh said: “Great things are done by a series of small things brought together.”

All the best,

Guy