Building Grid Games as a Windows MAUI app: Part 1

This article describes my experiences attempting to build an accessible MAUI version of the Grid Games app.

Please note that thoughts here are entirely my own, and while I’ve tried to validate all I say here, it’s possible that some things I state may not be accurate.

The Story So Far…

Last winter I happened to come across an inaccessible puzzle app. The app’s inaccessibility left me somewhat discontented, and so I set about exploring how I might build an accessible version of the puzzle using the same Windows Forms UI technology that the original app had used. I shared my thoughts on that experience at How a Card Matching game is compliant with international accessibility standards, and published two related apps at Accessible Matching Game and Accessible Squares Game.

Having done that, I thought how handy it would be if I could build accessible versions of these puzzle games with one code base that would run on all of Windows, Android, and iOS. I explored this using the Xamarin UI technology, and shared my experiences at Considerations on building accessible puzzle games for Android, Considerations on building accessible puzzle games for iOS, and wrapped it all up at And So Concludes Grid Games: V1. The Android and iOS apps coming out of that exploration are available at Grid Games for Android and Grid Games for iOS, and the code for these apps is at Mobile Grid Game for Android and iOS.

While I found the above investigations fascinating, and was pleased to have been able to end up with the apps at the various Stores, there was something missing. I’d really hoped to have been able to build a Windows version of my Grid Games app, in addition to the Android and iOS versions. Alas, this didn’t seem practical, as it seemed that the technology I was using didn’t support setting accessible names on the items in the games when the app was running on Windows. Without that support, there could be no Windows version of the app. Based on my own investigations, and the discussion at Xamarin issue 3261, it seemed I wasn’t going to be able to build an accessible Xamarin Windows version of my app.

Since then, there’s been a lot of discussion around the newly released MAUI UI technology. It was pretty inevitable that I’d end up investigating whether I could build my Grid Games app using MAUI, with a goal of having accessible versions of my MAUI app run on all of Windows, Android, and iOS.

This article describes my experiences in this attempt.

Figure 1: The Pairs game with upturned cards.

So where are the Android and iOS versions of the new MAUI Grid Games app?

When I built the Squares game in the Xamarin version of the Grid Games app, I used the?SyncFusion ImageEditor control. That control made it practical for me to chop up an image into fifteen jumbled pieces, and then have the player reassemble the original picture from all the pieces. I’ve been told that that ImageEditor control will become available for MAUI apps in December 2022, and so I can’t start working on the Squares game for my MAUI Grid Games app until then.

I don’t want to replace my existing Grid Games app at the Android and Apple Stores with versions that don’t include the Squares game, so until I can build a Squares game for my MAUI app, I’ll focus only on a Windows version of the app.

Adding a new “Where’s WCAG?” game

I didn’t really fancy publishing a Grid Games app with only one game in it, so I built a new game into the new MAUI version of the app. I felt it might be interesting to explore having the new game relate to the Web Content Accessibility Guidelines, “WCAG”. After all, the more people who are aware of WCAG, the better it’s going to be for everyone.

The new game presents a question asking which WCAG number covers a particular WCAG grouping, and the player selects a square in the grid showing that number. The game then optionally asks a bonus question relating to WCAG, and the player selects one or more specific WCAG to answer that question.

As with all my Grid Games apps, the main goal of the new app is to explore how I can build a game-playing experience that’s the most usable as possible for all players, while leveraging all that the relevant UI technology has to offer. I’m hopeful that this particular new game has the additional benefit of helping players get a little more familiar with WCAG.

Figure 2: The Where’s WCAG? game showing the question Where’s “Keyboard Accessible”?

A quick reminder about Accessibility Insights

In this article, I mention something called Accessibility Insights for Windows. This free tool is a really handy tool for helping you get familiar with your Windows app’s programmatic interface. It’s always important to keep in mind that you have customers who are depending on the quality of that programmatic interface.

Accessibility Insights can also run “FastPass” tests, which can draw your attention to potentially problematic UI. The tool is quick to run, and once you’re familiar with it, it’ll feel natural to always run it whenever you update your UI.

In this article, I also mention something called UI Automation, (UIA). UIA is the technology that Windows uses to generate your app’s programmatic interface for assistive technologies such as screen readers. I wrote a two-part introduction into UIA a while back, starting at UI Automation: An Incomplete Guide for UI builders.

Figure 3: Accessibility Insights for Windows reporting the UIA hierarchy of the plant cards shown in the Pairs game.

For a real-world example of how helpful Accessibility Insights for Windows can be, here’s an issue that it’s made me aware of in the new app. When I created the app’s Help pages, I put the help content in a scrollable control which supports the UIA TextPattern. A screen reader can navigate through that help content in whatever manner my customers prefer. “Great!” I thought to myself. When I later pointed Accessibility Insights for Windows at the UI, I realised that I’d forgotten to give the control containing the text an accessible name. This was an important omission, and I’ll fix this with my next update to the app. Thank you Accessibility Insights!

Figure 4: Accessibility Insights for Windows reporting the UIA hierarchy of the Pairs Help page.

Setting the accessible name on the grid items in the game

Given that I couldn’t build an accessible version of my Xamarin Grid Games app for Windows, the starting point for me in this new investigation was, will I be blocked for the same reason using MAUI?

And the very satisfying answer was: No. ??

All the grids in the games are implemented using a CollectionView, and the outer container for the CollectionView’s items is a Grid. I can bind that Grid’s SemanticProperties.Description to whatever property I want to, and that Description gets exposed to screen readers as the UIA Name for both the UIA group element representing the Grid, and also the UIA list item representing the CollectionView item. All this means that I can expose whatever accessible name seems most helpful for each item, and that name gets announced by screen readers as players arrow through the items in the grid. Without this support, I couldn’t have started to build the MAUI app.

I did hit one unexpected issue where despite being able to dynamically set the accessible name on the CollectionView item’s Grid during game play, sometimes the updated name didn’t get propagated up to the parent UIA list item in the UIA hierarchy for the app. Maybe I’d botched the binding somehow. So as a temporary workaround for this issue, whenever I set the name, I set it twice, both with and without a trailing full stop. By doing that, I always got the list items’ names updated. I’ll revisit that action at some point and hopefully tidy up the code.

Note that the current version of the app does not attempt to dynamically set a SemanticProperties.Hint on the CollectionView items’ Grids, so I don’t know if an updated Hint gets reliably propagated up to the UIA list items as the items’ UIA HelpText.

Figure 5: Accessibility Insights for Windows reporting the UIA hierarchy of the safety cards shown in the Pairs game.

Preventing use of mouse and touch at the games

It seems that at the time I built the games, MAUI’s CollectionView does not support UIA hit testing. What’s more, if a UIA hit test is attempted, the app hangs. I logged a related issue at UI Automation hit testing on CollectionView items hangs Maui app. This means that if any player inputs with touch or mouse while using a screen reader which uses UIA hit testing, such as Narrator, the app hangs. Similarly it’ll hang using the Accessibility Insights for Windows tool with the tool’s default settings.

One option here would be for me to tell players that they can’t use touch or mouse at the games while a screen reader’s running. Another option would be for me to try my best to ship an app that provides an equal experience for all players, and that would mean preventing touch and mouse input for all players.

I’ve gone for the latter option, and if a player uses touch or mouse at a grid when a screen reader’s not running, they get told that the game can only be played with a keyboard.

When UIA hit testing does at some point work at the CollectionView, I’ll update the app so that all players can use touch or mouse at the app.

Figure 6: The Where’s WCAG? game presenting a message stating that the game can only be played with the keyboard.

So can the games be played using a keyboard?

Now that touch and mouse input is intentionally blocked, which also unfortunately means that Windows Speech Recognition can’t be used at the app, keyboard use had better work. Otherwise the games can’t be played by anyone, and where’s the fun in that?

I tried to make the games playable through use of SelectionChanged event handlers alone but didn’t manage to come up with anything sufficiently robust. I really wanted to use traditional key event handlers to help me be very precise in how the app would react to presses of the Space and Enter keys. As far as I could tell, platform-independent key event handlers weren’t available in MAUI at the time I built the app, so I added some Windows-specific PreviewKeyDown code to achieve what I needed.

While I was doing this, I kept in mind some player feedback I’d received relating to my original Windows games. When I first shipped those games, they only supported a press of Space to invoke an item in the grid, and the player feedback was that it was easy to unintentionally interact with the touchpad while pressing the Space key. In response to the feedback, I added support for an Enter press to invoke an item, in addition to supporting a Space press. That’s why I made sure that my new MAUI app supports both a press of Space or Enter when keyboard focus is in a grid.

Another very important step here was to document how to play the games with the keyboard, so I updated the Help content to explain all this. No player should be left wondering what’s going to happen in the app if they press a key.

Presenting and navigating through a set of many WCAG

The Bonus Question feature of the new Where’s WCAG? game raised an interesting design question for me. The game would ask a question, and the player would then select one of more WCAG from a set of nearly ninety WCAG. How should all those WCAG be presented, and what interaction experience would lead to the most efficient way of selecting the WCAG?

At first I’d hoped to show the full set of WCAG on the Bonus Question page, with each WCAG having an associated CheckBox. I’d provide some navigation links at one side of the page to provide quick navigation to the various groups of WCAG on the page. The links would always be in view, and would have access keys to activate the link regardless of where keyboard focus happened to be. But it seems that cross-platform accessible links aren’t supported in MAUI today, so that design wasn’t an option.

I also found that it took too long to load up a page which showed all the CheckBoxes, so I’d need to go with an approach where only a logically grouped subset of the WCAG was shown at any given time.

Another approach that seemed interesting would be to have a tabbed page design, with each tab item being associated with a WCAG grouping. When the tab item was selected, the page would get populated with all the WCAG CheckBoxes associated with the relevant grouping. I explored this design further with the?SyncFusion TabView control, but it seems that that control can’t be used with the keyboard, so that’s not an option either.

In the end I went with a collection of Pickers near the top of the page, and when a selection’s made in a Picker, a specific group of WCAG CheckBoxes is shown. This design might seem a little unconventional, but it does mean that with a small number of key presses I can have any specific group of WCAG shown, then tab through the CheckBoxes in the WCAG group and select specific WCAG of interest, and then submit my answer with a press of another access key. While this player experience will take some getting used to, hopefully it can be fairly efficient. I’ll consider how I might redesign the page as MAUI continues to develop.

Figure 7: The Where’s WCAG? game’s Bonus Question page.

A few other unexpected things

During development, I did hit a few other mysterious behaviours which I couldn’t figure out. Once I was familiar with these and knew how to work around them when coding or when interacting with the app, I found they didn’t block me from playing the games. I expect over time these issues will go away, either through updates to MAUI, or me learning more about how I’m meant to build MAUI apps.

I logged the issues I hit at Github.

How does MAUI help make the app accessible?

MAUI does provide a number of very important features by default relating to making my games accessible. For example, I can use the keyboard to move keyboard focus through the UI and interact with controls, and a UIA hierarchy is exposed by default for assistive technologies such as screen readers to consume.

What’s more, MAUI provides me with some additional features which I can leverage as I work to build games which are more usable to more people.

This section mentions some of the MAUI features leveraged.

Semantic Properties

By default, MAUI will expose a programmatic interface associated with the UI presented in an app, and a dev can tune that default interface to make the experience more usable. At Build accessible apps with semantic properties, it describes how devs can set a variety of semantic properties on UI elements if the default accessibility of those elements is not sufficient for users. The new Grid Games app leverages all of the SemanticProperties properties available, resulting in helpful UIA Names, HelpText, and HeadingLevels being provided to screen readers.

Note that the app also leverages the ?AutomationProperties.IsInAccessibleTree property with a goal of exposing the most uncluttered UIA hierarchy as practical.

Figure 8: Accessibility Insights for Windows reporting the UIA properties for a label in the Pairs Settings page.

Semantic screen reader

The app raises some custom notifications for screen readers to announce in response to specific events happening in the app. Care must be taken when raising such notifications, as they can easily become very detrimental to the player experience. I am looking forward to feedback on how I can improve the app’s use of custom notifications.

The app leverages MAUI’s very handy cross-platform support for raising notifications for screen readers, using the semantic screen reader. To achieve this in my previous versions of my app, I had to add platform-specific code. I do feel MAUI’s support for cross-platform notifications is an important step forward.

During development I found that one notification I raised soon after the app had started, did not get announced. I’m not too surprised about that, given that the UI probably wasn’t fully loaded by then. So for that early announcement I added a short delay before calling SemanticScreenReader.Default.Announce().

I also dabbled with SetSemanticFocus() during development but found an exception was raised whenever I called it, even if I called it long after the related UI had been created. A new sample MAUI app seemed to be able to call SetSemanticFocus() without a problem. I don’t know what the issue was with my attempt, and I expect I’ll revisit that at some point.

Figure 9: Accessibility Insights for Windows reporting UIA events being raised while playing the Where’s WCAG? game.

Light and Dark modes

While building some of my earlier apps, I’d experimented with support for Windows high contrast themes, and in-app support for a dark mode. My new MAUI Grid Games app simply leverages MAUI's styling support for Dark and Light modes. If a player selects Light or Dark in the Personalisation page of the Windows Settings app, then the Grid Games app will automatically show colours appropriate to the selected mode. This all seemed a pretty straightforward way for the app to support the player’s preferred mode, and it felt good to not be adding an in-app feature for Light and Dark mode.

I made sure that whatever mode was selected, the app shows foreground and background colours with a reasonable contrast for all visuals. Not only does this relate to text, but also such things as grid items’ borders and keyboard focus feedback around an item. A couple of related WCAG that are helpful to keep in mind are WCAG 1.4.3 Contrast (Minimum) and WCAG 1.4.11 Non-text Contrast.?

I’ve taken no specific action in the app related to the app running while one of the Windows High Contrast themes is active. I expect I’ll explore that topic at some point.

Figure 10: The Pairs game shown in the app’s dark mode.

Figure 11: The Where’s WCAG? game shown in the app’s dark mode.

Support for Access Keys

MAUI provides support for setting Access Keys on interactable UI elements like Buttons and CheckBoxes, and I leveraged that support all over the app. Use of access keys can significantly reduce the amount of tabbing required to reach all parts of the UI when playing the games, and as such, can make it far more efficient to complete tasks in the app.

I feel really strongly that use of access keys in a Windows MAUI app can make an important difference to the interaction experience, with very little dev work required. So all you Windows app devs out there – please do consider how you can leverage access keys to help your customers!

In my experiments, I couldn’t seem to set an access key on all the types of controls I’d like to. For example, I’d like to set one on a CollectionView in a game, with the intent that a press of its access key would set focus to an item in the CollectionView. If access keys do become available on more controls in the future, I’ll leverage them in whatever way seems helpful to players of my games.

Figure 12: The Pairs Settings page showing the Access Keys available on the page.

Making the app understandable

WCAG 3 Understandable describes many important topics relating to making an app easier to understand. Much of that isn’t about implementation, but rather, like much of accessibility, it’s about design. I have tried to make the app more understandable through use of instructions on pages, details in the app’s Help content, and even ways to reach sample datafiles when customising the game experience.

Importantly, I show a message when the app is first run which makes the player aware of the app’s Help content and how to reach that content.

While what I’ve just said isn’t at all specific to MAUI, I just wanted to say it given how critical it is that all your users can understand how to efficiently get the most from your app’s features. It’s easy when building an app to get bogged down with implementation details, and not take a step back and consider just how understandable your features are for each and every one of your customers, even when the features are perceivable and operable.

Publishing the app

I’ve been working on my new MAUI Grid Games app using the Visual Studio Community 2022 Preview, and as always, VS made it very straightforward to prepare the app for submitting to the Microsoft Store. My app submissions initially got rejected for a couple of reasons.

The first reason was apparently because my screenshots of the app included something in addition to the app. I’d deliberately included the NVDA Speech Viewer in some pictures, in order to show what NVDA had been saying as I’d interacted with the app. Given that that is apparently not acceptable, I updated the screenshots to only include the app.

The second reason for the app being rejected was because of the low-quality image I’d provided as the app icon. In the past when submitting an app’s visual assets, I’d included PNG images at a bunch of resolutions, but now, I apparently needed to supply an SVG image which would later get scaled to whatever resolution was required. I don’t have an SVG editor app, so I manually edited in Notepad the SVG assets that automatically get created with a new MAUI app, and ended up with an icon that was acceptable to the Store.

Summary

This exploration is clearly very much a first step along a journey. The app has a number of issues and constraints, and I won’t try to make out it’s even a first version of the app. It has however reached a point where a player with a particular interest in using the app, could try it out and provide me with feedback.

I really am very excited about the potential for MAUI to one day help me build fully accessible games for all of Windows, Android, and iOS.

The Pairs game can be played with the keyboard, with or without a screen reader. And players can use their own sets of pictures with associated accessible details, resulting in a much more personal game-playing experience.

The new Where’s WCAG? game can be played with the keyboard, with or without a screen reader, and can help players get a little more familiar with various WCAG groupings.?Players can have bonus WCAG-related questions asked while playing the game, and can replace the default bonus questions with whatever questions they feel might be more helpful in their situation.

I acknowledge how there’s certainly a lot more I can do around exploring how the current version of the app does or does not meet various WCAG. I can hardly tout the importance of respecting many WCAG guidelines, if my own app does a poor job at respecting some of them. And I get how WCAG is aimed at web UI, but a lot of EN 301 549 is based on WCAG and is relevant to apps like mine. I’ll dig deeper into all this as I continue working on the app.

The code for the app is available at Grid Games code, but is only made available to demonstrate some coding steps relating to accessibility. It is not an MVVM app despite what the source file hierarchy suggests.

Over time, I’ll update the app to remove some of the constraints that it currently has today. Most importantly, as I get feedback from players on how the app can be made more usable and fun, I’ll do my best to update it accordingly.

The app’s available at Grid Games at the Microsoft Store. I’m looking forward to your feedback!

All the best,

Guy