The Sa11ytaire Experiment – Reacting to Feedback Part 1: Keyboard efficiency

This article describes how a solitaire game was updated to become more efficient for players who only use the keyboard to input at the game. The game’s available at Sa11ytaire at the Microsoft Store.

For some previous discussion on the accessibility of the game, visit Microsoft Windows UI Automation Blog.

Introduction

A few weeks ago I had an absolute blast! The week had started off with the Hackathon, where I got to experiment with a few exciting ideas around accessibility, and then on the Friday, I got to demo the Sa11ytaire app to a group of students. The demo wasn’t just a demo though, it was an opportunity for me to learn from the students. I wanted their thoughts on what changes should be made to the game to improve its accessibility. And goodness me, I got some great ideas from the students!

Since that day, Tim, (my Sa11ytaire co-developer,) and I have been considering the feedback from that day and making related updates to the app. I’ll be splitting the discussion on these updates across a couple of articles, and this first one relates to playing the game with a keyboard. I was already very pleased with how playable the game was with the keyboard, but the students’ ideas took this way further.

While there’s certainly more that could be done around these keyboard-related changes, we’ve reached a point where the updates are now available at the Store, and we can look forward to more feedback, and feel confident we’re always focusing on the most useful changes.

A short video demo’ing the new super-efficient keyboard interaction models is at Four efficient ways of playing the Sa11ytaire app with the keyboard.

A note on how efficient the game already was

By default, a player must be able to move keyboard focus to any and all interactable controls. And sure enough, through a mix of Tab key and Arrow key use, a player can move keyboard focus through all interactable controls in the Sa11ytaire app. Jolly good.

But simply being able to move keyboard focus to a control is often not sufficient. If a player is going to want to reach some control often, they’re not going to want to have to press Tab or Arrow many times to get there. That’s a time consuming and irritating process, and really won’t do. As such, the game needs to provide a more efficient mechanism to trigger the functionality associated with the control. And one way to enable that, is through use of access keys.

I love access keys. In many situations, implementation of access keys is one small step for an app dev, one giant leap in the usability of an app. And for a UWP XAML app, all it takes is to add a little markup to the control associated with the access key. For example, the Clubs pile in the Sa11ytaire app includes the following markup:

AccessKey="C"

As a result, all the player has to do to move a selected card over to the Clubs pile, is press Alt+C. Brilliant!

And it just so happens, when moving a card, and so pressing the access keys for two cards in turn, I don’t have to lift the Alt key after selecting the first card. So, to move a card from the second dealt card pile to the fifth, I’d do Alt+2+5.

So for players using the access keys, this seemed like a really efficient means of playing the game. It was so exciting for me to learn from the students how much further this could be taken.

Feedback from the students

This section describes three updates made to the app in the area of keyboard efficiency, made as a result of the students’ ideas.

Reduce the number of times that the Alt key must be pressed

While the existing access key functionality was efficient for some players, was there a way to reduce the number of times that the Alt key had to be pressed. For example, press the Alt key such that it became effectively “locked” in a depressed state, then press the access keys to make whatever moves are of interest in the game, and then take action to release the Alt key. Well, yes, that seems useful.

We wondered if perhaps the Sticky Keys functionality in Windows might be relevant here. Sticky Keys provides a way to have the press and release of a modifier key like Alt or Control, effectively lock the key in a depressed state. Then when the next non-modifier key is press and released, the resulting action is the same as if the two keys were pressed together. However, when the non-modifier key is released, the modifier key is moved out of its locked state, and that behavior doesn’t match what the student described. We need the Alt key to be effectively locked throughout multiple presses of the non-modifier keys.

So I was all set to add custom functionality to Sa11ytaire, such that a press and release of the Alt key could be considered to be a lock of the Alt key, if that’s what the player wants. But then I thought to myself, doesn’t the app already have functionality that’s somewhat relevant here?

Indeed! When I ported the app to Xbox a while ago, I wanted to make the game very efficient for some players using the Xbox controller’s chatpad. In that scenario, I didn’t want the player to be messing around with multiple tiny chatpad keys concurrently, and so enabled the player to turn on a mode whereby a press of a chatpad letter or number could trigger the same action as the related access key when playing the game on a desktop device with a keyboard. For example, “N” turns over the next card, or “U 3” moves the upturned card to the third dealt card pile.

Perhaps this existing functionality could be leveraged as part of reducing the number of times the Alt key is pressed on the desktop? Worth a try at least. So I added a way to toggle the availability of this functionality on the desktop, through a press of F9. It’s definitely worth noting that this isn’t implementing the functionality that the student described, as it’s not locking the Alt key. But perhaps it’s sufficient to meet the goals of reducing the number of key presses required to play the game, and certainly it means the feature’s available much sooner than it would be otherwise.

Note: There’s a growing amount of clean-up work required to be done in the app, both in the UI and in the code. The new functionality is only invokable through function keys, and I need to update the app to provide related options in the Settings UI. And we also need to add a Settings page. Today, all settings are jammed into the AppBar, and it’s a mess. As always, I hope to address all that “at some point”. (I must add a big thanks to Tim here, as he’s been much more attentive to keeping the code tidy than I’ve been.)

Figure 1: The Sa11ytaire app showing UI indicating that a press of the Alt key is not required when using an access key to interact with a card. The UI says, “Select card without Alt key.”

As it happens, I love playing the game with this new feature. Three key presses is an efficient way to move a card from one place to another. But not as efficient as two key presses!

Reduce the number of key presses required when specifying where a card should go

Most of the card-moving operations in Sa11ytaire are two-step actions. First the “source” card is indicated, and then the “destination” for the card. For example, the top card on the upturned card pile is to be moved to a target card pile, or a card in one of the dealt card piles is to be moved to another dealt card pile. So the students raised the idea of having an option to reduce the card-moving operations to be a one-step process. Interesting…

My first thought was to add a custom feature, such that in response to a particular key press, all available moves would be presented, and could be triggered with a single key press. Today, following a press of the Alt key, all cards can be selected in response to their associated access keys. But say instead of using Alt, in response to another key press, pop-up menus appear beside all interactable cards, each menu containing a list of unique keyboard presses associated with every available action. For example, say the four of clubs in dealt card pile two, can be moved to the five of diamonds in dealt card pile four, or to the five of hearts in dealt card pile seven. When the new “super-access” key is pressed, the popup menu appears beside the five of clubs, containing the two items representing the six of diamonds and six of hearts. Either of those actions could then be triggered through one more key press.

Now, while that is a really interesting idea, it would take some figuring out as to how the most intuitive experience could be delivered. What would be the unique keyboard action that triggers every possible move? (After all, in the example above, a four of spades might also be around somewhere and can be moved to the same cards as the four of clubs.) And once the player experience has been figured out, how long’s it going to take to implement and test all this? Technically, this seems all very doable, but quite time consuming to build. Creating and positioning custom visuals, which must support the active high contrast theme, could take a while. 

So I’ve taken a different approach which is hopefully sufficient to generate feedback, and then we can take things from there. The approach takes advantage of the following point: At any given time, if a card can be moved, it can usually only be moved to a single destination place.

As such, let’s add a new mode to the game, whereby when a card is selected, it will automatically move to the first available destination that the game finds, assuming any move is actually available. 

The action to turn over the next card is unaffected by this, given that there’s only ever one place that the next card can go. But for the card at the top of the upturned card pile and the cards in the dealt card piles, we first check whether there’s a move available to a target card pile. And then, if there’s no such move to a target card pile available, we check if there’s a move available to one of the dealt card piles, looking from the first dealt card pile and then through the other piles.

Constraint: Moving a card away from a target pile with a single key press is not supported. That’s only because I’ve not had time to add support for that, and such a move is relatively uncommon.

Constraint: This new mode does not provide a way for the player to move a card to a destination that’s not the first destination available as determined by the app. So if such a move is to be made, the player must turn this new mode off, make the desired move with the traditional two-step action, and then turn the move back on again.

The new mode is toggled with a press of the F8 key.

Figure 2: The Sa11ytaire app showing UI indicating that a press of a key to select a card will also move the card over to the first associated destination card found. The UI says, “Single key to move card”. 

While discussing this new feature with Tim, he pointed out that there’s some related functionality in the “Hint” feature that he added a while back. That feature includes searching through the dealt card pile lists, looking for available moves which can then be announced to the player. Wherever practical, it would be best to share code across features like this. That said, I’ve been terrible, and copied and pasted code like it’s going out of fashion. So, Tim’s right, there’s plenty of opportunity to clean the code in the interests of future app robustness and coding efficiency. 

As it happens, I love playing the game with this new feature in conjunction with the feature that avoids the need to press the Alt key. Two key presses is a really efficient way to move a card from one place to another. But not as efficient as one key press!

Remapping the keyboard shortcuts in the game

Now, I’m really excited about the efficiency enhancements described above. But what if it’s not efficient to reach the default access keys in the app? Those access keys are the numbers one to seven, and the letters ‘N’, ‘U’, ‘C’, ‘D’, ‘H’, and ‘S’. Those keys are contained across a huge physical area on the keyboard. Is it really necessary to require input across that huge area?

So the related question from the students was: Can the access keys be remapped?

And the answer was: No. But change is good, particularly when it comes to that answer. 

The result is that the app now shows a “Set access key mapping” button crammed into the AppBar. When invoked, a new “Set access keys to use in Sa11ytaire” dialog appears. With the new dialog, all thirteen access keys can be remapped to any other letter or number.

Figure 3: The new “Set access keys to use in Sa11ytaire” dialog, showing the app’s default access keys as being currently active.

Constraint: Only the access keys can be remapped today. The Function Key action in the app cannot be remapped. (Again, this constraint only exists due to the time available to work on the feature, and the priority of that work.)

Constraint: The dialog makes no checks for missing or duplicate access keys. (Again, this constraint only exists due to the fact that I’m a slacker.)

Even with the various constraints associated with the access key remapping feature, hopefully the current version of the feature will be sufficient for players to try the remapping and provide feedback.

An interesting issue which I’ve not investigated yet relates to the use of access keys in the dialog itself. I added an access key of ‘R’ to the “Restore defaults” button, as that seemed like a useful thing to do. (The Set and Cancel buttons have shortcuts of Enter and Cancel respectively.) But if I press Alt while the dialog’s up, the ‘R’ access key appears as expected, and all the access keys associated with the main (disabled) app UI in the background also appear – and are invokable. Yikes! That doesn’t seem right. Maybe I’ve botched something. Something to add to the list of things to investigate anyway.

Regarding the implementation of all this, the new access keys are persisted in local storage, using setting names such as things like “AccessKeyN”, “AccessKeyC”, and “AccessKey1”. When the app starts, it initializes a Dictionary with the mapping from the default values to the current custom values. In fact, most of the new code is simply there to support the tiny bit of code that actually controls the related player experience. And that is, to set the AccessKey property on the thirteen UWP XAML controls. By setting that value on a control, the control reacts to the new access key as required, and the UI Automation (UIA) AccessKey property is updated automatically for players who use screen readers.

For a very brief introduction into Windows UI Automation, visit Introduction to UIA.

Actually, there is one other place I needed to update the app. In order to support the feature which enables a card to be moved via its access key, but without the Alt key needing to be down, I had a big switch statement with cases such as “N’, “C”, “1”, etc. Those case statements needed to be updated to reflect the current access key remapping. 

The following picture shows the access key remapping dialog specifying a set of access keys that are located near the left-hand side of a keyboard. Boxes are superimposed over the keyboard in the picture to show the grouping of the keys related to the three main areas in the app.

Figure 4: The access key remapping dialog showing a set of access keys that are located near the left-hand side of a keyboard.

And to take this a step further, consider the seemingly unusual set of custom access keys shown in the picture below. In the picture, the thirteen access keys include slash, star, dot, and the numbers zero through nine in an apparent random allocation through the UI.

Figure 5: Access keys appearing by cards in the Sa11ytaire app.

Why would such access keys be interesting? Well, consider the external number pad shown in the image below. With the number pad connected, the groups of keys on the number pad can be used to access the three main groups of cards in the app. Boxes are superimposed over the number pad in the picture to show the grouping of the keys related to the three main areas in the app.

Figure 6: The access key remapping dialog showing a set of access keys that are located on an external number pad connected to the laptop.

Constraint: Well, this one’s more than a constraint. I realized after some testing, there’s some functionality missing from what I’d built. In order to play the game with the keyboard, the player uses all the access keys, and the up/down arrow keys. The arrow keys allow keyboard focus to move around a pile of dealt cards when multiple cards in the pile are face-up. So the access key remapping dialog needs to support remapping the arrow keys too. Perhaps in the case of the external number pad shown above, the arrow keys could be remapped to the Minus and Plus keys. (I’m also wondering about remapping the Enter key functionality too, when selecting a card.) Anyway, that’s an interesting topic to add to the list of things to consider once feedback comes in around the new features.

I did take my own action to have the text in the TextBoxes automatically selected when the TextBox gains keyboard focus. This seemed a preferable player experience. I couldn’t find a property on the TextBox related to this, so I subclassed the TextBoxes and call SelectAll() in their GotFocus() event handlers.

Given how the dialog is mostly a set of paired TextBlocks and TextBoxes, I should also mention how each TextBox uses “AutomationProperties.LabeledBy” to reference the associated TextBlock. For example, if the “Spades pile” TextBlock has a Name of “AccessKeySpadesPileLabel”, then the associated TextBox has the following markup:

AutomationProperties.LabeledBy="{Binding ElementName=AccessKeySpadesPileLabel}"

The above action means that screen readers make the appropriate announcements when keyboard focus moves into the TextBoxes.

Note that all the new features described in this article are also available on Xbox, using the Xbox controller’s chatpad. Once the right bumper is pressed, a card can be moved using its associated access key on the chatpad. A press of the left bumper toggles the option to move a card with a one-step action. And the access key remapping feature is also available.

As it happens, I love playing the game with this new access key remapping feature, in conjunction with the features relating to not requiring an Alt key press, and to moving a card with a single key press. One key press can be a really efficient way to move a card from one place to another. But not if reaching that key is a challenge.

Summary

All in all, I’m absolutely thrilled to have got the students’ help here. They provided me with some excellent ideas, and it was practical for me to implement related features. We’re now at a point where we can get more feedback to learn where things are working well, and where the features require more work before they’re useful in practice.

Because the new features are all optional, a player who uses the keyboard can pick the features that they find most useful. Maybe a player will leverage one or more of them, or perhaps none of them if the default access key experience is preferred.

And of course, Sa11ytaire can be played with far more that just the keyboard. When I met the students recently, I also demo’d speech control, eye control, and switch device control using the Xbox Adaptive Controller at a laptop. I am always absolutely blown away when I do this. To plug in an eye tracker into a laptop at a podium, rotate the tracker so that it notices me, and with my glasses own, I play the game. Just brilliant. What else can I say?

Anyway, keep that feedback coming, so over time we can make it more and more efficient for everyone to play a game of solitaire.

And for software designers and developers, keep in mind that efficient access isn’t only about playing games. It’s a core aspect of any quality product, be it related to communication, employment, or anything else. So always solicit feedback from a wide range of diverse people, such that a wide range of diverse input and output methods can be accounted for.

‘Til next time…

Guy