Is Accessibility for the Birds?

An old friend of mine recently told me how he’d built a popular app called BirdUp, which listens to bird calls and bird song, and uses AI built specifically for the app to identify the birds. Following the app’s mention on a BBC radio show for people who are blind or partially sighted, my friend learnt that there were some opportunities to improve the app’s accessibility. I’m always interested in exploring the accessibility of apps, and suggested that we port the app from its current Android-only codebase to .NET MAUI. By doing this we could not only concentrate on delivering the most accessible experiences that .NET MAUI technology provides, but also begin a journey of having the app run on both Android and iOS.

We’ve made great progress on the new app and hope to publish it in 2024, and we’re looking forward to learning from users how we can improve its accessibility further. Some aspects of the accessibility-related development of the app have been straightforward, but there are some very interesting questions which can only be answered by users of the app. For example, while the app’s listening to bird calls, what’s the ideal screen reader behavior during that time? Could a screen reader’s announcement impact the app’s attempt to identify the bird calls? We can’t wait to tune the app to make it as useable and enjoyable as possible for everyone!

How to port nearly 200 java files to C#

Typically whenever porting an app to a new platform, I’d take the opportunity to consider whether any significant design changes could help with the usability of the app. In the case of this app, we decided that the overall design would be kept as it is.

This meant that the first step for the app along its new journey would require converting all its java files to C#, and this was made practical thanks to Paul Irwin’s very handy Java to C# converter. After running all the existing Android app java code through the converter, it was then mostly a case of tweaking the converted code to compile and where necessary adjust things to run as expected using the various .NET Android API namespaces.

We decided that the first version of the new app will run only on Android, but by building it as a .NET MAUI app, it’ll have made a significant step towards also running on iOS.

Designing for all users

While the bulk of the new app’s codebase will be based on the existing app’s converted java files, its UI will be written from scratch, and this provides an opportunity to consider how the new UI can support as many users as possible. For example, the app will present colours appropriate to the user’s choice of the Android Light or Dark mode setting, and show text which respects the Android “Font size” setting without the text being clipped. While I do like the app’s Light mode colours, I’ve yet to try using the app at night in the garden, and wonder if I’ll be using the Dark mode colours then.

Accessibility Guidelines

A very helpful resource when considering the design of UI is the Web Content Accessibility Guidelines, and many of the topics included in the guidelines apply to any UI regardless of where the UI’s being displayed. The guidelines include the following Accessibility-related groupings.

Understandable

Presenting content that’s understandable does seem to be an area that the tech industry can sometimes struggle with today, yet this is a critical part of delivering a great experience for all users. I expect many of us have at one time or another sat in front of an app wondering exactly what the content being presented to us is intended to mean. That said, I have made certain assumptions around how familiar the user is with some bird-related and audio-related terms shown in the app. For example, when presenting terms such as: "tac" and "2-KHz".

I’m always conscious of how understandable icons are if they have no accompanying text, so in such cases I’ve tried to present icons that are very similar to those shown in many other apps for related functionality. For example, the "Delete", "Share", "Record" and "Stop recording" buttons.

Perceivable

Perceivable content is leveraged by all your users regardless of whether access to that content involves a screen reader, screen visuals, a braille device, or some other method. In support of that principle, the app provides a text-based description of bird calls known to the app, in addition to the associated bird call audio.

In addition, all text and icons shown in the app have a contrast against their backgrounds that meets the accessibility guidelines, regardless of whether the app is being shown with its light theme or dark theme colours.

Operable

Operable content is under full control of your users regardless of the input method being used, and regardless of the time it takes to interact with the content.

A very important access method available in Android is Voice Access, and because the app UI uses many standard controls provided by .NET MAUI, speech input can be used by default to interact with those controls.

Supporting users who use screen readers

The app uses semantic controls that come with .NET MAUI, and so goes a long way to delivering a usable experience by default for its users who use screen readers. For example, when presenting a button, the app uses a Button control rather than only rendering custom visuals that give the impression to some users that a button exists in the app.

In some cases, the app needs to provide additional information to support all its users. For example, some of the app’s buttons show only an image with no accompanying text, and this might result in a screen reader being unable to convey the true meaning of the button. The apps’ users mustn’t have to hope that either AI or a screen reader’s best guess can reliably determine the intent from the icon’s visuals.

The Build accessible apps with semantic properties resource describes how to set fixed (but still localisable) accessible names on controls for screen readers, such as the name of the button in the app that shows only a Settings icon. It also describes how to bind an accessible name on a control, such as the accessible name of an image that shows a specific bird in a list item. In addition, the resource describes how to set supplementary helpful information on controls, and to set a semantic heading level on text that is shown visually as a heading. The app takes advantage of all this very important support for users using screen readers.

Going back to the topic of design, every now and again I was faced with a choice of designs, and I wasn’t sure exactly what design would be most usable to users. This is where feedback from as many users as possible in the future will be helpful. For example, the app shows some media playback UI, and I typically don’t change the accessible names of buttons in an app, so how would I feel about having a button be named either “Play” or “Pause” depending on the situation? Fortunately for this particular question, I could benefit from Sarah Higley’s Playing with state article.

And considering again those helpful accessibility guidelines I mentioned earlier, other topics include Info and Relationships and Meaningful sequence, and .NET MAUI helps with both of those by having the hierarchy of such things as tabs and list items programmatically accessible, and supporting screen reader navigation that matches the logical path that I intended when building the UI.

Before the app is published, I know I’ll be focusing very closely on the screen reader announcements as the state of UI transitions in the app. For example, when moving between pages in the app, and when matches are found as the app listens to bird calls. I’m looking forward to learning whether I’ll end up introducing custom screen reader announcements in order to deliver a more helpful experience using SemanticScreenReader.

What are the most interesting challenges so far?

While the port of the Android app’s two hundred java files to working C# files in the .NET MAUI app has been fairly time-consuming, the bulk of the work has not been particularly challenging. Perhaps two more interesting challenges relate to the accessibility of the app’s charts and its real-time scrolling sonogram.

Charts

Accessible charts have historically been a rare thing, but at least some chart libraries are giving it some thought these days. I’m not aware of an accessible cross-platform chart suitable for plugging into a .NET MAUI app, so after viewing Gerald Versluis’s Visualize Your Data with Charts in .NET MAUI, I opted for presenting Microcharts cross-platform charts, and to accompany the charts with text-based equivalents of the data shown in the charts.

Sonogram

The app’s sonogram shows the frequencies of the sound heard by the app, and superimposes any matches to known sounds regardless of the app’s confidence in the match. Only when the confidence reaches a certain threshold does the match appear in a list below the sonogram. The sonogram scrolls in real-time to show its results, and also supports touch gestures for real-time panning and zooming.

I’ve made some progress on porting the original sonogram functionality to the new app, but at this time, I don’t have a feel for how the sonogram will be made fully accessible to all users. I expect this will be an iterative process, based on users’ feedback.

How you can help

All being well the new accessible .NET MAUI bird identification app will be published in 2024, and we’ll need your feedback to help us deliver the best experiences possible for all users. So whatever mix of access methods you use, such as voice access, screen visuals, or a screen reader, we’ll look forward to hearing from you.

Whether it’s a nuthatch or a linnet (or should that be “.NEThatch” or “.linNET”?) enjoyment and appreciation of birds is for everyone!