Mobile Application Testing

Device and platform diversity, short release cycles, lack of mature testing tools and the variety of network connectivity options result in frequent cost overruns and missed deadlines in today’s mobile application testing environment. A comprehensive mobile testing strategy that includes device and network infrastructure, optimized selection of target devices, and an effective combination of manual and automated testing tools to cover both functional and non-functional testing is essential for getting your mobile applications to market on time and within budget.

I’d like to take you through the thought process of testers and discuss the types of things they consider when testing a mobile app.

Testers Ask Questions

At the heart of testing is the capability to ask challenging and relevant questions. You are on your way to becoming a good tester if you combine investigative and questioning skills with knowledge of technology and products.

For example, testers might ask:

• What platforms should this product work on?
• What is the app supposed to do?
• What happens if I do this?

And so forth.

Where To Start Testing

In an ideal world, testers would all have up-to-date details on what is being built. In the real world, this is rare. So, like everyone else, testers make do with what they have. Don’t let this be an excuse not to test! Information used for testing can be gathered from many different sources, internally and externally.

At this stage questions, testers might ask these questions:

• What information exists? Specifications? Project conversations? User documentation? Knowledgeable team members? Could the support forum or an online company forum be of help? Is there a log of existing bugs?
• What OS, platform and device should this app work on and be tested on?
• What kind of data is processed by the application (i.e. personal, credit cards, etc.)?
• Does the application integrate with external applications (APIs, data sources)?
• Does the app work with certain mobile browsers?
• What do existing customers say about the product?
• How much time is available for testing?
• What priorities and risks are there?
• Who is experiencing pain, and why?
• How are releases or updates made?

Based on the information gathered, testers can put together a plan on how to approach the testing. Budgets often determine how testing is approached. You would certainly approach testing differently if you had one day instead of a week or a month. Predicting outcomes gets much easier as you come to understand the team, its processes and the answers to many of these types of questions.

EXAMPLE: SOCIAL COMMENTARY ON THE FACEBOOK APP

I love using the Facebook app as an example when I’m gathering information as a tester. Complaints of it are everywhere. Just check out the comments in the iTunes App Store for some of the frustrations users are facing. Plenty more are dotted across the Web.

The Creativity Of Testers

You probably know what the app is meant to do, but whatcan it do? And how will people actually use it? Testers are great at thinking outside of the box, trying out different things, asking “What if” and “Why” constantly.

For example, mobile testers will often adopt the mindset of different types of people — not literally, of course, but the ability to think, analyze and visualize themselves as different users can be quite enlightening.

Testers might put themselves in these shoes:

• Novice user,
• Experienced user,
• Fan,
• Hacker,
• Competitor.

Many more personalities could be adopted; much of this really depends on what you are building. But it’s not just about personalities, but about behavior and workflows, too. People use products in strange ways. For example, they:

• Go back when they are not supposed to,
• Are impatient and hit keys multiple times,
• Enter incorrect data,
• Can’t figure out how to do something,
• Might not have the required setup,
• Might assume they know what they are doing (neglecting to read instructions, for example).

Testers look for these situations, often discovering unexpected results along the way. Sometimes the bugs initially found can appear small and insignificant, whereupondeeper investigation uncovers bigger problems.

Many of these issues can be identified up front with testing. When it comes to testing mobile apps, these might not all be relevant, but perhaps try asking questions such as these:

• Does it do what it says on the tin?
• Does the app perform the tasks it was designed to do?
• Does the app perform tasks that it wasn’t designed to do?
• How does the app perform when being used consistently or under a load? Is it sluggish? Does it crash? Does it update? Does it give feedback?
• Do crash reports give clues about the app?
• How can one navigate creatively, logically or negatively around the app?
• Does the user trust your brand?
• How secure is the user’s data?
• Is it possible to break or hack the app?
• What happens when you push the app to its limits?
• Does the app ask to turn on related services? (e.g. GPS, Wifi)? What if the user does? Or doesn’t?
• Where does the app redirect me? To the website? From website to app? Does it cause problems?
• Is communication and marketing consistent with the app’s function, design and content?
• What is the sign-up process like? Can it be done on the app? On a website?
• Does sign-up integrate with other services such as Facebook and Twitter?

There is no quick way to identify issues easily. Every app and team faces different challenges. However, one defining characteristic of testers is that they want to go beyond the limits, do the unusual, change things around, test over a long period of time — days, weeks or months instead of minutes — do what they have been told is not possible.

Where’s All The Data?

Testers like to have fun with data, sometimes to the frustration of developers. The reality is that confusing either the user or the software can be easy in the flow of information. This is ever more important with data- and cloud-based services; there is so much room for errors to occur.

Perhaps you could try checking out what happens in the following scenarios:

• The mobile device is full of data.
• The tester removes all of the data.
• The tester deletes the app. What happens to the data?
• The tester deletes then reinstalls the app.
• Too much or too little content causes the design or layout to change.
• Working with different times and time zones.
• Data does not sync.
• Syncing is interrupted.
• Data updates affect other services (such as websites and cloud services).
• Data is processed rapidly or in large amounts.
• Invalid data is used.

For example, they might try to do the following:

• Test the limits of user input,
• Play around with duplicate data,
• Test on brand new clean phone,
• Test on an old phone,
• Pre-populate the app with different types of data,
• Consider crowd-sourcing the testing,
• Automate some tests,
• Stress the app with some unexpected data to see how it copes,
• Analyze how information and data affects the user experience,
• Always question whether what they see is correct,

Creating Errors And Messages

Errors and messages are such common places for testers to find problems.

QUESTIONS TO ASK ABOUT ERROR MESSAGES

Consider the following questions:

• Is the UI for errors acceptable?
• Are error messages accessible?
• Are error messages consistent?
• Are they helpful?
• Is the content appropriate?
• Do errors adhere to good practices and standards?
• Are the error messages security-conscious?
• Are logs and crashes accessible to user and developer?
• Have all errors been produced in testing?
• What state is the user left in after an error message?
• Have no errors appeared when they should have?

Platform-Specific Considerations

Becoming knowledgeable about the business, technology and design constraints of relevant platforms is crucial for any project team member.

So, what types of bugs do testers look for in mobile apps?

• Does it follow the design guidelines for that particular platform?
• How does the design compare with designs by competitors and in the industry?
• Does the product work with peripherals?
• Does the touch screen support gestures (tap, double-tap, touch and hold, drag, shake, pinch, flick, swipe)?
• Is the app accessible?
• What happens when you change the orientation of the device?
• Does it make use of mapping and GPS?
• Is there a user guide?
• Is the email workflow user-friendly?
• Does the app work smoothly when sharing through social networks? Does it integrate with other social apps or websites?
• Does the app behave properly when the user is multitasking and switching between apps?
• Does the app update with a time stamp when the user pulls to refresh?
• What are the app’s default settings? Have they been adjusted?
• Does audio make a difference?

Connectivity Issues And Interruption

Funny things can happen when connections go up and down or you get interrupted unexpectedly.

Have you tried using the app in the following situations:

• Moving about?
• With Wi-Fi connectivity?
• Without Wi-Fi?
• On 3G?
• With intermittent connectivity?
• Set to airplane mode?
• When a phone call comes in?
• While receiving a text message?
• When receiving an app notification?
• With low or no battery life?
• When the app forces an update?
• When receiving a voicemail?

These types of tests are a breeding ground for errors and bugs. I highly recommend testing your app in these conditions — not just starting it up and checking to see that it works, but going through some user workflows and forcing connectivity and interruptions at particular intervals.

• Does the app provide adequate feedback?
• Does data get transmitted knowingly?
• Does it grind to a halt and then crash?
• What happens when the app is open?
• What happens midway through a task?
• Is it possible to lose your work?
• Can you ignore a notification? What happens?
• Can you respond to a notification? What happens?
• Is any (error) messaging appropriate when something goes wrong?
• What happens if your log-in expires or times out?