Mobile Testing – How The Numbers get Crazy

Mobile testing doesn’t sound too challenging, until you scratch the surface.

As I discussed in my recent post, to carry out proper mobile testing you need to use a test tool. Today we’re going to look at how the numbers get crazy when you’re mobile testing…

Mobile Testing is Not as Simple as it Sounds

At its most basic level, you open your app or website, run a manual test against it, and check the results. Job done? Not quite, in fact nowhere near. At least, not if your solution is designed to run on multiple devices.

Why There’s More to it than Meets the Eye

Ensuring proper Mobile Test coverage is not just a matter of repeating your test on the next device, and the next, and the next… 

• Did you test against different resolutions on the devices? Users have different settings depending on their preferences. Techies, for example, will always use the smallest font size available on any device. Those with less acute eyesight might opt for larger font and few icons. Did you test against these?
• What about network conditions or roaming? Have you tested your solution with lower bandwidths? Do you know where your functionality cutoff is? What will cause your solution to hang? How painful will that user experience be?
• What about power consumption and battery life? Phones will highlight any apps that are draining battery, and users will generally delete them – or worse leave a review.

All these contribute to user app ratings – It is tough to recover from bad ratings

How the Numbers get Crazy…

As you’ve seen above, Mobile testing even a simple solution will have so many combinations and permutations. Myriad device and operating systems combinations, with different user settings against different network conditions. Let’s look at how the numbers quickly spiral upwards:

How the numbers get Crazy - Stage 1

Let’s say it takes 30 minutes to run you’re a single test against a mobile device. This includes all the necessary testing overheads. You know the trivial stuff, like recording results and evidence capture. Let’s also say that you have 5 process-flows in your solution. So far we have 2.5 hours of effort. Not too bad.

Stage 1 running total: 2.5 hours

(5 tests x 30 mins)

 How the Numbers get Crazy - Stage 2

Of course, you want to test your solution against the top 10 devices. The Apple devices have a limited range of operating systems, but the Android devices open up a pandoras box. To keep it simple, and conservative, let’s say on average each phone can have only 2 different operating system configurations. So, 20 device/OS combos.

Stage 2 running total: 50 hours

(20 combos x 2.5 hours per test cycle)

How the Numbers get Crazy - Stage 3

Now, let’s introduce user settings. Let’s say there’s only 2 different resolution settings. Let’s also add in 2 different power settings – normal and power-saver. So that’s another 4 (2 x 2) variations to consider for ALL of your previous tests. 

Stage 3 running total: 200 hours

(4 user combos x 50 hours)

Yes, 200 Hours to Test 5 Simple Processes!

I could go further but I’m sure you get the point. Remember, this is a conservative estimate And you’ve only tested 5 process flows against 10 devices. We haven’t even looked at tablets, network conditions, roaming, different generations of devices, etc.. 

Plus, with more people now working from home, do you need to provide all these devices to each of your testers?

The Solution – Mobile Test Automation

Mobile solutions are evolving faster than any software before them. The pace of change is astounding. The ability to regularly release updates, patches and trials is absolutely key to a successful mobile solution – things just don’t stand still in the mobile space.

Do you have the bandwidth to run (at least) 200 hours of testing every time you want to try something new with your solution? 

Practically, and realistically, Mobile Test Automation is the only way to carry out Mobile testing. With the right solution, you can access as many devices as you need, you can run hundreds of tests covering all of your combinations and configurations.

Best of all you can run this overnight, as often as you like

Every night if you want. The results will be waiting for you in the morning and you’ll be on your way to a truly dynamic mobile solution.

Next Time – A Solution to Your Mobile Testing Challenges

In my next insight, I’m going to look at a specific Mobile Testing Solution that will address all of these challenges and transform your Mobile Testing…

Stephen Davis helps test professionals get best-of-breed tools that increase software testing productivity, consistency & coverage.

If you or anyone you know is looking for tools, or just interested in finding out what is new, please visit his website at www.calleosoftware.co.uk or email him at [email protected]