What to Test When Testing a Mobile App

Mobile app testing is a hot topic. The number of downloadable apps is barely comprehensible ( Android Apps: 1,5 million apps; Apple’s App store: 1,4 million apps ) and the device fragmentation is nearly beyond imagination ( 24,093 Distinct Android devices seen up until August 2015). The only way to develop apps is to test on as many devices as possible through mobile app testing.

This post will focus on what you have to test when testing a mobile app. If you don’t know how important mobile app testing is, read this article first.

The first thing to understand when engaging with mobile app testing is that there is no easy way and no standard way of doing things, even though apps have standard requirements.

The standard requirements that need to be tested are:

• Usability
• Functionality
• Performance
• Edge Cases

Other important elements when testing mobile apps are the target devices (the devices you choose to test your app on) which should be chosen wisely, must be at least 20 and they should include, based on territorial coverage, the most used screen sizes and OSs platforms and versions. So, as you can see, there is a lot of work to do before actually start testing. We put together regularly the top 20 devices of different countries based on their territorial coverage and you can find them here.

When you actually start testing you need approximately 50 use cases for each OS platform (Android and iOS). Use cases test the above standard requirements, in other words, they test what users might do when using an app. They need to be extensive to find as many bugs as possibile to keep users happy.

Different tests are run during different life-cycle stages of an app. Tests can be run manually or automated depending on the performed test. Different tests are performed by different departments (development, QA and production), depending on the app’s life-cycle stage.

As mentioned before, there are mainly two types of approaches: the manual approach (manual testing) and the UI automation approach (automated testing).

The manual approach is used for Checkups, Usability, Edge Cases and Performance, but it’s not meant for repetitive tests. There are pros and cons to this approach:

• Pros: The actual behaviour gets tested
• Cons: It’s time consuming and inefficient

Development is an ongoing process, there is no start or finish when talking about app development, but continuous testing for continuous improvement that result in continuous delivery: it’s the never ending cycle of continuous integration!

So let’s think about having new builds or releases each week and every minor improvement needs to be tested. Those minor changes result in 100 different use cases (50 for Android and 50 for iOS) that need to be tested on 20 different devices. It’s about 2000 different combinations to test each week… each week! It’s impossibile to do this manually: the answer to this would be the UI automation approach.

The UI automation approach is used for Regression, End-to-end and Integration tests. Test scripts need to interact with your app and the tests need to be black box tests: you test exactely what the users will see. The test will check if the behaviour and the rendering of the app is what was expected. The regression test is very important: it checks if something that was working broke with recent changes. But as you can’t do everything manually, you can’t automated everything either.

The pros and cons to this approach are:

Pros:

• High test coverage
• Scalable
• Time-to-market
• Releases resources for manual testing

Cons

• Initial investment
• Requires technical know-how
• Maintaining of scripts
• No edge cases

Test automation is an extensive topic and we have already covered a few important pieces that will complete your mobile app testing puzzle: “Top Benefits of Automated Testing“, “Automated UI Testing Frameworks” for your automated tests and “How To Integrate Your Infrastructure with Jenkins for Powerful Testing” if you are using or decide to use Jenkins as your CI server.