How to choose the right mobile test devices

In addition to all our mobile test automation, we also do a lot of manual and exploratory testing within our mobile team. We test devices with different hard- and software, with different browser versions, and on different carrier networks to be sure that our apps work in the way our customers will use it.

Figure 1 illustrates some of our test devices during testing on all mobile platforms. Due to the fact that the mobile market is growing, the key question for a mobile Quality Assurance person is:

Which devices are right for testing? It’s simply not possible to do testing on every device!

Fragmentation only on Android?

I bet every time people hear the word mobile fragmentation, they always think this only happens in the Android world! Yes, the Android world has a really big challenge due to fragmentation of different vendors with different hardware and software installations as well as customized user interfaces. However, if you look at iOS there is some fragmentation as well. Sure, you have just one vendor, but also different software versions and devices. The range of touch devices starts with the iPod touch, the iPhone classic, the iPhone 3G/ 3GS/ 4/ 4S, the iPad/ iPad2 and the new iPad. All of the devices also have differing hardware specifications and can run different iOS versions. So as you can see, the fragmentation issue also exists in the Apple iOS world.

If you think about mobile websites there is another challenge that can be really painful: the various mobile browsers!

There are browsers likes Safari on iOS, Opera Mini for iOS and Android, Dolphin Browser for iOS and Android, the native Android browsers, Google Chrome for Android, Firefox for Android, IE 9 mobile browser on Windows Phones, the BlackBerry browsers and some other feature phone browsers.

So what is the right choice for testing? Just use the latest devices and browser versions? Buy every device on the market?

No, this can’t be the solution because not every user will upgrade their device or software every time a new version is available. The “typical” user normally buys a mobile device with a 2-year contract and will stay with that smartphone for at least that period of time.

So the best way to choose the right devices, operating systems and browser version is to know your customers! If you know which devices and software versions they use, you can downsize the development and testing effort required on mobile devices. Otherwise it’s very hard (impossible, actually) to test your apps on nearly every device on the market! Sure you can use external testing services like utest or apkudo, but maybe you’ll end up spending a lot of money to test devices your customers don’t use!

User statistics – You’d better know your mobile users

We at the XING mobile team know which devices and software version our users have. A good starting point when it comes to getting information about your customers is the app store of the specific vendor (Apple App Store, Google Play Store, Windows Marketplace).

If you already have an app in one of the stores, you will receive information about the devices, software versions, languages and carriers used by your customers. As an example, see Figure 2 below from the Google Play Store which shows statistics from our XING Android App.

For mobile web pages there is no app store to provide user data. Besides the market statistics, we also have information about the ‘user_agents’ (devices, browsers) used within the mobile website. With this data from the apps and the mobile web app we’re now able to setup a mobile test infrastructure for developing and testing purposes.

Test devices

We split our test devices running Android, iPhone and touch.xing.com into three different groups. Group 1 represents older phones with a less powerful CPU, memory (RAM), and a low screen resolution. The fact that we only have a few users with that kind of device group 1 is a low priority: C. Group 2 covers average devices with a good single-core CPU (1 GHz) and with memory of around 512MB. These devices mostly have a screen resolution of 480×800 pixels (HDPI). This group is graded with medium priority: B. The third group represents high-end devices with at least a dual-core CPU and memory >512MB. Most of these devices have also a big screen with a really good screen resolution (XHDPI). This group has the highest priority: A.

Besides the hardware specifications the groups also cover the software versions (operating systems, browser versions).

Overview of the group categories:

• Group 1, priority C: Small devices with less CPU, RAM and low resolution. Older software versions
• Group 2, priority B: Mid-range devices with avg. CPU, RAM, good screen size and resolution. The software is not the latest.
• Group 3, priority A: High-end devices with a dual-core CPU, RAM (>512MB) and a high screen resolution. Latest software versions.

These three groups cover most of our users and represent other phones on the market that fit into the same group.

Android

For the 1st group running Android we found out that the HTC Legend with Android 2.2 fits into this category. It only has a small CPU (600MHz), just 384MB of memory, a small screen and low screen resolution (MDPI). The HTC Legend runs Android 2.2 which is the lowest Android version we support.

For the 2nd group we chose:

• HTC Desire with Android 2.2
• Samsung Galaxy S with Android 2.3
• Sony Ericsson Arc with Android 2.3

For the 3rd group we chose:

• Samsung Google Nexus with Android 4.0.4
• HTC Sensation with Android 4.0.3
• Samsung Galaxy S2 with Android 2.3.4
• Motorola Razr Droid with Android 2.3.5
• Samsung Galaxy Note with Android 2.3.6
• LG P990 with Android 2.2

iPhone

We also performed the same grouping for iPhone devices and came up with the following three groups:

Group 1, slow iPhones with iOS versions from 4.0:

• iPhone 3G iOS 4.0
• iPhone 3G iOS 4.2.1

Group 2, iPhone 3GS with different iOS versions:

• iPhone 3GS iOS 4.3
• iPhone 3GS iOS 5.0

Group 3, latest iPhones with different iOS versions:

• iPhone 4 iOS 4.3
• iPhone 4 iOS 5.0.1
• iPhone 4 iOS 5.1
• iPhone 4S iOS 5.0.1
• iPhone 4S iOS 5.1

We have around 10 devices we use for manual and automated testing of the native iPhone and Android apps. Each new feature we develop is tested with at least one device from group 1 with priority C. When it comes to groups 2 (B) and 3 (A), we used a lot more devices for testing. For manual testing we avoid using mobile simulators in order to have a real testing environment.

touch.xing.com

Based on the data we know from touch.xing.com, we also have three groups of web browsers that should be used for testing on real devices. Most of our touch.xing.com users are iPhone users, while the second most-used platform is Android.
With that knowledge we created the following groups for mobile browsers:

Group 1, non-iPhone and Android Devices:

• BlackBerry Bold with Native Browser (Webkit)
• Nokia N8 with Native Browser (Webkit)

Group 2, a mixture of iPhone and Android Devices with different browsers:

• iPhone 3GS with Safari (Webkit)
• Samsung Galaxy S with Native Browser (Webkit)
• HTC HD with Dolphin Browser (Webkit)

Group 3, latest iPhone and Android devices with latest browser versions:

• iPhone 4S with Safari (Webkit)
• iPhone 4S with Dolphin Browser (Webkit)
• iPhone 4 with Safari (Webkit)
• Samsung Galaxy S2 with Native Browser (Webkit)
• Samsung Galaxy Nexus with Google Chrome (Webkit)
• HTC Sensation with Firefox (Gecko)

By creating priority groups for our test devices and browsers, we have managed to downsize the effort required during developing and testing to make our daily work more efficient. All of the groups are updated from time to time with new devices, software and browser versions.

An overview of our mobile apps can be found here.

If you want more information about our mobile test automation please read the following blog posts.

• Robotium @ XING. Automated regression tests on mobile Android devices
• Keep It Functional – iPhone Test Automation

About the author

2 thoughts on “How to choose the right mobile test devices”

1. Great Article; its a very good complementary of your presentation at STUGHH, especially the thoughts behind the grouping and actual examples and names.

   Keep up the good work.
   MaikNog

2. Nice Article. I being a part of mobility company (http://www.techendeavour.com) do understand the challenges of testing in mobile and the most important part of this strategy is to identify the devices on which to validate the application. This in itself ensure 60% of the success of your solution in the marketplace. Well written.