Mobile testing: So many devices, so many variables

by Joseph K. Clark

As organizations have shifted to a mobile-first approach, they’ve found that the main challenge is testing for various devices. Also, tests need to be run on multiple telecom providers in different parts of the world. This infinite web of possible combinations has prompted a need for automation and framework consolidation in mobile testing

Testing for Android devices is more challenging than iOS, according to Shawn Wallace, principal architect at Centric Consulting, a business consulting and technology solutions firm providing full-service management consulting services.

Android sits at the top at 73% market share, and iOS is second at 26%, a report by showed. But, what makes testing for Android complex is that so many more active devices use it with various sizes and form factors — not to mention the many different versions of the Android operating system still in use.

Because of this sheer number of devices, buying out all the appliances to manually test them by rote is no longer a feasible option for most organizations, explained Joachim Herschmann, a senior director analyst at Gartner. 

Buying everything is no longer the option for testing mobile web applications that run on a browser or natively on the phone. “Organizations no longer can afford to even, let’s say, buy the top 10 most relevant mobile devices. In the old days, if it’s an internal application for employees, they could say, well, there are only two or three mobile devices admitted in our organization,” Hirschmann said. “This will not work if we’re talking about an application rolled out to the public because you just don’t know what people are using.”


According to Centric’s Wallace, the drastic form factor changes are not the most difficult to test for; they can be the subtle changes between devices, such as a bezel taking up a little more space on one device than another. Sometimes, these subtle changes are the ones that can have a drastic effect on the crucial functionalities of the app. 

“The classic example is that some elements are partly off-screen, like a button or something you could potentially still hit, or maybe not. But the point is that that is not a good user experience, and in an extreme case, it could prevent you from actually interacting with it. I had this yesterday with an app I installed where one of their buttons would settle low at the bottom. But every time I tried to hit it with my finger, I hit the return on an Android device,” Hirschmann said. 

One way around this is to develop flexible layouts in apps, and cross-browser testing on laptops and computers can help avoid compatibility and sizing issues during the testing process, according to Artem Minaev, the co-founder of the digital startup platform FirstSiteGuide. This service provides tools and tutorials for bloggers, web admins, online entrepreneurs, and more. 

Adding to the already crowded device market, IoT devices also factor into conversations around mobile testing. “There’s a reason why we see so much news about not only security issues but simply just bugs because there is very little thought given about what’s happening on that device,” Gartner’s Herschmann said.

Smart devices expanding

With the expansion of smart devices, mobile testing even encompasses testing your car as a mobile device. “You could even consider the Tesla car itself as a mobile device because it’s at the end of this connectivity because it’s connected to the internet. You can set updates to your Tesla, so if you think about it that way, it becomes clear to ask yourself how we are testing the Tesla software,” Hirschmann said.

Another major challenge with mobile app testing is how a mobile app renders while using specific telecom providers and whether that’s through 5G, 4G, or 3G. “I mean …  if you have an iOS phone, how many times do you use Safari, and it stops responding per week? It probably does it three or four times a week. Well, let’s do this at scale. Let’s run 10,000 tests or 5,000 tests. How many times is our app going to fail? Not because the app is broken too, because Safari fails for whatever random reason,” Wallace said.

The sizable mobile app testing providers offer the ability to, for example, access a device in the U.K. through 4G, access it through an Indian operator with 3G, or access the same device in the United States through AT&T using 5G, Herschmann explained. “So technically, I’m accessible ng the same phone greatly, but the difference is how I access it. And I bet you will see differences in the test results because latency is different; the apps may react differently to the kind of network connectivity with latency, jitter, and stuff,” Hirschmann said. 

Another tricky aspect of testing is how an application behaves when switching between different providers, a situation prevalent in Europe due to its numerous providers in various countries. Other things to consider are when signal strength goes down or if the user enters a room with thick walls. 

In these scenarios, crowdsourcing providers, where people run software on their mobile phones to inspect for failures, come into play. Their input is vital for regression testing, feature testing, usability testing, user story testing, cross-browser testing, accessibility testing, performance testing, routine sanity testing, etc. 

Solutions to device sprawl

While manual testing is necessary for certain aspects, the difficulty with testing manually is that the number of tests needed to test the codebases increases exponentially whenever a feature is added. “If you’re adding a new feature, you need to test all of the old parts, which can slow you down,” Centric’s Wallace explained. 

“Two years ago, we talked about ruthless automation, just automating everything. Now, it’s about to let’s automate what we need to automate,” said Diego Lo Giudice, vice president and principal analyst at Forrester. “You don’t want to automate something that needs to run once. You want to automate repetitive things, but if you’re building Agile and DevOps, anything you build in sprint one becomes a regression test in sprint two. And so if you don’t want to rerun those tests manually, you’re better off automating them.”

The goal is to minimize the number of tests that have to be run on the user interface on the mobile application and isolate some of the business logic into the APIs or server-side code where there are great patterns for testing at scale and testing in a continuous integration environment, according to Centric’s Wallace.

Giudice said that automation tooling for mobile had reached the same level of maturity as it has for browsers. For browsers, the standard has been Selenium. Now, the measures that have grown around mobile testing include Appium, Cypress, and the ones maintained by Google and Apple for their applications on Android or iOS. 

Still, highly skilled test automation engineers must “write code” for automation test scripts, which has opened the need for low-code tooling for testing and new advancements in A.I. The future of A.I. in mobile testing can help teams know which tests to run in a complex environment and do it for them; however, this technology is still in its early stages, according to Hirschmann.

“The message I always give clients today looks like this A.I. stuff is not fully worked out; it’s the early days, but start looking at it today because you will take some time to understand and comprehend it. And so you can’t just sit back for another year or two and wait until someone else has figured that out; by that time, you’re lagging.” Hirschmann said.

While organizations should automate as much testing as possible, there is still no way to escape manual testing completely. “I know of cases where they literally have manual users in a car, and they drive from Germany to the Netherlands, and they have to switch from their German network to their Dutch network as they’re driving and testing; how does my application react to this? So as you can imagine, that’s not something a developer could test locally on their machine. You would have to do something like this,” Hirschmann said. 

Another major trend around simplifying mobile testing is consolidating a framework so that tests don’t need to be rewritten entirely when dealing with new operating systems or devices. Testing can be much more simplified when using one of the powerful frameworks by doing a lot of the testing before getting to the instrument because testing on the machine is painful, Centric’s Wallace explained. It can get quite expensive when an organization decides between native development or using a platform such as React Native.

Another way to do manual testing is by buying virtual machines, putting the emulator on them, and letting testers use a V.M. rather than a device. This is effective for testing offshore with people in other countries when they cannot install the app on their device or can’t have the data in their network. Still, they can connect to a V.M. because there’s no data transfer, Wallace explained. 

“The paradigms of how I build an application have changed. There’s still native-app development, native in the sense that I write specifically with the widgets or elements on iOS and those on Android, but this becomes less frequent,” Gartner’s Herschmann said. “The more frequent approach is now to use one of the common frameworks that would take what you write in one abstract framework layer that gets sort of translated and compiled into the core platform that it’s rendered on.”

Also, the shift to using a public device cloud, which contains a couple of thousand devices for anyone to test on, is pervasive now. Whenever the next person wants to run their test, the devices are reset to prevent interference with any data. 

To a lesser extent, some organizations work within a private device cloud. The provisioning and administration mechanisms work the same as in a public cloud, but the devices are still reserved for that organization. 

The third and rarest option by far are the cases where there’s a device rack somewhere in a server room, but the way it’s accessed is also through the cloud, according to Gartner’s Herschmann.

“Over the years, it had shifted to the public cloud because the private option was much more popular a couple of years back when people didn’t trust it so much,” Hirschmann said. “Let’s say you’re a consumer application or a retail website. Well, you want to put it out in the open anyhow. So there is no reason why you would want to do it in your closet and not go out.”

Cloud device providers often have contracts with various device and telecom providers to have day-one availability or even give developers access to the technology sometime before to run tests before launch day. 

Simulators and emulators can also be used for more operating system features, such as Google’s new API for COVID, which isn’t very hardware hungry. On the other hand, you have to test the device for non-standard uses of the camera, LiDAR, or other features specific to the hardware.

Mobile testing is vital in a mobile-first world.

Mobile app development has been integrated into mainstream products, which has increased the importance of diligent mobile testing. The same testing scripts can be used for desktop or mobile, prompting consolidated testing. 

“We’ve gone from a world where people thought we were going to be mobile-first or mobile-only 5 or 6 years ago, but it ended up being mobile-first because mobile-only is not a reality,” Forrester’s Lo Giudice said. 

So mobile has to be integrated into a broader omnichannel strategy because these applications that run on mobile also have to run on a desktop or a tablet. The pandemic’s progress to mobile-only was mainly stifled since people worked in front of their computers at home. Before, companies tried to push as much as possible onto mobile phones.

“I think the assumption that everything would go onto a mobile phone was just wrong,” Lo Giudice said. “I mean, the web is still well, but what changed is that if I have to design an application, I will think that design for the mobile phone, for the smaller screens and then scale it eventually to the bigger ones, rather than five years ago, the apps were thought for a large screen. And then we had to squeeze them onto the mobile phones.”

This mobile-first approach has led organizations to hire specialists that can test for omnichannel applications. Organizations are no longer just looking to hire specialists that only write tests for the current iOS version, for example, but that write tests against a standard so that new versions don’t require them to change the test code beyond some compatibility issues, Centric’s Wallace explained. 

“I think a lot about what the person that’s going to maintain this, what their experience is going to be like, and what the shelf life of these tests are going to be because you maintain an app for a much longer time than you spend building it,” Wallace said. “When we don’t run tests, our clients test our software. Software is always tested. We like to test it closer to when the code was written,” Wallace added. 

When establishing a testing strategy, organizations first have to think about the tech stack they’ll be testing for and who will try, whether that’s the developer or the technical side of the business. There are solutions out there that can tailor to each type of tester, according to Gartner’s Herschmann.

“It’s no longer that hard to write a mobile app. Anybody can do it basically with no-code tools, but building the right app is all about building an app useful for potential consumers,” said Herschmann. “So you have to have a compelling value proposition to get people interested. And then don’t screw it by bringing up a choppy version of that app because that will kill it immediately. That’s why I think the testing quality is so important because you’ve only got one chance today.”

Popular open-source testing frameworks for mobile

This partial listing of popular open-source testing frameworks for mobile was taken from a compilation by SmartBear, including commercial products. SmartBear provides tools for application performance monitoring (APM), software development, software testing, API testing, and API management.

  1. Appium is an open-source test automation framework for use with native, hybrid, and mobile web apps. It supports cross-platform testing and code reuse, writing tests with multiple programming languages, and many integrations. It drives iOS, Android, and Windows apps using the WebDriver protocol.
  2. Espresso is a native open-source testing framework for Android to write U.I. tests. The tool is a part of the Android SDK and is easy to use for native mobile development. It offers automatic synchronization of U.I. elements and test actions.
  3. XCUITest is an open-source testing framework for iOS apps developed and maintained by Apple and is ideal for iOS-specific U.I. test automation. XCUITest enables developers or Q.A. engineers to test the user interface of native iOS apps using the Swift or Objective C programming languages.
  4. Robot Framework is a generic open-source automation framework for test and robotic process automation (RPA). The framework was initially developed at Nokia Networks and open-sourced in 2008. It has easy syntax, utilizing human-readable keywords. Its capabilities can be extended by libraries implemented with Python or Java.

Real users for accurate testing results

While automation is a vital tool in many aspects of mobile testing, it has to be coupled with manual testing for applications that rely heavily on location, as in the case of Bloc. This social events app rewards people for attending venues.

“We suggest venues that are local to someone, and a lot of the app’s functionality relies on people being in the same area. This makes it extremely difficult to outsource the testing to a third party because their testers will be dotted around the world,” said Josh Wood, the founder, and CEO of Bloc. 

The company initially set up simulators and faked people’s location, but it wasn’t a comprehensive enough strategy. Then, Bloc had to find and onboard its testers, which became problematic when scaling to thousands of users. 

The company relies on internal tests from real users who don’t necessarily know they’re testing the app. “The best feedback and log files we’ve got are from real users using it in real-time, and that’s how we’ve created a pretty decent app today that works well,” Wood added.

Related Posts