Mobile App Testing

We’ve spent the last year (253 posts for those keeping score) explaining to the world that mobile app testing is an entirely different animal than its web and desktop cousins. Whether the differences be in terms of OS, browser, screen size or GUI – you name it, we’ve covered it.

Yet this concept is….well, just a concept…until it’s experienced first-hand. Matt Heusser, one of the very best testing writers out there, recently wrote about his experience with mobile app testing for SearchSoftwareQuality, where he covers screen-size discrepancy, the device matrix, GUI and other areas we all know and love.

I was particularly drawn to the “ah-ha” moment in the second paragraph (emphasis added). Take a look:

So there I was, on my iPod Touch, trying to get to a list of users whose name started with the letter “I.” It worked great on the simulator with a mouse, but with the actual iPod, my finger was too fat to click the single line of pixels.

Suddenly it hit me: This is different. Sure, all of the old GUI rules apply, but suddenly we have a new set of ways the application can fail. This tip provides a quick set of guidelines to consider, primarily for Web-based mobile applications, but much of it applies to native applications as well.

Screen real estate

You might use a mobile device just like a regular 1024×768 pixel application, but your users probably won’t. Try to actually use the application on a number of devices — just use it. You’ll likely come away suggesting a mobile interface, perhaps an automatic re-direct on login when your application senses a mobile device. Even then, you’ll want to explore the application in a number of devices, looking for usability problems.

Read more…

I just posted a Testing the Limits interview with Jakob Nielsen – aka the King of Usability – over on the uTest Blog. The subject matter will of great interest to readers of this blog, as we had an in-depth discussion on the past, present and future of mobile applications. Here are a few clips where he discusses native apps vs. the mobile web, tablet usability issues and his take on the iPhone vs. Android situation.

On native apps vs. the mobile web:

JN: Apps are superior for 3 reasons:

• Empirically, users perform better with apps than with mobile sites in user testing.
• Apps are much better at supporting disconnected use and poor connectivity, both of which will continue to be important use cases for years to come. When I’m in London and don’t feel like being robbed by “roaming” fees, any native mapping app will beat Google Maps at getting me to the British Museum.
• Apps can be optimized for the specific hardware on each device. This will become more important in the future, as we get a broader range of devices.

Apps have the obvious downside of requiring more development resources, especially to be truly optimized for each device. If a company doesn’t have enough resources to do this right, it’s better to have a nice mobile site than a lame app.

A second downside of apps is that users have to install them. Our testing shows poor findability and usability in Apple’s Application Store, and many users won’t even bother downloading something at all for intermittent use. So ask yourself whether you’re really offering something within the hardcore mobile center of need: time-sensitive and/or location dependent, and whether your offer is truly compelling in this crowded space. Most companies are never going to make it big in mobile. In some cases all they need is to make their main website somewhat mobile-friendly. Many others should deliver a dedicated mobile site but not bother with apps.

On usability problems with tablets:

Read more…

A reader alerted us to an exciting new mobile product last week – VeriShow Mobile for Android. Designed for professionals who are frequently in and out of the office (you know the types), VeriShow Mobile lets users participate in web meetings remotely with real-time collaboration. A product with much promise, to be sure. Here are a few of the top-line features from their website:

• View session content: files, PDFs, Web pages, shared screens & more
• See annotations and changes made to the document in real time
• View the list of session participants
• Exchange text chats with all participants
• Free!Download in just a few minutes
• Very easy to use
• Works on the Android devices. Coming soon: BlackBerry Playbook, iPhone and iPad (second quarter 2011)

Naturally, I wanted to learn more about how such a product was tested. Mobile testing is a challenge in and of itself – throw a real-time component and things really start to get complicated. Here’s what a rep from the company’s R&D team had to say with regards to testing (emphasis mine):

“Testing for VeriShow mobile was completed by the R&D team of HBR Labs. Given this was our first mobile app, the testing was extensive. Our biggest limitation was screen size. As the platform was originally built to run on personal computers and laptops, the screen size of a smart phone is significantly smaller than this. Thus we had to take into account the functionality of some of our apps on such a small screen. Another issue was the android enabled tablet devices, which has larger screens then the smart phones. We had to build an app that performed well on both smart phones and tablets. This was our biggest challenge. In testing we had users of all three devices (smart phone, tablet, and pc) enter into a collaborative session and we were more than satisfied with the results. We also provided some of our investors with beta versions of the app, who provided us with useful feedback.”

Read more…

The following post was written by mobile app tester Chris Polkamp as part of uTest’s “Crash Courses” series. You can read similar posts in the uTest Forums (membership required).

Background: In these days there is a wide range of software, this applies to desktop software, but mainly for web and mobile apps. At the time this course was written there are about 300,000 iPhone apps and 200,000 Android apps. My opinion and experience is that the performance and usability of an app can make the difference. I personally prefer a quick and easy app with less functionality than a slow and difficult application with many features.

Therefore, it is important for app developers and testers to focus on usability from the very beginning. An app that is is easy-to-use and intuitive, and similar to industry-accepted interfaces will tend to do well.

Usability for mobile apps is not the same for desktop apps: For mobile apps especially the input method, screen size and connection speed differs.

In this course, I will mention guidelines for developing and testing Android apps, which can also be used when testing the usability of an Android app. In the last item I will write some tips and examples to test an Android app.

These are ten general principles for user interface design, from Jakob Nielsen. Keep it in mind while testing usability (or print it to keep it on your desk) and it will help you a lot.

• Visibility of system status
• Match between system and the real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognize, diagnose, and recover from errors
• Help and documentation

Read more…

Though it was written with developers and designers in mind, Mashable’s “5 Things to Consider When Designing Your Mobile App” reads more like a how-to guide for mobile application testers – covering the basics of native apps vs. mobile web, screen size, user location, UI intuitiveness and other common design issues.

Of course, when these issues are not dealt with properly on the developer/designer side, they eventually become the problem of mobile test engineers. So with that in mind, let’s take a quick look at their five things to consider:

1. Weigh the Options — Mobile App or Mobile Website?

“Do you need a mobile application, a mobile website, or both? Before even starting the design process, you need to figure out what format or formats are best suited for your goals. Sometimes this can be really easy. For example, if you want to build a utility or game, you may be better served building a native application rather than worrying about how different mobile browsers will interpret your content.”

2. Consider Where Your App Will Be Used

“Once you’ve decided to make a native mobile app, you’ll want to consider where your application is most likely going to be used. This is important because where and how an application is used can directly impact how it can be designed.

For instance, if you have an application that is going to be used while walking around — a geo-location app or an app that takes advantage of a device’s GPS — making sure that core app functions are easy to see and access is very important.”

Read more…

A few weeks ago, when I shared my thoughts on the iPad, I noted that while the iPad will run iPhone apps, they probably wouldn’t look that great. Instead, developers would have to create new iPad apps.

“That’s fine!” you exclaim, thinking that you’ll just uprez your widgets and artwork from your iPhone app to the new iPad screen size. Problem solved, right? Apparently Apple thought so too and tried creating iPad sized versions of their default iPhone apps. And apparently that idea sucked. From Daring Fireball:

It’s not that Apple couldn’t just create bigger versions of these apps and have them run on the iPad. It wasn’t a technical problem, it was a design problem. There were, internally to Apple (of course), versions of these apps (or at least some of them) with upscaled iPad-sized graphics, but otherwise the same UI and layout as the iPhone versions. Ends up that just blowing up iPhone apps to fill the iPad screen looks and feels weird, even if you use higher-resolution graphics so that nothing looks pixelated. So they were scrapped by you-know-who.

Think this is just an Apple problem? No, it’s a mobile device problem!

Desktop and web app developers have it easy. Most computer screens are large, and any variation in size can usually be glossed over by either the OS or web browser. Nobody really uses computer screens smaller than 640×480, while many people now have 48″ screens that leave HD in the dust.

Mobile is a totally different ball game. Apps are modal, meaning your app has to account for all of the screen real estate. If it’s too big or too small, it will either fail to display or display incorrectly. Even Engadget recently lamented how they had trouble getting apps to run on the Nexus One:

Read more…

This post by Bernard Lelchuck originally appeared on the uTest blog.

If you haven’t noticed, the use of mobile applications has skyrocketed over the past few years. And while most mobile companies are lagging behind Apple’s success, the market itself has nevertheless become a multi-billion dollar endeavor. As one might expect, this success has prompted competitors of all sorts to rush and open their own mobile application stores. They naturally seek greater market share, and who could blame them?

According to a recent report published on the Wireless Expertise website, “the global mobile app market – including games – will be worth $4.66 billion in 2009, rising to $16.60 billion, in 2013.”

This of course would help explain the sudden entrance of Microsoft, Google, Research in Motion (RIM) and Palm, along with mobile vendors like Verizon and AT&T into the mobile market. As I like to say, they are trying to catch the fast-riding “Mobile App Train.”

And what a ride it’s been! Since the 1^st gen iPhone was released in June of 2007, almost every leading mobile vendor has changed their products to look, feel and be as cool as the iPhone (with varying degrees of success).

Which brings me to mobile testing. But before I discuss the testing implications of this iPhone mimicking trend, I’d like to address how I got into mobile testing in the first place. It’s my hope that this story will encourage other testers to consider furthering their careers by hopping on board the Mobile App Train.

Read more…