Mobile App Testing

Safe to say that mobile app development has greatly outpaced mobile app testing over the last few years. In other words, while the applications, platforms and operating systems have seen tremendous advances, the same cannot be said of mobile testing methodologies.

Case in point: The majority of mobile app developers are still overwhelmingly reliant on internal beta testing.

Here with proof is VisionMobile, who recently published a fascinating report on the mobile app ecosystem. A must-read for anyone – and I mean anyone – involved with mobile applications (i.e. developing, marketing, sales, etc.) the report should serve to demonstrate that mobile app testing has a long way to go before it’s fully matured. Here’s the key excerpt:

Internal beta testing is the most popular technique used by the vast majority (nearly 70 percent) of respondents, with beta testing with users and peer reviewing the next most popular techniques. Only 20 percent of respondents use focus groups or research of their own. Overall, North American developers are somewhat more sophisticated in their application planning, with 97 percent using beta testing as a standard part of application development and with broader use of a portfolio of planning techniques as well.

Yet, small development firms have limited means today to beta test and peer review their applications with a crosssection of representative users. Given the hundreds of thousands of mobile apps, we believe that efficient
(crowd-sourced) testing of apps in a global market of users is considerably under-utilized. This presents an opportunity for the few solution providers in this segment – Mob4Hire and uTest.com, for example – but also for network operators, who can generate a channel for testing applications with end users, and provide an open feedback support system back to developers.

Other interesting findings included:

Read more…

The mobile industry is being increasing cluttered with jargon, says Anuj Gupta, our latest guest blogger for mobileapptesting.com. In this post, Anuj clears the air with an essential primer for anyone remotely involved in the mobile testing space, including discussions on the most popular operating systems and application platforms for iPhone, Blackberry, Windows mobile and more.

Mobile devices are evolving into increasingly sophisticated general purpose computers and this has led to the development of a variety of platforms and operating systems in the mobile space. Today, the operating system lying inside your mobile device is equally important as the device manufacturer (OEM). The various features, GUI, processing speed, and most importantly, the applications available for your device will depend greatly on the underlying OS.

Unfortunately, industry jargon has cluttered our understanding of this emerging technology. It is my intention to clear the air on a few of these misunderstandings. Here goes….

Manufacturers launch devices based on various user segments, and giving rise to a number of Mobile OS and Application platforms. A mobile OS manages the hardware and software resources of a mobile device, similar to that of a computer OS. Some OS platforms cover the entire range of the software stack, while others may only include the lower levels (typically the kernel and middleware layers) and rely on additional software platforms to provide a user interface framework.

Today’s phones are expected to run a growing range of software such as internet browsers, navigation packages, games and music/video players. Application platforms are supposed to provide a ground for them.  Application platforms are built over lower-level kernel operating systems (OS) such as Rex, Linux or compact real-time OS’s such as Nucleus. Software running on the device accesses platform resources through a set of application programming interfaces or APIs. Application development platforms, such as Brew, Symbian UIQ, Android, LIMO, ALP, Qtopia or WIPI provide programming resources for native and Java applications.

Mobile developers typically develop for multiple platforms to maximize their available market. This can be a difficult and time-consuming task as multiple platforms use different API calls for common OS operations and accessing OS resources such as accessing memory and files.

There is a variety of both the OS and the application platforms in the market. Obviously, some are more dominant than others. Here is a list of major mobile operating systems and Application platforms.

Read more…

From the mobile-savvy folks at Telegraph.co.uk:

Agree or disagree?  Speak your mind!

The WAP Catalog has just posted a terrific app review of Twitter for Blackberry (in case you planned on downloading it). Mobile testers could learn a lot from this thorough content, especially from a usability perspective, so go check it out when you get a second. In the meantime, here are a few brief excerpts:

The Good: Push notifications for Direct Messages (DMs) and integration with BlackBerry’s native message folder make for an impressive experience.

The Bad: This release is still pretty buggy; it hangs and lags too often for a final build, despite perfect 3G and Wi-Fi coverage.

Recommendation: Yes.  Despite the flaws, which can honestly get quite annoying at times, the features and execution are such that I would recommend this application to BlackBerry users.

The Review

I have been using Twitter for Blackberry Public Beta (T4BB for short) for nearly a month now since its release to the public in early April.  Testing of the app has taken place in the Metro Atlanta area with pretty solid 3G and Wi-Fi coverage throughout, on a T-Mobile branded BlackBerry Bold 9700.  T4BB is a feature-rich application that contains all the functionality of the Twitter website, with added flare that will make it instantly familiar to seasoned BlackBerry users (such as the ‘t’ and ‘b’ shortcuts that instantly take you to the top or bottom of your timeline). T4BB will suck the life right out of your battery depending on usage and update settings, but so will any other application that updates itself on regular intervals.

Read the rest of the article.

This post by Bernard Lelchuck originally appeared on the uTest blog.

If you haven’t noticed, the use of mobile applications has skyrocketed over the past few years. And while most mobile companies are lagging behind Apple’s success, the market itself has nevertheless become a multi-billion dollar endeavor. As one might expect, this success has prompted competitors of all sorts to rush and open their own mobile application stores. They naturally seek greater market share, and who could blame them?

According to a recent report published on the Wireless Expertise website, “the global mobile app market – including games – will be worth $4.66 billion in 2009, rising to $16.60 billion, in 2013.”

This of course would help explain the sudden entrance of Microsoft, Google, Research in Motion (RIM) and Palm, along with mobile vendors like Verizon and AT&T into the mobile market. As I like to say, they are trying to catch the fast-riding “Mobile App Train.”

And what a ride it’s been! Since the 1^st gen iPhone was released in June of 2007, almost every leading mobile vendor has changed their products to look, feel and be as cool as the iPhone (with varying degrees of success).

Which brings me to mobile testing. But before I discuss the testing implications of this iPhone mimicking trend, I’d like to address how I got into mobile testing in the first place. It’s my hope that this story will encourage other testers to consider furthering their careers by hopping on board the Mobile App Train.

Read more…

It wasn’t left behind in a bar, but BoyGeniusReport.com has obtained some pretty cool screenshots and details about the new BlackBerry OS. So do they like what they see? You bet. Below are a few excerpts, but you should go read the article.

Usability
“RIM has finally implemented system-wide kinetic scrolling with rubberbanding. This makes the experience a bit more fluid when scrolling through lists, web pages, emails, and the like. There is also multitouch system-wide, from the web browser to the photos application, and yes, pinch to zoom is in there.”

Email Inbox
“Things look a bit cleaner, and a bit more modern, but it’s obviously not a huge departure from what we have now, just another evolutionary upgrade which is par for the course with RIM. But, we definitely do like the look of the new inbox. It also features larger thumbnails to the left of each message to identify the type of message (Facebook, Twitter, etc.) and we even saw RSS feeds built right in which looked great. We can’t describe this part… but the new inbox feels right. It’s not all work and dry, and it’s not over the top with cheesy graphics or animations, rather a nice compromise of the two.”

Read more…

Here’s the second part of Jigar Patel’s post on mobile web app testing.

Safari and Mozilla Firefox support some great extensions that can make testing your mobile sites on desktop browser much easier. You can get more details here.

Testing mobile applications on the desktop browser, as mentioned, does provide several major advantages. Let me illustrate: From the browser metrics of the application, prepare a list of user agent strings for each browser in your metrics. User agent strings can be easily found through a Google search. From there, setup the Firefox browser to test with mobile application as mentioned in this article. Then, set the user agent sting in Firefox for each device, one by one, and navigate to the sign-in page of the application. See if the site renders the mobile version of the application or the PC version if available. If it does, then we are on the right path, and there is a good chance that real handset seating on the end user hand will redirect to the correct mobile version of the site.

If it renders the PC version of the site, this could be a bug in your application. If so, there are less chances for handsets being supported by your application, provided that it doesn’t render the mobile version of your site. Be sure to document each browser on which this behavior occurs and ask dev to fix it. If it can’t be fixed, then there is less of a chance that the real handset will support it. At this point, we can make the decision to cut down the devices/browser from the metrics which doesn’t redirect to the mobile version of the site. Now you can understand the advantage of testing on a desktop browser, that we can kick out the devices from this analysis before purchasing it to test and hence it will help to reduce the cost!

Read more…

We’ve made a lot of bold promises here at mobileapptesting.com – free cars, dates with celebrities and user-generated content, to name a few. Here to help us make good on at least one of these promises is guest blogger Jigar Patel of India (pictured right), who has written a two-part post on some of the best practices for mobile application testing.

Jigar has over five years experience as a software test engineer, and has recently taken a keen interest on the mobile side of testing. In this post, he tackles dev metrics for browser, device, region, carrier and other criteria, in addition to the testing process. Check back soon for part II.

***

This post is based on my working experience as a software test engineer – specifically, my thoughts on the growing trend of mobile web app testing. Since this trend is new to a lot of people, I hope to provide a road map of sorts for achieving efficient mobile app testing. Let’s start with metrics.

Browser/Device Metrics
Deciding on browser and device metrics should ideally occur in the early stage of the development process. Of course,   much depends on the nature of the application being developed, but below are some general criteria on deciding which metrics should be considered.

• Browser capability: If the application being developed will be dynamic or interactive, then the browser must support the following:
  • JavaScript support: Mobile browsers must support JavaScript in order the build an interactive application UI.
  • XML HTTP Request Object:  XML HTTP request object support is required to communicate with the back-end server and to update the page with new data without reloading the page. This will give the user a smooth browsing experience for the site.
  • CSS support: CSS defines how page elements are to be displayed and enables you to change appearance and layout of all of the pages by editing a single file.

There should also be performance analysis of the browser. If the browser performance is poor, then it should not be a part of the rich mobile application metrics that is going to be developed.

Browser metrics, of course, are subject to change. This can depend on changes in technology, including the growing (or diminishing) popularity of the browser. In this case – when the browser that is not currently part of the metrics is gaining in popularity and major market share – then it should be included in the metrics at some point to support the application. The sooner, the better.

Read more…

This post by mobile testing pro Brad Sellick (@sellib) originally appeared on the uTest blog.

There’s a challenge in developing and testing applications that is as old as the personal computer itself. The challenge being that the design, development, and testing of applications often takes place in an environment much different from “the real world” where users run and interact with those applications.

With the rise of mobile platform devices like the Blackberry, iPhone and now the iPad, I believe that challenge is greater now than at any time in the past. We now find ourselves producing applications in an environment (a desktop or laptop computer) that is completely different from the device the application will run on.

This challenge is compounded by the fact that development platforms have a very convenient method of testing a mobile app on the desktop via a simulator tool.

My own experience with iPhone development was a major eye-opener. I spent a lot of time building and testing applications on my Mac desktop and the simulator application. However, when I finally loaded an application onto an iPhone for the first time, it was a completely different experience.

Read more…

If media coverage equaled market share, then I’d be writing this post from my iPhone (I’m not) and every single one of you would be reading it from your shiny new iPad (you’re not). In case you haven’t been near a TV… or a computer… or a radio… or people… you’re aware that Apple launched a new product last week called the iPad.

And with the apparent ubiquity of the iPhone, one can only assume that Apple’s mobile market share hovers somewhere between 97% and 109%. Unless, of course, you look at those pesky “statistics”, which is exactly what the fine folks at Comscore do each month. As Jason Kincaid (@jasonkincaid) covered recently, the latest mobile market share might surprise you:

• RIM and their family of Blackberry devices continue to hold the pole position. In fact, their actually gaining ground, picking up 1.3% since Q4 of 2009 to its current position of 42.1% of the market.
• Android continue to gain market share rapidly jumping from 3.8% of the market in Q4 to 9.0% in Q1. Android still has a lot of ground to make up to catch up with Apple and RIM, but a few more quarters like this and they’d catch (and pass) the leaders.
• Despite the high-profile product launches, Apple’s share of the market was actually flat (down .1%)
• Microsoft and Palm continue to shed market share (down 4.0% and 1.8%, respectively)

For those of you who have switched phones in the past few months, I’m curious to know if these stats are in line with the recent purchasing decisions you and your friends/co-workers are making. What say you? What smartphone are you packin’ these days?

This post originally appeared on the uTest blog.