Mobile App Testing

There have been a plethora of studies looking at mobile shopping trends this holiday season (especially now that Cyber Monday is almost as well know as Black Friday). Most of the reports say vaguely the same thing – shopping on mobile devices has increased dramatically this year, especially when it comes to shopping on tablet devices. Were online retailers ready for this new trend? A study by Compuware doesn’t think so. From TechCrunch:

Despite the increases in tablet traffic, many retailers are not prepared to accommodate these new mobile shoppers. Compuware also prepared a chart showing the top retailers’ sites, and whether or not they offered an iPad-optimized website. Surprisingly, none of them do, not even Apple.com. What’s worse, Apple is also among the retailers who don’t offer a native iPad application. (The iPhone Apple Store app runs on the iPad, of course, but it’s not a universal app). For shame!

Apple is not alone though. Around half of the 30 top retailers Compuware looked didn’t have an iPad application, either.

That is one big, blank column! It’s pretty clear that retailers have adapted to the reality of mobile shopping when it comes to smartphones, but they’re a bit behind in optimizing for tablets. I wonder if any of these retailers have rushed to get a tablet optimized web (or native for those that didn’t have one) app out after the Black Friday – Cyber Monday rush. If they did, I hope they took the time to do some decent functional, usability and load testing before going live. Having a buggy, poor-performing app might just be worse than not having an app at all.

You may have heard that uTest (the people who write this blog) just announced a $17mm Series D round of financing. We’re pretty excited. The press release included some great graphics showing the growth of uTest and the incredible explosion of testing that’s taken place since the company was founded (technically in late 2007, but we’ll just go ahead and call it 2008). These graphs not only detail one company’s growth, they reflect the current environment of mobile app testing and the growing trend of testing in the wild with a crowdsourced company. I thought the Mobile App Testing community might be interested.

Those are some pretty steep curves! Hopefully the trend keeps going – and I’m saying that because an upward trend is good for testing in general, not just because I’d like my company to keep succeeding (though having a job is nice).

Thanks for allowing a little bit of self-indulgence. If you’d like to read more about uTest’s Series D and what it means for the company’s future check out the official blog post from uTest CEO Doron Reuveni.

Carrier IQ has been causing quite a fuss lately. It started with one guy (Trevor Eckhart) doing some research, followed by a poorly thought out cease and desist letter from Carrier IQ, and has since spread like wildfire – touching Verizon, Sprint, AT&T, Apple, Android, RIM, just about everyone in the cell market. Stories have been flying around online about which devices and carriers do, don’t, or might have Carrier IQ. Through all this, developer supercurio smelled an opportunity: an app that will detect Carrier IQ on your Android.

It’s a clever leveraging of a very hot topic right now, I have to give supercurio credit for that. However, in his rush to market he didn’t take time to adequately test the app. Here’s what CNet has to say:

All the controversy surrounding Carrier IQ has prompted a developer to create an application that helps Android device owners determine if their handset is running the software.

Dubbed Voodoo Carrier IQ detector, the application, which is available in the Android Market, helps “you find out as easily as possible if your Android device hosts CarrierIQ rootkit or not.” The free application is open source, and its developer, “supercurio,” says he’s willing to work with others “for collaboration and contributions.”

But before you download the program, beware that it might not be so accurate. Supercurio admits in the app’s listing that its “results are not reliable yet,” and he plans to provide several updates that will eliminate the current false positives it’s generating.

So is it better that supercurio got his version of the app – even if it is imperfect – into the Android Marketplace before someone else could scoop up the opportunity? Or should he have taken the time to ensure that the app actually work properly? Testers and QA specialists, what do you think?

ReadWriteWeb just released its list of “Top 10 Mobile Products of 2011.” It looks like they didn’t take the task lightly and made sure all the major players in each category got considered. Here’s what they had to say about the process:

Looking back on 2011, it may be remembered as The Year Of Mobile. Sure, iOS, Android, BlackBerry, Windows Phone and all the other platforms existed in previous years but historians will look back at 2011 and say that it was the year that the way an entire populace interacts with information fundamentally changed. Mobile is not just for the early adopters anymore. Smartphones are everywhere.

What made waves in the mobile realm this year? … To make the mobile list, a product had to be built to fundamentally work inside mobile platforms, hence the platforms themselves (iOS, Android flavors etc.) do not make the list. …

Our 2011 list includes location services, security, social networking, payments, HTML5, NFC among other topics. The next year will be fascinating to watch these platforms grow into things that normal people would have never dreamed of creating. In December 2012 this list will have more HTML5 components and NFC will be in the hands of many more people.

Read more…

A reminder for all of you usability testers out there: less is more. That’s the short version. Here’s the longer version courtesy of ZDnet.com:

Lately I’ve noticed some developers are getting away from this focused approach. Apps I have used for a long time are gradually getting buggier and bulkier, as one feature after another are added by the developers. The extra features don’t usually make the app better at performing the primary function, instead they add new capability just in case the user might find it beneficial.

Unfortunately, this rarely works for most users, especially the installed user base. It is jolting to update an app that one has used for a while, only to find that now the task it has always been used to perform is buried under new features and options. The user has to stop for a moment to see what the new stuff cluttering the screen may be. This is a failed effort if this prevents using the app for the original purpose, as it has turned the user experience from rock-solid to “now where did that go”.

I do a lot of private beta testing for new apps, and recently I’ve run across this developer attitude with a couple of them. One app was designed to do a specific function, but do it very well. The developer had done his homework, and the early versions of this app were really good. Then, over time the urge to add functions set in, and every new version of the app got worse at performing the primary task. Feature after feature kept getting added, just in case they added perceived value, and with that the app got harder to use and in some cases failed to perform the primary function.

Read the entire article >>>

‘Tis the season for shopping! People are hitting the stores, shopping online and, increasingly, using their mobile devices to get that holiday shopping done in time. According to a comScore study of mobile shopping in September 2011, mobile browser shopping is a favorite among smartphone owners with app-based shopping coming in second. Here’s what comScore has to say:

A look at how U.S. consumers accessed retail content via their mobile phones revealed various usage patterns across platforms. Nearly half of all smartphone owners used only their mobile browser to access retail content in September 2011, with 26% using just apps and 18% using a combination of access methods.

Apple iPhone users were more likely to access retail content via apps only (36%) when compared to Android users (24%).

It’ll be interesting to see this report recreated following the November-December shopping season. Just looking at Black Friday (one day) shows a significant mobile shopping jump over last year. IBM put out a Coremetrics Benchmark Report looking at Black Friday and Cyber Monday. Let’s concentrate on the Black Friday mobile stats (though they don’t go deep enough to separate mobile browser shopping from native app shopping):

The Mobile Bargain Hunter: Black Friday witnessed the arrival of the mobile deal seeker who embraced their devices as a research tool for in-store and online bargains. Mobile traffic increased to 14.3 percent on Black Friday 2011 compared to 5.6 percent in 2010.
Mobile Sales: Sales on mobile devices surged to 9.8 percent from 3.2 percent year over year.
The Apple Shopper: Mobile shopping was led by Apple, with the iPhone and iPad ranking one and two for consumers shopping on mobile devices (5.4 percent and 4.8 percent respectively). Android came in third at 4.1 percent. Collectively iPhone and iPad accounted for 10.2 percent of all online retail traffic on Black Friday.

The question as we move forward is no longer about online versus mobile shopping, it’s about mobile web versus native app shopping. Right now, it’s looking like mobile web might win.

Chances are, if you’ve got an iPhone or Android device, you’re probably staying put. Loyalty reigns supreme in the world of smartphones, yes? But just in case you are considering making a switch – in this case, to Windows Phone 7 – you can now give it a test run via Microsoft’s cool new simulator.

Here’s TechCrunch with the details:

The simulator is cute, and definitely a great example of what can be done with some incredibly clever scripting — but I’m not sure that it’s actually helpful, either for Microsoft or the curious user. For a laundry list of reasons, the simulator only scratches the surface of what Windows Phone 7 can do; while you can tap into certain screens (the People hub, the Phone dialer) and kinda-sorta simulate what you’d do there, the vast majority just kind of sit there. You’re really just walking through a series of animated screenshots. For anyone who’s written a line of code, the limitations make sense — but for the user trying to get that full-fledged Windows Phone experience, it could easily come across as broken.

Still, it’s worth checking out if only to admire the scripting work. You can find it at http://bit.ly/vvvU05, or by clicking this direct link from your smartphone. It’ll (sort of) work from your desktop browser as well — but as it’s tailored around swipe gestures and touch input, mouse input tends to choke it up.

Read the rest of the article >>>

Here in the U.S. it sure seems like everyone and their mother has a smartphone these days. But when you take a global snapshot of the cellphone market it turns out that only 27% of cell owners are sporting a smartphone, according to a recent study by VisionMobile. TechCrunch does a nice job of summarizing the report and including some great infographics:

With all the talk of iPhone vs. Android these days, it’s easy to forget how the majority of the world’s mobile users still make calls and access data: via feature phones. A recently released report (download) from mobile strategy firm VisionMobile takes a look at today’s mobile marketplace finding that, despite the sharp rise in smartphone shipments over 2010 and 2011, global smartphone penetration (by OS) is at just 27%.

Smartphone adoption varies wildly by region, the report finds. Not surprisingly, those markets where 3G coverage is extensive and subscription plans are “post-paid” (as opposed to pre-paid) see the highest smartphone adoption rates. Meanwhile, in markets dominated by pre-paid subscriptions, the real battle is price. Here, Nokia’s mid-tier Symbian platforms and BlackBerry consumer-targeted models are still holding onto significant market share. However, both platforms are now facing threats from low-cost Android phones, thanks to the latter’s pricing versatility. Android devices today sell for anywhere from $100 to $750 (USD), allowing the phones to compete both on the high-end and the low-end of the pricing spectrum.

In the North American and European markets, smartphone penetration is the highest, with 63% and 51% market share, respectively. In the Asia-Pacific region (19%), Africa/Middle East region (18%) and Latin America (17%), it’s much lower.

Read more…

Well, that depends on who you ask. Informationweek.com just published a lengthy analysis on the usability and “practicality” of the big three – iOS, Android and Windows Phone 7.5 – in the context of the enterprise. You should go read the entire article, but allow me to briefly quote a few of their key points:

iOS

The Good: “Siri is still a work in progress, and it needs to broaden out its scope beyond a handful of services; in fact, when developers are finally given access, that’s when Siri will truly start to shine. Until then, however, it’s still a pretty compelling reason to buy an iPhone.”

The Bad: “While Apple seems to have what many enterprises want for encryption, the problem is that iOS stores the keys for encryption on the phone, making them easily recoverable by an attacker. Thus, Apple’s encryption is not enterprise ready.”

Android

The Good: “Android takes this a step further with widgets–a thin version of an app sprawled across the display, showing actual content where applicable–say the newest entry in your Facebook news feed, or the latest Twitter post. In this sense, then, these phone systems are evolving from an application-centric world to an information-centric one.”

The Bad: “Android has had voice-enabled search for a while and there are third-party applications, like Speaktoit, but none of this quite compares to Siri.”

Read more…

There’s no question that Apple’s iPad dominates the tablet market. But what if the iPad didn’t exist? What would the market look like then? The NDP Group wanted to find out and this is what they came up with in a recent study of U.S. Tablet Sales:

U.S. tablet sales, excluding iPad sales, soared to more than 1.2 million units sold from January through October and brought in $415 million in revenue at retail, according to leading market research company The NPD Group’s monthly Connected Handhelds Report.Total tablet industry sales, for both hardware and accessories, brought in nearly $700 million. …

“If you look at the tablet market without Apple there are a number of high-profile brands vying for that number two spot,” said Stephen Baker, vice president of industry analysis at NPD. “According to NPD’s Consumer Tracking Service, 76 percent of consumers who purchased a non-Apple tablet didn’t even consider the iPad, an indication that a large group of consumers are looking for alternatives, and an opportunity for the rest of the market to grow their business.”

PC manufacturers are dominant in the tablet space, as four of the top five tablet brands already have a strong U.S. consumer PC presence.

 Top 5 U.S. Tablet Brands (excluding Apple)
Retail Sales Jan-Oct 2011

Check out the study >>>

Read more…