Mobile App Testing

You may have heard that uTest (the people who write this blog) just announced a $17mm Series D round of financing. We’re pretty excited. The press release included some great graphics showing the growth of uTest and the incredible explosion of testing that’s taken place since the company was founded (technically in late 2007, but we’ll just go ahead and call it 2008). These graphs not only detail one company’s growth, they reflect the current environment of mobile app testing and the growing trend of testing in the wild with a crowdsourced company. I thought the Mobile App Testing community might be interested.

Those are some pretty steep curves! Hopefully the trend keeps going – and I’m saying that because an upward trend is good for testing in general, not just because I’d like my company to keep succeeding (though having a job is nice).

Thanks for allowing a little bit of self-indulgence. If you’d like to read more about uTest’s Series D and what it means for the company’s future check out the official blog post from uTest CEO Doron Reuveni.

Here’s a great quote from Etsy test architect Noah Sussman on how the company handles the challnege of mobile testing:

uTest: Does the introduction of mobile, social media or location-based functionality make testing more difficult for a company like Etsy? What are some of the specific challenges that come with these variables?

Noah Sussman: Automated test tools for the mobile Web are less mature and so working with them incurs a high research and development cost. Of course that’s part of the fun, because we get to help invent the future by figuring how to leverage new tools like GorillaPhone and Kif.

Social media integration just means hitting a bunch of external APIs that you don’t control. It can be frustrating when a test fails and you can’t immediately fix it, but that’s a lot better than not knowing that an external service has gone down. Nagios is a great tool for these kinds of automated checks.

Read the full interview here >>>

From my interview on the uTest blog:

“Certainly there’s a great deal of extra complexity to be dealt with in the mobile space, when we look at the number of different systems and functions through which a given bit of data passes, or the enormous number of platforms on which people want to run apps.  Before Windows came along to abstract the hardware, there were drivers for each video card, each printer, each mouse, each network card times each operating system.  But then each application program came with special drivers to talk to each kind of hardware.  Developing and supporting all that stuff was completely nuts.

Since there’s a perception of lots of opportunity and lots of money in the mobile space, there’s a gold rush and lots of people are heading for the Klondike. Now there are competing mobile OSs, times all those versions of those OSs, times all those handsets and tablets and mobile browser versions and interconnecting apps and services.  So in a way, we’re back to the late 80s and early 1990s, back in the DOS days, when I first got involved with programming and support and testing.  Hey you kids, get out of my yard!”

Read the rest >>>

For readers of this blog – and for those who are familiar with the basics of crowdsourced testing – articles of the following variety will drive you insane.

Here’s a story from Wired.com on the problems wireless carriers are having with scalable testing. The execs interviewed here have half of the story correct; emulators and simulators are indeed ill-equipped  to handle modern-day testing in the mobile world. However, they are wrong about the impossibility of testing on multiples devices.

I’ve highlighted a few key sentences in bold to demonstrate. Take a look:

“Mobile emulators and debuggers are very weak, they’re immature. They’re not very useful,” Basso says. “So developers have to download the app and test it out on a phone themselves.” This can be expensive, if not impossible, if you want to test your app on every available device.

There are services that allow remote testing, but they can be of limited value to serious developers. The vast array of screen sizes and spec changes mean apps often aren’t “one size fits all.” The availability of APIs and SDKs is helpful, but seems to assume there won’t be any device problems.

“It’s very hard to test across all the different permutations, and you do need to test across them, ” Basso says, because there always are device problems.

And unlike browsers, easy-to-access metrics about device popularity aren’t available, so it’s difficult to choose which handsets are going to end up being the most popular.

Basso thinks that, with regard to AT&T’s Foundry program, it would be immensely helpful if carriers built developer centers in every city (or teamed up with local companies to do so), made all their devices available for testing and made it easy to wipe the device’s data afterward.

Basso sees the carriers’ developer-focused efforts as a way to deal with an area in which they’re woefully deficient. They’re putting energy and money into an area they don’t quite understand. But, he says, if they start doing things like providing physical developer centers with pre-release demo devices to work on, it will definitely spur innovation and attract more notice.

A developer center in every city? Aargh!!! There’s a much easier, cost-effective way to perform mobile testing across virtually any device/carrier. It’s called crowdsourced testing, and you can learn more about it here.

I just posted an interview on the uTest Blog with Michael Cooper of T-Mobile, where we chat about some great topics related to mobile app testing. Here are a few excerpts:

On testing advice:
I recommend standardizing your testing deliverables and keeping track of metrics so you can manage by facts.  Be independent – by that, I mean do everything in your power to not have to report to development.  Speak your mind…without whining and complaining.  Learn from your mistakes!  If you or your team make a mistake or miss a defect, take responsibility and do whatever it takes to ensure that it doesn’t happen again.  And, finally, treat your team and your peers with the utmost respect and dignity.

On mobile app testing:
Regardless of the industry, software testing is software testing; however, the pace of the telecom industry is moving extremely fast, and the speed to market is a key success factor. I enjoy getting knee-deep into projects to really understand the complexity mobile application testing, as well as the new tools and manual and automation techniques.

On his mobile device:
My primarily devices are both Androids – the HTC G2 4G phone and a Samsung Galaxy Tablet. T-Mobile USA was the first carrier to roll-out the Android operating system with Google and it was real exciting to be a part of this pioneering project. It’s amazing to think that just a few years later, there are now more Android devices then there is BlackBerry or iPhones.

On mobile app testing (again)
Test on the device itself, as soon as possible; you can miss defects if you only test on an emulator. Automate as much a possible so that you can efficiently test with many devices, platforms, and configurations. Be sure to pay attention to the feedback of you customers. I have found that mobile app testing requires a more technical type of testing, and automation is key. We are also looking into crowdsource testing to augment our current testing handset application testing efforts.

Read the entire two-part interview >>>

Straight off the wire:

uTest, the world’s largest software testing marketplace, won the Stevie(R) Awards’ top honor as “Most Innovative Small Company of the Year” at the 9th Annual American Business Awards Banquet in New York City. In addition, uTest Express — a service aimed at giving startups affordable access to the company’s crowdsourced professional testing service for web and mobile apps — was named a Finalist for “Best New Product of the Year.”

Read the full press release >>>

Here are a few excerpts from a guest post by Elena Houser, a mobile app tester for uTest.

Testing mobile apps is interesting, fascinating, challenging and frustrating. It has been an amazing and eye-opening experience for me. I am still very new to it and have more to learn. I tested my first mobile application during the uTest Bug Battle last November. Since then, I have participated in over 30 mobile test cycles through uTest and reviewed a variety of applications from games to browsers to enterprise resource planning (ERP) apps.

I want to know what kind of devices, applications and providers my friends and neighbors are using. I bombard them with questions on what they do and don’t like about their smartphones and providers, what kind of apps rock their world or make them want to throw their phones against the wall. Do they use mobile vs. native apps? How do they figure out navigation and flow, what do they think about a particular page layout, colors, sounds, portrait vs. landscape view, web-surfing and social media experience? Many of them check their bank statements and pay their bills, write memos, take snapshots, watch movies, use it as FiOS remote control, listen to the radio, play Angry Birds, Stupid Zombies and other games. I take notes. I try those apps myself. I install some of them on my Motorola Xoom even though many are still not optimized for it. It’s fun to launch an app and try to figure out how it works, what buttons to tap, and also picture the data moving behind it.

Just as with a web site the following three things matter the most for mobile apps – presentation and page layout, organization (flow and structure) and interaction (what happens in response to user actions). The difference is that a phone screen is very small and the capacity of a mobile device is not the same of a web site, which makes functional testing pretty challenging. Not to mention that some mobile applications may work on iPhones but then fail on either installation or launch on Android phones.

Read more…

Little known fact about Massachusetts: It’s the center of the mobile universe. BostInnovation.com just released some details of study proving my point:

The study which was released today at the MassTLC spring meeting states that the mobile sector here in Massachusetts employs over 30,000 people at almost 400 companies. When you think mobile you think Massachusetts.

Massachusetts has created a mobile hotbed by combining its traditional strength in network communications with the surge of activity among young entrepreneurs specializing in applications development.

“Our real advantage is the vibrant start-up culture we’ve nurtured here in Massachusetts,” said Carl Stjernfeldt, General Partner at Castile Ventures and a Mobile Cluster co-chair.  “Young people in particular are naturally drawn to this technology and  Boston has the advantage of a deep pool of youthful entrepreneurs.”

The study also estimates that the sector is generating just under $540 million in sales.

The entrepreneurial scene is so strong in fact that more than 80 percent of the mobile companies had fewer than 50 employees here in Mass. The study also estimates that the sector is generating just under $540 million in sales. And investment in Massachusetts local startups rose to 82 million dollars in 2010, a 40 percent increase over 2009, according to data supplied to MassTLC by Rutberg & Co., a leading wireless research and investment firm.  Even more promising, Rutberg’s data for the last twelve months suggests local mobile investments are on track to double this year.

Of course, these figures do not include the thousands of mobile app testers who are making money through uTest (another Massachusetts-based company), but it’s still pretty interesting.

Usability testing is one – if not the —most vital task when launching a mobile application. Before you expose a new application (or version) to the scrutiny of real users, it’s important to conduct a trial beforehand.

While enterprise companies spend hundreds of thousands on lab tests, focus groups and beta communities, some developers must make do with less. This, however, does not mean usability tests should be conducted with familiar resources (i.e. family and friends). In a recent interview on the uTest blog, usability expert Jakob Nielsen explains how companies can conduct user testing without breaking the bank:

“The main thing I recommend is to study your actual users: invite a handful of representative customers to your location and run them through simple usability studies of your software. One day in the lab is worth a year in university lecture halls, in terms of actionable lessons learned. (And remember that your “usability lab” can be a regular office or conference room — as long as you shut the door.)”

Once you’ve selected your focus group, the next step is to determine what you want them to focus on. Here are a few ideas to get you started:

• Page Layout: Ask your users to evaluate the overall effectiveness of the page layout. Are they instinctively drawn to your application’s main features, or are they directed elsewhere? Also, are they required to scan/scroll for any important features? The layout might seem intuitive to you, but that’s probably because you designed it! Therefore, a fresh set of eyes is a must.

• Color Schemes: This can include an evaluation of the use of colors in the background, text, links, icons, button and other aspect of your mobile app.

• Findability: When conducting your user tests, give them a set of 4-5 items to find. This could include Help, About, instructions, search or others. This will help you determine your application’s level of accessibility. Remember, mobile users have far less patience than web users, so these items need to be placed in the best spot possible. Find out where that is.

Read more…

Funny video from uTest Express – a service that lets developers test any type of mobile app before launching.