Mobile App Testing

Carrier IQ has been causing quite a fuss lately. It started with one guy (Trevor Eckhart) doing some research, followed by a poorly thought out cease and desist letter from Carrier IQ, and has since spread like wildfire – touching Verizon, Sprint, AT&T, Apple, Android, RIM, just about everyone in the cell market. Stories have been flying around online about which devices and carriers do, don’t, or might have Carrier IQ. Through all this, developer supercurio smelled an opportunity: an app that will detect Carrier IQ on your Android.

It’s a clever leveraging of a very hot topic right now, I have to give supercurio credit for that. However, in his rush to market he didn’t take time to adequately test the app. Here’s what CNet has to say:

All the controversy surrounding Carrier IQ has prompted a developer to create an application that helps Android device owners determine if their handset is running the software.

Dubbed Voodoo Carrier IQ detector, the application, which is available in the Android Market, helps “you find out as easily as possible if your Android device hosts CarrierIQ rootkit or not.” The free application is open source, and its developer, “supercurio,” says he’s willing to work with others “for collaboration and contributions.”

But before you download the program, beware that it might not be so accurate. Supercurio admits in the app’s listing that its “results are not reliable yet,” and he plans to provide several updates that will eliminate the current false positives it’s generating.

So is it better that supercurio got his version of the app – even if it is imperfect – into the Android Marketplace before someone else could scoop up the opportunity? Or should he have taken the time to ensure that the app actually work properly? Testers and QA specialists, what do you think?

A reminder for all of you usability testers out there: less is more. That’s the short version. Here’s the longer version courtesy of ZDnet.com:

Lately I’ve noticed some developers are getting away from this focused approach. Apps I have used for a long time are gradually getting buggier and bulkier, as one feature after another are added by the developers. The extra features don’t usually make the app better at performing the primary function, instead they add new capability just in case the user might find it beneficial.

Unfortunately, this rarely works for most users, especially the installed user base. It is jolting to update an app that one has used for a while, only to find that now the task it has always been used to perform is buried under new features and options. The user has to stop for a moment to see what the new stuff cluttering the screen may be. This is a failed effort if this prevents using the app for the original purpose, as it has turned the user experience from rock-solid to “now where did that go”.

I do a lot of private beta testing for new apps, and recently I’ve run across this developer attitude with a couple of them. One app was designed to do a specific function, but do it very well. The developer had done his homework, and the early versions of this app were really good. Then, over time the urge to add functions set in, and every new version of the app got worse at performing the primary task. Feature after feature kept getting added, just in case they added perceived value, and with that the app got harder to use and in some cases failed to perform the primary function.

Read the entire article >>>

Is your mobile app accessible to a broad audience? Of course, there’s only one way to find out: testing!

Here with some accessibility pointers is The Guardian. Take a look:

Testing, testing
Talked to people with accessibility needs for point two? Test with them before (and after) your app is released. On iOS, developers are using services like TestFlight to put beta versions of their apps into the hands of testers before submitting them to Apple for approval – a golden opportunity to find out early if your app is falling short.

Meanwhile, once an app is released, it’s important to keep testing it, especially when there’s an update for the operating systems that it’s available for. Accessibility technology, and features in iOS, Android and the rest, are evolving steadily, so even if your app was accessible when it was released, don’t assume it can’t be improved after that point.

There are a few other great pointers for developing apps in general, but this one I found to be particualrly helpful:

Simplify wherever possible
Another accessibility principle with benefits beyond any particular group of users is simplicity. Cramming features, menus and on-screen prompts into an application is an easy road to go down, especially when you’re the developer, and so know your way around them.

Less options, clearer prompts and a well-defined pathway around the application are improvements that will pay off across the board, but they’ll be particularly appreciated by people using screen-reading technology, or just people who are fairly new to smartphones and apps.

Read the rest >>>

Twelve New York City startups are currently experiencing Microsoft’s Mobile Acceleration Week where “hand-picked startups can access hands-on support and training to build compelling apps for the Windows Phone platform,” according to TechCrunch. The startups ultimately have 60 days to push their new Windows app into the marketplace.

Jordan Crook, who wrote the TechCrunch article, choose four apps he’s particularly excited about. All the companies he highlights already have iOS versions of their apps and three of the four also have Android versions. Which leads me to the question of the day …

Seeing as how the companies are working on a set (publicly announced) timeline, does already having one or two versions of an app make it easier or harder to develop and test the new Windows Phone 7 app? Do you rest on your laurels and hope it works? Do you get so distracted by how you did it last time that it’s hard to see what needs to be done differently? Are you willing to change some the look and feel of the “old app” to make it fit the new device? How about the fact that the Microsoft Windows Phone Marketplace only has 40,000 apps versus the six-digit figures in the Apple and Android shops? Does that make you complacent or push you to make an even better app?

Or does none of that matter?

Developers? Testers? What do you think? Of course you should work hard to make each project the very best it can be, but what’s your gut reaction (before your better judgement wins over) to these questions? I’d like to see a lively discussion in the comments so let’s get this going!

And be sure to check out Jordan’s top picks from the NYC Mobile Acceleration Week at TechCrunch.com >>>

Here’s a great quote from Etsy test architect Noah Sussman on how the company handles the challnege of mobile testing:

uTest: Does the introduction of mobile, social media or location-based functionality make testing more difficult for a company like Etsy? What are some of the specific challenges that come with these variables?

Noah Sussman: Automated test tools for the mobile Web are less mature and so working with them incurs a high research and development cost. Of course that’s part of the fun, because we get to help invent the future by figuring how to leverage new tools like GorillaPhone and Kif.

Social media integration just means hitting a bunch of external APIs that you don’t control. It can be frustrating when a test fails and you can’t immediately fix it, but that’s a lot better than not knowing that an external service has gone down. Nagios is a great tool for these kinds of automated checks.

Read the full interview here >>>

Call of Duty has been having some load issues lately. Last week a stat tracker/social network called Call of Duty Elite (which was tied to the release of Call of Duty: Modern Warfare 3) crashed when it couldn’t handle the giant influx of excited gamers.

At launch, our registration and login systems were crushed by gamers trying to enter the ELITE site at the same time. We have now fixed the registration and login systems, but we have found that the greater than expected demand is crashing servers. We’re immediately deploying multiple additional servers to beef up the system. We are also going to temporarily limit access to ELITE services on both the console applications and website while we build additional capacity and scale. We’ll look to increase access to greater numbers of users as soon as possible.

The issue is still being resolved.

But since this is MobileAppTesting.com, let’s get into the mobile app side of things. Call of Duty was supposed to launch a mobile version of Elite. That plan is now on hold. Following the initial site crash, Elite’s creators, Beachhead Studios, have decided to delay the mobile release and do more testing to prevent another problem. Here’s what the latest Call of Duty Elite status update said:

While many people are curious and excited to use Call of Duty ELITE’s mobile applications, we plan to release the iOS and Android apps when we’re confident the service will be able to handle the extra traffic they will generate – so stay tuned.

I don’t know if there were simply more concurrent users than Activision & Beachhead predicted or if they just didn’t do enough load testing. But at least they’re taking the chance now to up their testing instead of just moving forward with a faulty product. As they say, “better late than never” … but it sure would have looked a whole lot better if Beachhead had tested all this before a public launch.

Back in July Mike wrote about a new AT&T Labs Research report that took a look at end to end data transmission paths in an attempt to figure out why some apps drain so much battery life or are slow. The researchers “discovered the source of the problem in the complex interactions between the application, device and the cellular network, looking into network architecture and lower level protocols that are typically hidden from application developers.” Here’s the root of the issue and an example from AT&T’s report:

In particular, application developers are usually unaware of cellular specific characteristics that incur potentially complex interaction with the application behavior. Even for professional developers, they often do not have visibility into the resource-constrained mobile execution environment. Such situations potentially result in smartphone applications that are not cellular-friendly, i.e., their radio channel utilization or device energy consumption are inefficient because of a lack of transparency in the lower-layer protocol behavior. For example, we discovered that for Pandora, a popular music streaming application on smartphones, due to the poor interaction between the radio resource control policy and the application’s data transfer scheduling mechanism, 46% of its radio energy is spent on periodic audience measurements that account for only 0.2% of received user data.

In the most recently released highlight of the report, AT&T Labs Research identifies 11 ways to build efficient apps that will work no matter what carrier you’re targeting (all the recommendations are aimed at carrier-independent standards and protocols). Here’s the list:

• Understand Mobile Networks
• Handle Multiple Simultaneous TCP Connections
• Reduce Inefficient Connections with Periodic Transfers
• Deal with Connection Opening Problems
• Close Connections As Soon As Possible
• Offload to Wi-Fi
• Handle Duplicate Content and Caching
• Utilize Pre-fetching
• Manage Peripherals
• Upgrade from HTTP 1.0
• Handle Screen Rotations

Now let’s break down a few. Since the first few tips were more or less covered in July’s initial post, we’ll jump down the list a bit.

Read more…

Last week I wrote about RootMetrics’ 4G report and PRWeb’s story detailing which carriers had the best 4g network. Today, PRWeb released another story looking at a different part of the report – which cities had the best 4G coverage. Nineteen of the 27 markets RootMetrics investigated passed their 4G test (which looked for consistent 4G speed). Denver came in at No. 1 while New York City, Boston and Washington D.C. (among others) didn’t pass the test. Here are the top and bottom scoring cities:

RootMetrics™, the first independent service to measure mobile experience from a consumer’s point of view, announced results from comprehensive testing in 27 different local markets* telling which of those cities qualify as “4G Cities.” Since March 2011, members of the RootMetrics test squad have traveled the country and performed more than 238,000 data tests. By aggregating all of these data test results, the company has identified patterns of performance and can isolate carrier consistencies. The good news for mobile users is that RootMetrics tests revealed that 19 of the 27 markets exceeded the 4G threshold. Eleven earned “Upper Tier” status, recording speeds above the 4G threshold in more than 50 percent of tests. These include Denver (61.8 percent), Portland (61.5 percent), Minneapolis (60.2 percent), San Jose (59.2 percent), Dallas (57.6 percent), Cincinnati (56.8 percent), St. Louis (55 percent), San Francisco (52.9 percent), Sacramento (51.8 percent), Cleveland (51.4 percent) and Seattle (50 percent). Eight qualified as “Second Tier” status, recording speeds above the 4G threshold in 40-50 percent of tests. These include Atlanta (49.6 percent), Los Angeles (46.4 percent), Houston (45.2 percent), Milwaukee (45.1 percent), Phoenix (44.5 percent), Pittsburgh (43 percent), Philadelphia (42.2 percent) and Chicago (41.1 percent).

Read more…

So you went with the free antivirus app on your Android (hey, we like free too!). Is it really good enough? According to AVTest, it’s not. Here’s what PCMag has to say:

Researchers at German antivirus testing lab AV-Test.org run regular certification tests on PC antivirus products under Windows 7, Vista, and XP. This month, though, they turned their attention to the many antivirus products available for Android devices. Their conclusion: most of the free ones are useless. Or rather, they’re worse than useless because their users gain a false sense of protection.

For testing they selected seven popular free antivirus apps from the Android Market. They also included F-Secure Mobile Security and Kaspersky Mobile Security for comparison purposes.

Each product was installed on an Android device containing inactive specimens of over 150 recent Android threats. Researchers ran an on-demand scan and recorded how many threats were detected. Kaspersky and F-Secure detected over half. The best free product, Zoner Antivirus, caught 32 percent. All the rest detected under 10 percent, and some didn’t detect any samples at all.

Check out how the other tests turned out at PCMag.com >>>

The big news in Boston this week is the return of Mark Zuckerberg, who’s come back in search of tech talent – engineers, developers and perhaps even testers. Based on their recent hiring of a mobile developer team in San Francisco, there’s good reason to believe this round of head hunting is also centered around mobile, something Facebook has been focusing on for the better part of two years. Here’s the story on his return from The Boston Herald:

Zuckerberg said he made the trip to Harvard — and to the Massachusetts Institute of Technology earlier in the day, meeting with 200-plus students at each school — because “there’s a lot of really smart people here, and a lot of them are making decisions on where they’re going to work when they graduate.”

At MIT, Zuckerberg told reporters that he talked to many venture capitalists after starting Facebook in his Harvard dorm room, but ultimately concluded the ones that he wanted to work with — people like PayPal co-founder Peter Thiel, who had started companies themselves — necessitated his move to Silicon Valley.

Zuckerberg also clarified remarks that he made recently at Stanford University about possibly staying in Boston if he had to do it all over again.

“The point I was trying to make wasn’t that I could necessarily have started Facebook in Boston or stayed here,” said Zuckerberg, who still looked like a student in a T-shirt, sneakers and jeans. “It was that I think there is more than one place for people to build companies. There’s a feeling in Silicon Valley that to be successful, you have to be in (there) because that’s where all the engineers are. I don’t think that’s true.”