We talk about the testing matrix, well, all the time. It can seem intimidating. And, in fact, that was the complaint from some readers regarding a recent TechCrunch article detailing how a few developers test their Android apps.
Striving to uphold journalistic integrity and remain unbiased, TechCrunch writer Kim-Mai Cutler took the complaints to heart and reached out to more developers to get a fuller picture of Android QA practices. Here’s a snapshot of how four developers with successful apps do their testing (from TechCrunch):
Red Robot Labs (Veteran founding team from EA, Playdom and Crowdstar. More than 3.5 million downloads. They currently have the #27 top-grossing game in the Google Play store.)
Red Robot uses about 12 devices in-house and has a quality assurance team of two people. They then use a U.K.-based company called Testology to get further coverage with 35 handsets.
Pocket Gems (More than 70 million downloads. Newer to Android, but they had two of the top 10 grossing iOS games for all of last year according to Apple’s iTunes Rewind. #35 top-grossing game in Google Play.)
They use a little more than 40 devices evaluated in a matrix they explain in the video below. They make sure they include both tablets and phones and then high-resolution and low-resolution devices. They also make sure to include all five major graphic processing units (GPUs) including Adreno, PowerVR, Tegra, Mali and Vivante. …
For the San Francisco-based startup, quality assurance testing is a 24-7 process that involves teams both in the U.S. and abroad. After the U.S. team designs and performs tests during the day, they hand their work to an offshore team that has all of the exact same 40 or so Android devices. This team does extra compatibility testing overnight and files all of the bugs into a defect tracking system, which go back to the U.S. team in the morning.
Storm8 (Totally bootstrapped. Four games in Android’s top-grossing 50. Founders are early Facebook alums.)
Storm8 uses between 30 and 50 devices, which they divide into groups of high-end, mid-range and low-end devices. They intentionally buy devices for each category. After they launch games, they have the apps send back different KPIs (key performance indicators) back to the company’s servers.
“This way, we can tell if we need to further fine-tune a certain class of devices, or even specific devices, to squeeze the last bit of performance from the devices,” says chief executive Perry Tam.
Animoca (More then 70 million downloads.)
“If we had taken the approach that 90 percent compatibility is good enough, we’d be lacking support for 7 million of [our] downloads,” the company explains. “Several millions of consumers would have had a bad experience as a result of our decision, and our app revenues would probably be short by around 10 percent.”
Read more at TechCrunch >>>
No matter how you sugar coat it, the Android matrix is giant and complicated. But as you can see, each developer has their own approach that supports their widely successful apps. So remember, it may be confusing and daunting, but there’s more than one right way to cut the Android pie.