Android Testing Challenges From Netflix

We write a lot on the subject of mobile testing challenges specific to Android – with its countless devices, OS versions and other permutations. But for some reason, it means a lot more coming from a company like Netflix…

Amol Kher – the Netflix Engineering Manager in Tools for the Android, iOS and Apple TV teams – recently penned a terrific blog post on their first-hand experience testing their app on Android. I highly suggest that you read the post in its entirety, but here are a few of my favorite parts:

On their testing challenges:
“When Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: a) We wanted to release rapidly every 6-8 weeks, b) There were hundreds of Android devices of different shapes, versions, capacities and specifications which need to playback audio and video and c) We wanted to keep the team small and happy.”

On automation:
“You probably guessed that automation had to play a role in this solution. However automating scenarios on the phone or a tablet is complicated when the core functionality of your application is to play back videos natively but you are using an HTML5 interface which lives in the application’s web view.”
On device priority:
“To put it other way, when it comes to watching Netflix, any device other than those ten devices can be classified with the high priority devices based on their configuration. This in turn helps us to quickly identify the class of problems associated with the given device.”

On their testing team:
“We keep our team lean by focusing our full time employees on building solutions that scale and automation is a key part of this effort. When we do an international launch, we rely on crowd-sourcing test solutions like uTest to quickly verify network and latency performance.  This provides us real world insurance that all of our backend systems are working as expected. These approaches give our team time to watch their favorite movies to ensure that we have the best mobile streaming video solution in the industry.”

Read the rest >>>

Leave a Reply

Your email address will not be published. Required fields are marked *