The following post was written by mobile app tester Dom Wolf as part of uTest’s “Crash Courses” series. You can read similar posts in the uTest Forums (membership required).

Background: The release of the iPhone in 2007 introduced not only a new product, but also a new dichotomy to the world of software testing. The iPhone was the first mainstream phone to feature a touchscreen coupled with a 3G/Wi-Fi enabled web browser; two things that caused web developers to consider not only how their sites would appear, but also how they would be interacted with in such a different way.

Approximately one year after the launch of the iPhone, the iPhone SDK and App Store were introduced. The popularity of the iPhone at this point spurred many developers to submit applications at an unprecedented rate. Some apps were simple utilities, easily to develop and easy to test. Some, on the other hand, made use of many frameworks built within the newly introduced iOS and were incredibly complex. As the store’s popularity grew, Apple found itself on the receiving end of criticism over the content, and reliability of applications being submitted. The latter point is where we come in.

So there are two deliverables that are typically evaluated through uTest – native applications and mobile sites (this includes so-called web-apps, which are basically extensions to a mobile or web site). For the most part, the same test procedures can be applied to both types of deliverable, and this is what will be covered in this Crash Course. There may be some forms of testing which are deemed out of scope for a particular project you’re working on.

It’s paramount that you check the scope before you start testing and raising bugs. Not doing so could lead to having bugs rejected, which may affect your eligibility to take part in future releases. So, assuming that nothing is considered ‘out of scope’, we could include testing of the following:

• Installation
• UI and Usability
• Functionality
• Removal

Installation and removal typically only applies to third-party applications. The other areas could also apply to mobile sites/web-apps, and if there are any notable differences, I’ll touch upon those. There are other forms of non-functional testing that could apply to deliverables running on iPhones, such as security or performance. However, I’ve chosen to exclude these as

1. I’m not aware of any tools which could be used to check for application or web site security while running sessions on an iPhone *
2. Performance testing would essentially form multiple users carrying out steps in unison.

These may or may not be pre-determined steps. Either way, there isn’t really much to say about this that wouldn’t be covered in other sections of this Crash Course.

* One thing that can be checked manually is the use of SSL when logging onto a secure site (https://).

Installation

As previously mentioned, this only applies to native applications, as web-apps are simply extensions to a mobile site. They don’t comprise any installable components, such as dlls or executables, and rely on the iPhone’s Safari browser to run.

There are two (above-board) methods of installing applications onto the iPhone. The most common is via the iPhone App Store. Some clients release versions of their software available on the store for testers (and anyone else) to download. This can be achieved by either connecting the iPhone to iTunes running on a PC or Mac, or by using the in-build App Store. This installation process for beta software mirrors that which would be used for production software, giving application developers confidence that their end product can at least be installed during the beta testing phase.

The other installation method involves manually adding application files to your iTunes library (running on a PC or Mac), and then syncing the iPhone to this library. This method removes the App Store from the equation entirely, and is commonly used by developers before submitting their application to the store. This process involves the developer authorizing their application to run on your iPhone, and this is where your iPhone’s UDID comes in. Therefore, for uTest projects, you need to provide your UDID via the uTest platform, or email, depending on the uTest project manager’s request. This installation method is used during beta testing, and before the application is ready to be submitted to Apple. It allows a limited number of people to test the software early on to ensure proper installation and setup. However, this should never be used as the final installation test as that would be carried out via the official channel of iTunes / App Store.

UI and Usability

Considerations-

• Navigation – logical, progressive, easy to navigate, too much scrolling, intuitive, easy to use.
• Theme- use of fonts, consistency, colors, easy to read, issues with contrast,  are icons clear, size of buttons, font size.
• Help – Do I know where to go for help? Is Help actually helpful?
• Do aspects of the app/site (such as objects, or pop-up keyboard) obscure what I’m trying to do?

Functionality

Considerations-

• Test according to test cases/instructions if provided. use experience if not.
• Does copy/paste work as it should?
• Do zooming/landscape portrait work as they should?
• Invalid/valid input
• Check error handling- do error messages make sense? Are they defined?
• Negative testing first- bugs tend to collect around here
• Data use- does moving back and forward through app/site cause previously entered data to be lost? If so, should it?

Removal

To complete end-to-end testing of any form of application, it’s generally expected that removal and subsequent re-install processes are checked. Historically, the reason for this is that removal processes have been known to be incomplete, leaving files, such as dlls on a device. The issue here isn’t so much orphaned files wasting space, but that subsequent installs could fail.

The iPhone manages the removal process automatically when the user does one of two things. Firstly, the user can hold the phone’s ‘Home’ button, causing the applications’ icons to wiggle. A white cross in a red circle also appears beside icons of third-party apps, meaning they can be removed. Pressing the icon will prompt the phone to ask if the user wants to remove it. Selecting ‘Yes’ will do so.

This process can also be used for web-apps which appear on the Home screen. This simply removes the icon, as there are no application files to remove.

The alternative process is to deselect the app from those currently installed in iTunes. Once done, a sync will cause the app to be removed.

Unless your iPhone is jailbroken, you won’t have access to its internal file structure, so there is no way of validating that all application files have been removed. So there’s an element of trust in Apple’s removal process- in effect we’re black-box testing here.

Given the little effort involved in installing iPhone apps, and for the sake of completeness, it’s now worth re-installing the app we’ve just tested. This is also worthwhile when considering the potential impact if a re-install were to fail. What we’re doing here is checking that any orphaned files (if present) don’t conflict with those being installed on the device. For this reason, it’s not worth doing this for web-apps, since they don’t install files in the first place. It’s important to note that our focus here in re-installation testing, and not a second round of general functional testing. There’s generally little or no gain in going further than a simple install.