Testing Techniques for Mobile Applications

Testing Techniques for Mobile Applications

Indira Pai, Basant Chandran

MangoSpring Technologies Pvt Ltd.

Introduction

• With the fantastic growth of mobile computing platforms there has been a dramatic increase in the value of mobile applications for most companies.

• One of the biggest challenges that one faces when developing a mobile application is how to test it.

• There are a number of factors to consider for mobile testing which aren't present for desktop or web.

Agenda

• Types of mobile applications

• Mobile testing generics

• Platforms

• Comparing testing on simulator with testing with device

Types of applications in mobile

1. Browser Based Applications

2. Pre Installed Applications

3. Installable applications

Browser Based Applications

• Applications built for mobile browsers

• Can be accessed by entering the specific URL in mobile browser.

• No Installation/Un-installations involved

• No Upgrade involved

Browser Based Applications (contd.)

• Browser Based Application users always expect connectivity• Network speed and coverage become an

important test case

• Speed is a critical aspect:• User entered values can’t be stored locally in

device database, until user allows storing information in Cache, he has to enter all his information manually every time.

Pre Installed Applications

• Applications which are shipped as in built software with the mobile device• If devices are not launched yet, a prototype of that

device model is given by the device manufacturer.

• No Downloads involved

• No Installation/Un-installations involved

• Automatic Upgrades can be done.• This is specific to application.

Pre Installed Applications (contd.)

• Testing of these applications in prototypes is very critical

• They deal with the very core data and functions of the device. • If anything goes wrong, the core database might

get corrupt.

• They cannot be uninstalled/deleted ever.

Pre Installed Applications (contd.)

• Crashes can cause severe damage to ROM

• Only way to remove these applications would be to flush ROM and then upgrade it.

Installable Applications

• Can be upgraded:– This is specific to the application design

– This also depends on the source from where application was downloaded

• Applications whose executable files/packages can be downloaded / received by wireless media or wired media.– Platform Specific

Installable Applications (contd.)

• Can be installed/uninstalled in the device

• They can be downloaded from:– App Store

– OTA (Over The Air)• A URL which leads to download of the application.

– Transfer via USB cable

• Can be transferred from wireless media like Blue Tooth, Infra Red etc.

Mobile Testing Generics

• Testing in various network strengths– No Network

– Low

– Medium

– High

– Testing during change of network strength from:• No Network/Low to High• High to Low/No Network

0

20

40

60

80

100

Defect in %

No Network

Low

Medium

High

Change instrength

Mobile Testing Generics (contd.)

• Testing in various network speed– Low

– Medium

– High

– Testing during change of network speed from:• Low to High• High to Low

0

20

40

60

80

100

% of issues

LowMediumHighChange in speed


• Testing in various network types:– 2G : • GPRS • CDMA • EDGE

– 3G

– Wi-Fi

– Different types of Plans based on service provider


• Testing in various Battery Strengths– Critical

– Low

– During Charging

– High

0

20

40

60

80

100

% of Issues

CriticalLowDuring ChargingHigh


• Monitoring Battery consumption patterns– Observe battery consumption rate as the

application is being• Run in Background• Run in Foreground

– Run the application for long time.


• Monitoring Memory usage patterns– Observe memory usage pattern as the application

is being• Launched• Run:

• In Background• In Foreground

• Exit

– Run the application for long time.


• Check memory usage patterns when in device apart from the pre installed applications – No other applications are installed• Lot of free memory available in device.

– Some third party applications are installed• Lesser free memory available in device

– Lot of third party applications are installed • Very less free memory available in device


• Memory Consumption Pattern (RAM usage)– Check memory consumption pattern when multiple

applications are running while the application is being:• Launched• Run

• In Background• In Foreground

• Exit


• Interruptions: – Activities which can occur parallel within the

device while application is being:• Installed• Launched• Run• Exit• Upgraded• Uninstalled/ Deleted


• Examples of Interruptions:• Incoming call

• Receiving incoming call

• Receiving message

• Device shutdown

• Remove battery

• Camera activated

• Lose network connectivity and then regain it


• Debug Build:– A very important asset in helping reproduce

“difficult to reproduce” bugs

– A build released in debug mode which has logs enabled.

– Enabling logs based on sequence of keys

– Logs can be read continuously, and can help retrace steps to any issue, provided application is running.


• Build Signing :• Required for a build which has to be pushed to an

app store. • Confirms that norms as per the authority of OS or

device manufacturer are followed.

• Testing round after build signing needs to be done• Ensures the final build being submitted is of

appropriate quality standard.

Platforms covered

• iPhone

• Android

• BlackBerry

• Symbian

• Windows Mobile

• J2ME Applications

Input modes in different platforms

• Touch Screen– Testing with single touch inputs– Testing with multiple touch inputs– Testing for Virtual Keypad

• Track Ball/ TrackWheel• Keypad• Test cases around multiple inputs at the same

time

Device Modes in different Platforms

• Portrait

• Landscape– UI should be properly

aligned as per mode

– Test cases around mode change when application is running in background.

iPhone• Additional Testing aspects:– UI Guidelines from Apple need to be adhered to.– Backward OS compatibility – Allows running only one application at a time.– Can debug via USB cable connection– Device Reboot not much needed until for clearing

cached data

iPhone – Testing Tools

• Screenshots:– Can be taken by clicking the Home key and the

Lock key together. – No other explicit tool needed

• Memory Sweep– A tool with which a memory snapshot of iPhone

device can be obtained at any given time.

Android

• Additional Testing Aspects:– Allows running multiple applications in background

– Application gets normally minimized on exiting. • So a reboot needed to launch application again.• Reboot needed often

Android - Debugging

• Select “Enable USB Debugging” in device settings.

• Then connect device to system via USB cable.

Android -Testing Tools

• Dalvik Debug Monitor Server (DDMS)– Port-forwarding services– Screen capture on the device– Thread and heap information on the device– Logcat– Process, and radio state information– Incoming call and SMS spoofing– Location data spoofing, and more.

Android -Testing Tools (contd.)

• Monkey Tool– Runs on simulator or device– Generates pseudo-random streams of user events

such as clicks, touches, or gestures, as well as a number of system-level events.

• TaskKiller– A utility application used to kill process. – Useful to skip device reboot.

BlackBerry


– Device Reboot • Not much needed until for clearing cached data• Needed mainly for while uninstalling/installing any

application.

• Build Signing/Verification– Signed/certified by RIM

BlackBerry – Debugging

• USB cable connected debugging

• Can be done via JDE console after attaching debugger

BlackBerry – Testing Tools

• Native Logs:– A combination on Alt+LGLG keys opens the

device logs in device.

– Has 3 types of filters:• Debug• Information• Error

– This helps analyze type of exceptions.

BlackBerry – Testing Tools (contd.)• JL_Cmder:– A little batch script to manipulate JavaLoader.– JavaLoader, a tool shipped with JDE, can be used

to easily:• Display Device Info• Display an event log• Take a screenshot • Wipe the handheld • Reset it to factory conditions

BlackBerry – Testing Tools (contd.)

• BBScreenshooter – Desktop Based tool

– Needs device to be connected to desktop system for capturing screenshots

• CaptureIt – Device Based tool

– Device need not be connected to desktop for capturing screeshots.

Symbian


– Device Reboot not much needed until for clearing cached data

• Build Signing/Verification– Signed/certified by Symbian

– By device manufacturing authority like Nokia

Symbian - Debug

• Can be connected to the IDE via:– USB cable connected debugging• More preferred

– Via BlueTooth• Less Preferred due to device driver issues

Symbian – Testing Tools• Screenshot tools:– All tools are device based– After installing the tool, • Set a shortcut key for screen capture. • Start the tool and then let it run in the background while

application is run.• Clicking on the shortcut key captures screenshot and

stores it in device. – Examples: • ScreenSnap S60 (5th Edition) • ScreenTaker for Nokia 7650/3650/N-Gage/SX1

Symbian – Testing Tools (contd.)

• Yucca Tools:– File browsing in Symbian device

– Crash Monitor

– Shows all supported fonts in device with preview.

– Memory status

Windows Mobile

• Additional Testing Aspects-– Allows running multiple applications in background

– Can exit the application from task manager• Needed in case application gets hanged in background

and its process needs to be killed

Windows Mobile - Debugging

• Debugging– USB Cable connection needed between device

and desktop

– Can use visual studio

Windows Mobile - Testing Tools

• Hopper– For stress testing.

– Throws random events such as click events, entering text if text field in application run in background etc.

– Logs are stored in device ,screenshot is also stored if device hangs or if application crashes.

J2ME Applications

• J2ME is a Development Platform

• Input Modes:• Follows input modes of the platform, some

restrictions might exist.

• Device Modes:– Based on platform

J2ME Applications (contd.)• Additional Testing Aspects:– For mass markets handsets– Porting Friendly• Should work well on the supported device/ platform on

which its installed.• Should be able to work well after the device firmware

upgrades.• Build Signing/Verification– Verified by Verifying Authority.– Verified by device manufacturing authority like

Nokia

J2ME Applications - Testing Tools

• Tools provided by device platforms can be used.

• Wireless Toolkit

• Debug mode build.

Testing on Simulator

Testing on Simulator (contd.)

• Simulators :– A software application that can accurately imitate

mobile phone.

• The need to test on simulators:– Non availability of device

– Testing same build in multiple device screens

– Non availability of certain network for device testing in testing area.


• Advantages:– Can help in isolating issues which are not volatile

network connection dependent.

– Can provide a wide variety of testing over different types of device simulators for the same build.


• Disadvantages:– Simulators of older generation handsets don’t

resemble the device as closely

– Some issues which are hit by the speed at which input was given cannot be reproduced easily

– Hardware/Firmware environment variations detectable in device testing only

– Device testing is always preferred as it represents more likely end user scenarios

Questions

Thank You

Technology

Testing Techniques for Mobile Applications