Mobile App Assurance: Yesterday, Today, and Tomorrow

A foundation of quality

Robert V. Binder

Director of Innovation, Olenick and Associates

[email protected]

IMBUS Software QS Tag — November 8, 2012

The Big Picture

http://apod.nasa.gov/apod/image/0710/puzzle_cook.jpg

Overview

• The mobile deluge

• Mobile app testing retrospective

• Mobile app assurance challenges

• State of the art, 2012

• Crafting a mobile app assurance strategy

• Q&A

THE MOBILE DELUGE

SECURELY

RELIABLY SEEMLESSLY

MOBILE APP TESTING

RETROSPECTIVE

© 2004 mVerify Corporation 9

Mobile Test Technology, 2002

Typical

Mobile App,

End to End

Mobile

Switching

Center

Public

Switched

Network

Internet

BackBone

First Tier

Server

Business

Server

Base Station

Mobile User

Device

Parametric

Desktop

Load Test

Desktop

Serial Port

Interface

Critical Capabilities

Application Server

Functionality

Response Time

Server Resource Utilization

Billing/Provisioning/Security

Background contention

Dispatch/Allocation

Background IP Load

Client transaction saturation

End-to-End

Response time

Capacity

Reliability

Availability

Geographic Coverage

Handheld

Functionality

Response Time

MUD Resource Utilization

Airlink Variation

QOS Edge Combinations

In-cluster Hand Offs

Multiple Base St Protocol

Roaming

Location Services

Server Interaction

Server Exception

Configuration

Mobile Infrastructure

Op/Admin/Maintenance

Background load (“breathing”)

Packet Load

Weather, solar, etc.


Hand Held Testing

Mobile

Switching

Center

Public

Switched

Network

Internet

BackBone

First Tier

Server

Business

Server

Base Station

Mobile User

Device

Handheld

Functionality

Response Time


Airlink Variation




Roaming

Location Services

Server Interaction

Server Exception

Configuration

Base Station



Packet Load


Application Server

Functionality

Response Time




Dispatch/Allocation

Background IP Load


End-to-End

Response time

Capacity

Reliability

Availability

Geographic Coverage

•No Load

•No Mobility

•No Interaction

Transport Layer

Network Layer

Data Link Layer

Physical Layer

Parametric Testing

Handheld

Functionality

Response Time


Airlink Variation




Roaming

Location Services Server Interaction

Server Exception

Configuration

Base Station



Packet Load


Application Server

Functionality

Response Time




Dispatch/Allocation

Background IP Load


End-to-End

Response time

Capacity

Reliability

Availability

Geographic Coverage

Mobile

Switching

Center

Public

Switched

Network

Internet

BackBone

First Tier

Server

Business

Server

Base Station

Mobile User

Device

•Single connection

•No Back-end

•No Server

•No App Function

•Mobility & Load ?

Server Testing

Mobile

Switching

Center

Public

Switched

Network

Internet

BackBone

First Tier

Server

Business

Server

Base Station

Mobile User

Device

Handheld

Functionality

Response Time


Airlink Variation




Roaming

Location Services

Server Interaction

Server Exception

Configuration

Base Station



Packet Load


Application Server

Functionality

Response Time




Dispatch/Allocation

Background IP Load


End-to-End

Response time

Capacity

Reliability

Availability

Geographic Coverage

•No Mobility

•No Network

•Emulated Handheld

Mobile

Switching

Center

Public

Switched

Network

Internet

BackBone

First Tier

Server

Business

Server

Base Station

Mobile User

Device

End-to-end: Manual/Live Network Handheld

Functionality

Response Time


Airlink Variation




Roaming

Location Services

Server Interaction

Server Exception

Configuration

Base Station



Packet Load


Application Server

Functionality

Response Time




Dispatch/Allocation

Background IP Load


End-to-End

Response time

Capacity

Reliability

Availability

? Geographic Coverage

•Inconsistent

•Can’t Scale

•Time Consuming

•No Load

•Mobility (Drive test )

AMATE Project

Advanced Mobile Application Test Environment Goal: Achieve realistic end-to-end mobile testing

Model-based, Mobile-Centric

Signal variation related to mobility

User behavior related to mobility

Traffic related to mobility

NIST/ATP funded R&D

2002-2004


Load Model

• Vary aggregate input rate

• Arc

• Flat

• Internet fractal

• Negative ramp

• Positive ramp

• Random

• Spikes

• Square wave

• Waves

Actual “Waves” Loading

Signal Propagation Map

Mobility Model

1 Bar Signal

2 Bar Signal

3 Bar Signal

• Map generates real time itinerary for N virtual users • Location-specific signal strength • Location-specific end-user behavior • Controls Airlink Emulator

Virtual Users

© 2006 mVerify Corporation A Million Users in a Box ®

The Mobile Testing Nightmare

• Intense, high-stakes race to

market

• Configurations (platforms x

devices x airlinks) increase

exponentially

• More testing necessary for

competitive quality,

reliability, performance

• Ad hoc manual testing is

slow, costly, ineffective

© 2007 mVerify Corporation A Million Users in a Box ®

MTS: Any App, Any Platform

Agent Host

Client Host Under Test

Agent HostConsole Host

MTS

Console

MTS Test

Agent/STE

MTS Test

Agent/STE

Host Under Test may be

TEST RUN

REPORTS

MTS Remote

Agent

Client

Under Test

Server Host Under Test

MTS Remote

Agent

Server

Under Test

ü Cell Phone, PDA

ü Desktop or Server

ü Embedded Processor

ü Comm Interface

ü Network Equipment

ü Access Point

ü Base Station

Client Host Under Test

MTS Remote

Agent

Client

Under Test

Server Host Under Test

MTS Remote

Agent

Server

Under Test

Plug In

Plug In

MTS Model

Editor MTS

Repository

Channel

Emulator

MOBILE APP ASSURANCE

CHALLENGES

Functionality and Robustness

Using Launchpad 2.6 with a mobile

app, when the app is deactivated then

reactivated, all the elements of the

home view list are duplicated.

When the phone rings, the incoming

call screen pops up with options

accept, ignore and ignore-with-text.

Selecting ignore-with-text doesn’t

disconnect, pops up two more times,

and doesn’t send the text.

Same App, many platforms, locales …

• Android

• BlackBerry

• iPhone, iPad

• Mobile Web (HTML5)

• Windows Mobile

• Dozens of languages

Configuration Coverage

• Form factors

• Screen sizes

• Platforms and versions

• Accessories

• Wireless stacks

• Bandwidth

• Carrier policies and technology

• Logo/Store Certification

• Locales/localization

• Server-side performance

• How many ways can your

app be deployed?

Usability

Security

Scalability: Amazon Cloud Outage

• Amazon web service Elastic Compute Cloud crashes (power failure)

• The Elastic Load Balancing (ELB) system frantically tries to assign workloads to available servers.

• As Amazon’s cloud rebooted, “a large number of ELBs came up in a state which triggered a bug we hadn’t seen before”

• ELB reaction: try to allocate more larger servers.

• Backlog in the “control plane” results.

• Demand from customers in unaffected availability zones continues

• System swamped and crashes again.

Latent Positive Feedback = Dragon King

Sornett calls these exceptional events dragon

kings “to stress that we deal with a completely

different kind of animal, beyond the normal,

whose extraordinary characteristics [have]

profound significance.”

Dragon Kings

Here be Dragons

ü Latent positive feedback

External disruptors

Malicious attack target

ü Partial degradation

Provoked panic

ü Low testability

Multi-stack

Field-infeasible

Uncontrollable Inputs

STATE OF THE ART, 2012

Handheld Testability

• Many UI Test Tools

• UI event capture/replay

• Image capture/bitmap compare

• API for widgets/controls

• Invasive

• Brittle

• Several test suite composers

• Drag and drop

• Natural language/keyword

• No multi-endpoint capability

• Most are platform-specific

Crowd-sourced Testing

• Crowd Testing

• UTest

• MobTest

• Mob4Hire

• Others

“60787 people (mobsters) have 34142

different mobile handsets on 448 carriers

in 156 countries”

Test Environments as a Service

Remote Device Array

SUT

Mobile Client Emulation

Scalable Client Emulation

Live Target

SUT

Testing Technology for Mobile apps

What’s new

• Testable UI APIs

• Crowdsourced testing

• Cloud-based Test in

Production

• More stable infrastructure

What’s not new

• UI capture/replay

• Image capture/compare

• Virtual users

• Hand-crafted test suites

• Massively manual testing

• Remote device array

• Fragmented test harness

• No Model-based Testing

CRAFTING A MOBILE APP

ASSURANCE STRATEGY

System of Programs •Interfaces

•End-to-end

Program Systems Product

Program Product •Testing

•User Support

•Maintenance

3X 10X

3X

Brook’s Scope

Program

Tiered App •Interfaces

•End-to-end

Mobile System Product

Published App •Testing

•User Support

•Maintenance

3X 10X

3X

Brook’s Scope, Today’s Environment

One Billion Smartphones

2.5 Billion Endpoints

Mobile App

State of the Art

• Handheld testability

• A little better

• Mobile testing nightmare

• Remote Device Arrays

• Crowdsourced testing

• End-to-end coverage

• Infrastructure much more robust

• Capacity much improved

• No end-to-end test harness

• No support for Internet of Things

• Systems are bigger, more complex, and more critical

• Hic Sunt Dracones

Mobile App Strategy

“The best answer from 3 candidates with

over 10 years of claimed mobile application

testing experience listed on their CV, and all

ISEB qualified was, and I quote:”

“It’s more easy on the mobile app".

Hiring Manager:

“To what should you pay special attention

when testing a mobile application, in

comparison with say a standard web

application?”

Mobile App Strategy

• Design for testability

• Minimize variations

• You’ll need tooling for each

platform

• Cost of failure is very high

• App Store reject adds months

• No second chance with users

• Space is moving very fast

Mobile App Strategy

• Cover features and events

• Use and Abuse Cases

• Event Plan, pairwise

• Manual, automated, RDA

• Cover Configurations

• Remote Device Array

• Crowdsource if necessary

• Capacity Test

• Cloud Test Services

• Assess Dragon King Risk

• Model-based multi-dimensional

UC01 UC01 UC02 … UC99

Foregrnd/Backgrn

Background load

Power Sleep Cycle

Battery Drain

Incoming Call

Incoming Text

Camera Usage

Bar code scan

Accelerometer

Reboot

GPS Impairments

WiFi Impairments

Cellular Impairments

Content and Image Sources

Big Picture

http://apod.nasa.gov/apod/ap071021.html

How do people want to use their devices

http://network.cisco.com/how-do-people-want-to-use-their-devices.html

Internet of Things

http://blogs.cisco.com/news/the-internet-of-things-infographic/

Functionality … Configuration Coverage, Event Plan

http://www.udemy.com/how-to-test-mobile-apps

Usability

http://www.pagetrafficbuzz.com/google-survey-reveals-75-mobilefriendly-sites/14736/

NativeDriver

Matt DeVore, Tomohiro Kaizu, Dezheng Xu, Daigo Hamura. Native Driver Native App UI Automation with WebDriver API, 2011 Selenium Conference, San Francisco.

http://www.nativedriver.googlecode.com/files/NativeDriver_introduction.pdf

Amazon Outage

Hidden bugs that made Amazon Web Service outage worse. Cnet News, July 3, 2012.

http://news.cnet.com/8301-1009_3-57465761-83/hidden-bugs-that-made-amazon-web-service-outage-worse/

Dragon-Kings, Black Swans and the Prediction of Crises. Didier Sornette. International Journal of Terraspace Science and Engineering 2009

http://www.youtube.com/watch?v=FlTSbzOvKZI

Latent Positive Feedback = Dragon King

http://www.everythingselectric.com/forum/index.php?topic=244.0

Dragon King

http://wallpaper4me.com/wallpaper/Dragon-King-of-The-Land/

Crowdsourced Testing

http://mob4hire.com/

Hic Sunt Dracones

http://tomlytle.com/

All other content

Copyright © 2012, Robert V. Binder

http://apod.nasa.gov/apod/ap071021.html




























































































Why are mobile apps so popular?

• Low cost

• Any time, any where, any one

• Connectivity to everything

• Complete control

• Personal space

Current Technology

Test Goal Plus Minus

Functionality and

Robustness

Simple Tools for all

platforms

Stack Silos

Brittle testware

Deployment Coverage Crowdsourced Testing

Remote Device Array

Systematic coverage?

Superficial

Usability Crowdsourced Testing

Security Abuse Cases Manual/technical

Scalability Proxy and cloud load

testing

Happy paths can’t find

Dragon Kings

Network of Things No framework for adapters

Sophistication Platform IDEs Mostly manual

No Model-based Testing

Attitudes Seen as “easier”


Mobile Device

Testing with AMATE

Model Builder

Simulator

Digital Airlink

Emulator

Test Driver

AUT

WiFi

Access Point

1 Generate Model

2 Generate Test Objects

4 Test Driver Controls AUT

DALE Controls Airlink

Test Agent

3 Run Test Objects

Then and Now Handheld M Infras Server E2E

Scalability 1 2 4 2

Mobility 1 2 0 0

Connectivity 1 1 5 1

Usability 1 3 4 1

Security 1 3 4 1

Controllability 1 2 5 1

Observability 1 2 5 1

Coverage 0 3 5 0

Handheld M Infras Server E2E

Scalability 4 3 4 2

Mobility 2 3 3 2

Connectivity 4 3 5 2

Usability 4 3 4 3

Security 2 4 4 2

Controllability 2 2 5 2

Observability 2 2 5 2

Coverage 4 3 5 3

0

2

1

2

1

1

1

3

Beyond Manual Testing

• Automated Testing

• Device side

• Server side

• Test management

• Seat, open source, cloud

• Automated

performance/stress

• Remote device sharing

• Crowd-sourced testing

services

© 2011, Robert V Binder. All Rights Reserved