Techniques, Tips & Tools For Mobile App Testing

© 2015 SOASTA. All rights reserved. April 15, 2023 1

Brad JohnsonVP Product Marketing

@bradjohnsonsv

Session W12

Techniques, Tips and Technology for

Better Mobile App Testing

Vancouver, BCJune 24, 2015

48%TIME

37%EXPERTISE

35%PROCESS

?Says who

you.

250 Testers from 6 Continents

Mobile is your chance to make a CHANGE

Learn new TECHNIQUES

April 15, 2023 10© 2015 SOASTA. All rights reserved.

Seek TIPS from experts

Embrace new TECHNOLOGY

Learn new TECHNIQUES

REALITY always trumps speculation

Static aggregates (ok)


Real Time (best)


Data science - better than best!


ASAP

Simple

Time Consuming

Mundane

Rhythm

Goal : Continuous Integration execution

Know how to AutomateReturn person days to your test cycle


Continuous Integration Offers Speed of DeliveryWhat’s not to like?


o Small Batch Sizes

What’s not to like?

Continuous Integration Offers Speed of Delivery



o Small Batch Sizes

o Comprehensive Version Control




o Small Batch Sizes


o Simple Branch Strategies




o Small Batch Sizes



o Automation




o Small Batch Sizes



o Automation

o Continuous Feedback




o Small Batch Sizes



o Automation


o Emphasis on working builds




o Small Batch Sizes



o Automation



o Consistent environments




o Small Batch Sizes



o Automation



o Consistent environments

o Developer – Tester Collaboration



Device reaction to app functionality Battery, CPU, Memory

Dependent services and 3rd party apps/ads

Separate back and front end performance

User experience under every traffic condition

Performance to revenue correlation

Performance Is Everything!Every strategy needs perf coverage


Seek TIPS from experts


7 Steps to Pragmatic Mobile Testing


RetestTest Development Infra setup RepeatFixTest Analyze

TIME

Beware Automation Solutions that Hinder Speed

Developers code tests, but not faster.

The same delays with different people?


RetestTest Development Infra setup RepeatFixTest Analyze

TIME

Analysis is still slow & manual

The same delays with different people?

Beware Automation Solutions that Hinder Speed


Get Testers TESTING!

RetestTest DevelopmentInfra setup RepeatAnalyze

Develop plans Post-test ValidateTraditional Approach

Test Fix

/

Analyze

Fix

Test Dev Test/

Analyze

Fix

Test Dev Test/ Analyze

Fix

Test Dev Test/Analyze

Fix

Test Dev TestModern Approach

OR/

Analyze

Fix

Test Dev Test /

Analyze

Fix

Test Dev Test

AUT/device setup Manual


250 hours of manual tests compressed to 1 hour

Regression tests not run until feature complete now run nightly 4 months to overnight

25% faster Time To Market


“(Mobile test automation) has been critical in improving both the speed to deliver and the quality of our mobile apps.”

“We decreased our test cycles from three weeks to a couple of days.”

3 WEEKS 2 Days

Embrace new TECHNOLOGY

RUM = “What to

Test”

(real user monitoring)


Mobile Test Automation Criteria

o Rapid & accurate test development

o Reliable, maintainable test automation

o Stability across mobile OS platforms

o Device performance measurement

o Real-Time feedback (via Continuous Integration is a +)

Keeping up with mobile development


Rank active devices

Device usage

Weight matrix to device

Know problem areas per device

Device Access Private Public

Know your devicesBuilding a Matrix



Source: Mobile Test Coverage Index (Perfecto)


Private Device Clouds

o Behind the firewall

o Always available

o Online access for all your teams

o 24x7, secure, remote access to devices

o Manual & automated testing Hands on when you need them.


Public Device Cloudso Dedicated

o Latest in-market devices

o Maintained for you

o Secure

o API’s to install apps, reboot/wake-up, launch tests

Perfect for automated mobile regression testing


Analytics

Database

Main

SUT/AUT

Web Server

Web Server

App ServerWeb Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

CDNCredit Card

Web analytic

s

Other Svcs…

Test the Back End

Performance Tester(s)

Virtualized services

3rd party services

Load Testing Tool


Analytics

Database

Main

Analytics

Load GeneratorLoad

GeneratorLoad Generator

SUT/AUT

Web Server

Web Server


Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

CDNCredit Card

Web analytic

s

Other Svcs…



3rd party services

Cloud Servers

Test the Back End

Load Testing Tool


Analytics

Database

Main

Analytics

Load GeneratorLoad


SUT/AUT

Web Server

Web Server


Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

CDNCredit Card

Web analytic

s

Other Svcs…



3rd party services

Cloud Servers

Test the Back End

Load Testing Tool


Analytics

Database

Main

SUT/AUT

Web Server

Web Server


Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Analytics

Load GeneratorLoad


CDNCredit Card

Web analytic

s

Other Svcs…


Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,….

3rd party services Virtualized services

Cloud Servers

Real time analytics

Test the Back End

Load Testing Tool


Continuous Mobile Testing

Utilize Real User data to drive dev & test specifications



I JUST MADE A CODE CHANGE

CI runs unit tests





CI runs unit tests


Spin up suitable, consistent test environments in the cloud

or the lab



Mobile automation runs in remote or on premise devices


CI runs unit tests



or the lab



Iterative load tests run as CI tasks



CI runs unit tests



or the lab






CI runs unit tests

Daily (or more frequent) updates keep teams quality focused

Embedded failure details & bottleneck data isolate issues



or the lab






CI runs unit tests

Manage to a performance

baseline

Daily (or more frequent) updates keep teams quality focused

Embedded failure details & bottleneck data isolate issues



or the lab


?EXPERTISE


?TIME


?PROCESS


Mobile Test Automation

Techniques, Tips & Tools for Mobile



Real TestDevices




Performance Testing

Real TestDevices




Performance Testing

Real TestDevices

Solid Integration


Mobile is your chance to make a CHANGE

You CAN do it!


Resources:

SOASTA.com

o Whitepaper: 7 Steps to Pragmatic Mobile Testing• http://info.soasta.com/Whitepaper-7-Steps-to-Pragmatic-Mobile-Testing.html

o July 7 Webinar “Strategies for Mobile Automation Success”• http://info.soasta.com/strategies-for-mobile-test-automation-success.html

o Recorded webinars - e.g. “Pragmatic Mobile Testing”• http://www.soasta.com/webinars/

Others

o “Future of Testing” (search Stickyminds/Techwell)

o “Mobile Test Coverage Index” (search Perfecto)

See “7 Steps” Session @ 3:00pm!

http://info.soasta.com/Whitepaper-7-Steps-to-Pragmatic-Mobile-Testing.html

http://info.soasta.com/strategies-for-mobile-test-automation-success.html

http://www.soasta.com/webinars/


@bradjohnsonsv bjohnson@SOASTA

Brad Johnson