State of Mobile Testingjason arbonCEO, Appdiff

Jason Arbon CEO, Appdiff

How GoogleTests Software

App Quality

Problem

Fix?

Solution

Continuous Integration

CI (cloud): Commit Frequencies

15% less than 5 per week

33% between 5 and 25 per week

33% between 25-50 per week

19% more than 50 per week (~ 10 times a day)

CI: Survey: Top Priorities

48% Improve app performance

41% Faster release/update frequency

11 % Increase security/vulnerability testing

CI: % Builds with Tests?

Only 48% of CI Projects had any Tests

Not much Automated Testing

in Mobile

Testing: Hardest problem in Computer Science

Testing

Manual Testing For Mobile

Manual Testing For Mobile

Few Similarities

Zero (for startups)

Dev-only

Beta only

In-house

Often <200, 2000 for banking/etc.

Beta Testing Growing in importance and value

Continued rise of Crowd Testing

AutomationAwkward Teenage Years (still)

Unit tests

Maybe Unit tests

Only a few

Most products use < 2minutes of CPU for their unit tests!

UI Automated

Tests

Many try. Most fail

App changes cause test breaks

Maintenance quickly destroys ROI of UI automation

Often <50 tests. >150 is rare.

DevicesVs

Em/Sim-ulators

Hype cycle: Hardware, then pain, then simulators/emulators

Good news: Google/AMZN device clouds

Perfecto: Expensive, but most complete solution with device access

Smarter: Use engagement metrics for device matrix

Device cloud acquisitions

Good. Subsidized cloud services

Still complaint about connectivity

Perfecto and device anywhere high price but manual access and best of breed

On premise

Few happy with solutions

Cost

Lack of devise matrix . Most debug on hard to find devices

Support insufficient

Automation Hype Curve

Automation ROI Curve

Test Authoring

Developing test cases is the bottleneck

Developers Don't have time thanks to CI/CD/Agile

Teams can’t afford testers. Those that can...can’t hire or retain them.

Talent

Unicorns: Hard to Find Developer+Tester

Tornado: Harder to retain as they move into Development roles

Expensive!

Frameworks

Espresso and Xctest Solid, but platform specific

Appium still note quite stable, but best long term bet for cross-platform re-use

Calabash ok

Trend? Test Dev paid more than Developers

Production APM

APM

Server Performance Monitoring

In-App Performance Monitoring

Crash Collection

Engagement/Usage

SDKs

Image from http://www.analyticshero.com/

SDKs

Install SDK + Code: App and Server

Maintain/Update

Can cause crashes and perf issues themselves

Several in same app can conflict

AppDynamics

Twitter Fabric

New Relic

Runscope

Limitations of Production APM

Issues found *after* user impact

Difficult to interpret graphs, charts, dials

Measure API and Code--Not User Perception

Pre-Production, User-Focused APM

Why Pre-Production, User-Focused APM?

Catch issues *before* user impact

Measure user-perceived performance

Zero-Code, Zero SDK.

With CI, know which code change impacted performance

Peek at the future

We still ‘Hand-Craft’

Missed leveraging the availability and pricing changes in compute, networking and storage.

Tooling is still very much the same: unit test frameworks, mocks, and ‘selenium’-like DOM driving frameworks.

We still ‘hand-craft’

Future is Machine learning + bots

Only hope to deal with the exponential complexity is lots of smart compute

Kick the can down the road about a mile

Large, Centralized, Dedicated Test Team (80%)

Rainforest: 80% automated/AI in 2years

AI / Machine Learning for Testing

Input

Validation

Alerting/Prioritizing

AI / Machine Learning for Testing

Feature

Bug

Size

H/W

Contrast

Appdiff

Bot as Test Infra Interface

jason@appdiff.com

www.appdiff.com