|

In-The-Wild Testing:Your Survival May Depend On It

Presented by Matt Johnston | @matjohnstonSTPCon, Fall 2011

| 2

Agenda

• Current State of Affairs• SoLoMo Boom• Surviving In-The Wild• Crowdsourcing 101• Secret Sauce• Key Takeaways

The Challenge

|| 3

How We Arrived AtThe Status Quo

| 4

Stuff Gets Built; Then It Breaks

1. Marketing identifies a need

2. Product specifies a solution

3. Engineering builds the product

4. QA tests the application

5. Help desk listens to complaints

6. Sales & execs blame everyone

7. Then we do it all over again and

hope for better results… Vicious Cycle

| 5

Testing Innovation

Agile Testing

Manual vs. Test Automation Managing

QA Teams

Exploratory vs.Test Cases

Partnering Testers & Engineers

In-House vs. Outsource

Games, Tours& Quick Attacks

In T

he L

ab

| 6

Testing Innovation

In T

he L

ab

InThe

Wild

| 7

To Make Matters Worse

• Users are less tolerant

• Social media & app stores give everyone a megaphone

• So their problem… is your problem

The Challenge

|| 8

Why It Matters More:The SoLoMo Boom

| 9

Enterprise Consumerization

• The Consumerization of Enterprise Computing– Technological disruption is pervasive– NO industry is exempt– B2B is B2C– Everyone’s a retailer

• Not just for games and personal use– Docs– Data– Publishing

• Not just your public site– CRM systems– BI & analytics tools– Productivity suites

| 10

Social: Real-Time Communication

• Enterprise social revolution isn’t coming… it’s here– Nothing is disconnected– No industry is immune

• The stats on social media adoption in Fortune 100 alone:– 77% leverage Twitter– 69% utilize LinkedIn– 61% enable Facebook– 57% incorporate YouTube

• Gartner: By 2016, social integrated w/ nearly all B2B apps

| 11

Social: Impact On Testing

• Enterprise adoption of social presents security & privacy hurdles – Internally, a channel for spam, malware and data breaches– Confidential data may be posted and trigger regulatory penalties– Externally, increasingly used for user authentication on your products

• And functional testing challenges– Testing code & integrations that aren’t yours– That constantly changes– And you will get the blame

| 12

Local: Can You Hear Me Now?

• Businesses use location for more than just marketing• Numerous applications for location-based technologies

– Asset tracking– Maps & routing– Location finders– Check-in services– Geo-based personalization

• Web and mobile have gone local– 1 of 5 searches has local intent– 1 of 3 mobile searches has local intent

| 13

Local: Impact On Testing

• Not just a mobile problem– Localization testing– Geo-based personalization

• But it is a mobile problem too– Apps used outside the confines of the

QA lab, under in-the-wild conditions

• When apps & users are distributed around the country or globe, a portion of testing should be too

| 14

• ABI Research anticipates worldwide enterprise mobile data revenues to reach $133 billion by 2014

• Enterprise mobility no longer just for email– Business apps: CRM, ERP, HR systems– Productivity apps: docs, spreadsheets, presentations– Collaboration apps: email, IM, publishing – Medical apps: health records, patient education

• Tablet-mania, both inside and out– Aflac– Mercedes-Benz– Wells Fargo– SAP

Mobile: Enterprise Impact

| 15

• Mobile web vs. native apps• Form factors matter again

– Feature phones– Smart phones– Tablets

• Ever-expanding HW & SW costs• Back to the future

– A return to the late 90’s web- Lack of automation or load tools- Lack of security or usability standards- Lack of dev kits or tools- Extremely fluid landscape

Mobile: Impact On Testing

| 16

Web Testing Matrix Is Tough

OS & Browser

Features

1. Coverage

| 17

Mobile Testing Matrix Is Insane

OS & Browser

Features

Handset Makers& Models

1. Coverage

|

Mobile Testing Matrix Is Insane

18

OS & Browser

Features

Handset Makers& Models

WirelessCarriers

1. Coverage

|

Mobile Testing Matrix Is Insane

19

OS & Browser

Features

Handset Makers& Models

WirelessCarriers

Location

1. Coverage

|

2. Managing Resources & Costs

• Multiple products, roadmaps & launch schedules• Managing peaks & valleys; unexpected twists & turns

20

|

3. Time-to-Market

• Never-ending race for each new app and version• Can exhaust the best test managers and teams

21

|

SoLoMo: The Bottom Line

• Becoming impossible for QA leaders to assure quality

22

The Challenge

|| 23

Argh!!What Do We Do?

| 24

• Keep testing in-the lab– Keep training– Keep automating– Keep innovating

• Recognize that it will never again be sufficient• Identify the right way for you to test in-the-wild

– Real-world conditions– Mirror your user base

- Technologically: OS, browser, anti-virus, device, carrier- Geographically: Continent, country, city, language- Demographically: Age, gender, education, employment, industry, hobby

– Maximize relevant testing coverage– Manage signal-to-noise ratio

From Surviving To Thriving

| 25

• Two ways to get there: Beta programs and crowdsourced testing

Testing In The Wild

| 26

• Pros:– No direct costs– Lives outside the lab

• Cons:– Users ≠ testers

- Rarely diagnostic- Poor signal-to-noise ratio

– Puts unfinished product in front of customers

• Many firms discontinued beta as a core part of QA– Hidden costs– Takes too long– Not actionable results– Didn’t yield higher quality

Beta “Testing” Programs

| 27

• Pros:– Lives outside the lab– Profiled, professional testers– Mirrors user base– Protects company IP– Much higher signal-to-noise ratio

• Cons:– Carries direct costs– Requires careful partner vetting– Requires buy-in from QA leadership

• Increasingly popular complement to in-the-lab testing

Crowdsourced Testing

The Challenge

|| 28

A Word AboutCrowdsourcing

| 29

Crowdsourcing Across Industries

| 30

Mob Mentality

BUT crowds often look (and act) like unruly mobs

| 31

From Mob to Community

And mobs don’t work in every category

Example: delivering a skilled service at an enterprise level of predictability and efficiency requires an orderly “community” capable of consistently producing the desired results

| 32

So how do you turn this –

| 33

Into this?

The Challenge

|| 34

SecretSauce

| 35

Tenet 1: Know Thy Task

| 36

Tenet 2: Know Thy Partner

• Selection criteria are vital:– Referenceable customer successes

– By company size– By industry

– Ability to adapt to your legacy systems and processes– Ability to satisfy legal requirements

– IP protection– NDA

– Ability to do the job– Consistent– Predictable– Professional

| 37

1. Community profiling– Technical– Geographic– Demographic

2. Community ratings & micro-ratings– By testing type– By app type– By industry

3. Highly precise matching– Between each project & each tester

Tenet 3: Know Thy Community

| 38

Tenet 4: Know Thyself

• Is your culture highly cautious & risk-averse?• Are you in a highly regulated industry?

– Defense industry– PCI, PII or PHI

• Do you have an appetite for innovation?• Are you centrally organized or decentralized?

– Sourcing– IT & IS– Engineering– QA & QE

| 39

Tenet 5: Know Thy Trends(Enterprises That Already Leverage The Crowd)

The Challenge

|| 40

EnterprisePerspective

|

Global power Google taps into the crowd to augment its in-house testing resources– Challenge:

- Tech execs sought ways to scale testing and achieve in-the-wild coverage- Wanted to ensure testing had real-world relevance to end-user experience

– Strategy:- Have invested heavily in outsourcing, in-house resources and test automation- Google was eager to find a better, more scalable approach to app testing- Google explored a wide variety of alternatives and found uTest

– Results:- After extensive 2009 pilot project, Google has since expanded its use to 18+ apps

– Web, desktop and mobile- Now leverages in-the-wild testing broadly & frequently, as part of core QA playbook- Relationship is pioneering in-the-wild testing as complement to enterprise’s in-the-lab

41

Google

|

Security Essentials – a free anti-virus app– Challenge:

- v1.0 product in entirely new category, so IP preservation was vital- Needed to test functionality under real-world conditions on vast matrix of hardware,

browsers, Windows OSes, and third-party applications- Needed testing coverage in strategic geo-locations (China, India, South America) to

mirror expected user-base and fill blind spots

– Strategy:- Recognized that it couldn’t solve this solely in the lab- Assemble a targeted team of testers to perform a series of exploratory and test case

execution to cover stated testing criteria

– Results:- After two months of continuous testing in ten countries on four continents – hundreds of

detailed bug reports, executed test cases, and user reviews had been prioritized- The company continues to run regression tests on the software to coincide with major

upgrades or new releases

42

Microsoft Security Essentials

The Challenge

|| 43

Bottom LineSummary

| 44

Key Takeaways

• Apps Universe Has Forever Changed Testing– Exponentially more diverse user environments

– Devices– Software configurations– Locations– User demographics

– Users less tolerant– Quality issues no longer private

• QA Must Play Catch-Up– By improving inside the QA lab– By moving beyond it, where your users live, work & play

– Beta programs and/or crowdsourced testing

– App quality winners are combining: in-the-lab testing + in-the-wild testing

The Challenge

|| 45

The End. Well, actually it’s just beginning…

(more at InTheWildTesting.com)

Matt Johnston | CMO @ uTestmattj@utest.com | @matjohnston