|

Crowdsourced Testing Idea to implementation, complete with stories John Montgomery | uTest VP of Product Delivery | johnm@utest.com | September, 2013

|

Agenda

• Intro

• What is Crowdsourcing?

• Why do we need it for testing?

• Which Crowdsourced testing approach is

for you?

• Myths

• Our Approach

• Q&A

2

The Challenge

| | 3

uTest

Who we are, what we do

| 4

Who We Are

uTest is the world’s largest provider of

in-the-wild testing services

| 5

Types of Testing Services

Scalable, on-demand access to professional testers with the geographies and real-world infrastructure used by your customers and end users

On-demand test subjects and usability experts to provide comprehensive UX and user interaction testing services across most demographics and configs

Live and synthetic load and performance testing and analysis

White hat security experts providing automated and manual penetration testing from anywhere in the world

Real-world localization validation using native speakers from over 190 countries

| 6

Since Our 2008 Launch

1,500+

customers

100,000+ testers from 200 countries

& territories

Enterprises, SMBs & universities

Functional, load,

usability, L10N &

security

Web, mobile

& desktop apps

30,000+

test cycles

The Challenge

| | 7

Crowdsourcing

A Primer

|

Crowdsourcing Defined

“Taking a service traditionally performed by

a person or team and sourcing it to a large

online group of people as an open project”

- Jeff Howe

Wired journalist & author of Crowdsourcing

8

|

Who Offers Crowdsourcing?

9

|

Myriad Potential Benefits:

• Specialized skill sets

• Scale capacity to meet variable demand

• Increases fluidity of workforce

• Keeps fixed cost off balance sheet

What’s The Upside? And Why Wouldn’t They?

The Challenge

| | 11

In-the-Wild

Why Crowdsource Testing

|

Users Do Not Live In Labs

Your Test Lab Is Here

Your Users Are Here

Testing inside the lab is vital. But your customers and their

challenges do not necessarily live there.

Does this look familiar?

|

Test Matrix Are More Complex

• This is Android devices only

• Doesn’t account for OS versions, carriers or locations

|

A Lot More Complex

14

• Android alone (and that’s only by OS version)

|

Users Are More Vocal

• Users less tolerant of

spotty quality

• Social & app stores

give every user a

megaphone

• Switching costs

vanishing to zero

• Cost of poor app

quality has spiked

|

• Thousands of companies have moved a portion of testing

closer to where users work, live and play. Why?

1. Apps consumed under endless range of environments

- Exponential spike in devices, OSes & browsers

- Imperfect connectivity

- Infinite number of locations

2. Users have less patience than ever for buggy apps

- Switching costs lower

- App quality issues carry higher costs

3. In-the-lab testing alone cannot meet the demands of modern apps

Crowdsource Testing Can Help

|

Crowdsourcing Is In-the-Wild

18

In-House

Manual

Testing

Outsourced

Manual

Testing

In-House

Test

Automation

Outsourced

Test

Automation

In-The-Lab

Testing

The Challenge

| | 19

Crowdsourced Testing

Which is best for you?

|

How Do You Get From This

20

Crowds often look (and act) like unruly mobs

|

To This

21

Delivering a skilled service predictably requires a community capable of producing desired results

| 22

Choose Your Weapon

uTest Lives Here

|

Know Thy Task

• Automation

• Performance

• Usability

• Management

• Architecture

• Functional

• Localization

• Security

• Combination

of Group &

Specialist

• User panels

• Beta testing

• Data validation

• Ad quality

|

• What are your competitors doing to differentiate?

• What is your risk profile for quality?

• How do you compare against your competition?

Know Thy Competition

|

• Is your culture highly cautious & risk-averse?

• Are you in a highly regulated industry?

– Defense industry

– Pharma

– Banking

• Do you have an appetite for innovation?

• Do you have the budget to do it correctly?

Know Thyself

|

Which One Is Best?

26

Self managed White glove

Skills may vary

Any participant

Results may vary Exact results

Exact skills

Vetted

participant

The Challenge

| | 27

Lessons Learned

Myths, Misconceptions

|

Less In-House Staff = Less Overhead

TRUTH:

• Complement & scale “as needed”…not replace

• Only employees will fully understand strategy,

company position, internal processes

BEST PRACTICES:

• Build strengths around employees’ core competencies

• Where does the crowd ‘fit’ into your team?

– Fresh eyes, fresh perspectives, and fresh ideas

– Rote or repetitive tasks

– Outside of the lab

|

TRUTH:

• Intellectual property (IP) is a legit issue, esp. for the new &

innovative

– Copywriting

– Design

– Animation

– Code writing

• IP policies vary within crowdsourcing orgs

BEST PRACTICES:

• Establish the ground rules: What’s their IP policy?

– How is it communicated to the crowd?

– How is it enforced?

But we need to own the results

|

Tell the crowd? But we’re in stealth

mode!

TRUTH:

• Confidentiality is a serious matter with material consequence.

• Crowdsourcing is often used for pre-launch products

BEST PRACTICES:

• Ask upfront about preventative measures

– Precautions… NDAs? What else?

– Is the policy all-inclusive? e.g. incl. social media, message boards,

etc.

– What are the consequences for breaches?

|

Selection criteria are vital:

• Referenceable customer successes

– By company size

– By industry

• Ability to adapt to your legacy systems and processes

• Ability to satisfy legal requirements

– IP protection

– NDA

• Ability to do the job

– Consistent

– Predictable

– Professional

All Vendors Are Equal

The Challenge

| | 32

uTest & the Crowd:

Testing In-the-Wild

|

Focus On Your Needs

1. Community profiling – Technical: OS, browser, mobile devices, carriers

– Geographic: City, country, languages

– Demographic: Age, gender, education, hobbies

2. Community ratings & micro-ratings – By testing type

– By industry

3. Precise matching – Between each project & each tester

4. Services layer for clients – Onboarding

– Maximize throughput

– Minimize overhead

|

Not Restricted By Location

Diverse Community

Trained Community

Dedicated Offsite

Dedicated Onsite

34

• Expertise On Demand

– Who you need

– When you want

– Where you need

| 35

How It Works

Customers

uTest pays testers

Specify testing needs - OS, Browser, Carrier, Device - Location, Language & Industry

Customer approves testing results

Selected testers invited to project

Testers report in real-time

Testing Community

|

• Top c-sourcing firms use two forms of compensation

– Monetary

– Reputation

• Performance-based ratings based upon a dozen factors:

– Participation Level:

- Lifetime & recent participation: # active test cycles, # reported issues

– Quality of Participation

- Approval percentage for bugs, test cases and usability surveys

- Accuracy of bug type & severity classifications

- Ability to write test cases & create automated test scripts

36

Rating & Reputation System

| 37

• Better Applications

– In-the-wild testing – live testers, real devices, real-world conditions

– Unmatched coverage across large matrix of OS/browser/location

– Testers rated by customers

• Faster Time to Market

– Test cycles created in minutes; underway in hours; completed in days

– Test cycle can be executed overnight or during weekends

• Lower Total Cost of Testing

– Subscription based and project based contracts

– Lower total cost of testing than near-shore boutiques or offshore shops

• Scalability

– 100K + professional testers from around the world

– Scale size of test team up or down at will

How Our Customers Benefit

The Challenge

| | 38

Questions?

Answers