Embed Size (px)
DESCRIPTION
Crowdsourcing has become widely acknowledged as a productivity solution across numerous industries. However, for companies incorporating crowdsourcing into existing business practices, specific issues must be addressed: What problem are we trying to solve? How do we control the process? How do we incentivize people to achieve our goals? Ultimately, the key to successfully employing a crowdsourcing model is to move beyond the realm of the “mob” to create an engaged, interactive community of diverse and skilled professionals. In the world of quality assurance, crowdsourcing has the potential to effectively solve emerging challenges and take your testing to new heights. Using real-world examples, Matt Johnston explains how you can leverage the crowd to complement your internal systems, ensure systems work as intended under real-world conditions, and effectively manage the scalability of testing efforts.
Citation preview
T18 Concurrent Class
10/3/2013 1:30:00 PM
"Get Testing Help from the
Crowd"
Presented by:
Matt Johnston
uTest Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Matt Johnston
uTest
With more than a decade of marketing experience at companies ranging from early-stage start-
ups to publicly traded enterprises, Matt Johnston leads uTest's marketing and community
management efforts. Since joining uTest as employee #8 in late 2008, Matt has provided a
technical sensibility to uTest’s engagement of testers and engineers. As CMO, he leads uTest’s
strategic initiatives, such as partnership formation and helping to define uTest’s expansion
strategy.
9/19/2013
1
|
Getting Testing Help From The CrowdMatt Johnston | CMO/CSO, uTest | [email protected] | @matjohnstonOctober 2013
The Challenge
||
App Quality
Then vs. Now
9/19/2013
2
|
The Way Things Were
You Tell Them What To Think
|
The Way Things Are TODAY
They Tell You What They Think
9/19/2013
3
|
Mobile App Quality Is Judged More
Harshly
• Users less tolerant of
spotty quality
• Social & app stores
give every user a
megaphone
• Switching costs
lower than ever
• Cost of imperfect app
quality has spiked
|
So What Could Possibly Go Wrong?
9/19/2013
4
|
Cars
E-ReadersHousehold Appliances
Fragmentation of users, devices, OSes, browsers, locations
Oh, And It’s Not Just Mobile
Phones & Tablets
Laptops & Desktops
Connected TVs
Gaming Consoles
|
• Inside-out quality perspective is valuable but
incomplete
– Regardless of mobile app type or dev & test methodologies
• Replaced by outside-in quality worldview
– Quality signals come in many forms, and from USERS
- Bug reports
- Test cases executed
- Crashes
- User feedback
- Analytics
– And at many times (continuous)
No Turning Back Now
9/19/2013
5
|
• External Signals Inform Internal Decisions:
– What you think matters less than what users think
– How YOU measure app quality matters less
– What users feel and say matters MORE
– Court of public opinion rules
- App stores
- Social media
What’s That Mean?
|
Tools For Outside-In App Quality
InInInIn----HouseHouseHouseHouse
Test Test Test Test
LeadersLeadersLeadersLeaders
AppsAppsAppsApps
ThatThatThatThat
WinWinWinWin
9/19/2013
6
The Challenge
|| 11
Ummmm7
So What Do We Do?
|
Stuff Gets Built; Then It Breaks
1. Marketing identifies a need
2. Product specifies a solution
3. Engineering builds the product
4. QA tests the application
5. Help desk listens to complaints
6. Sales & execs blame everyone
7. Then we do it all over again and
hope for better results7
12
Vicious Cycle
9/19/2013
7
| 13
Lab-Based Testing Innovation
Agile vs.
Waterfall
Manual vs.
Automation Managing vs.
Leading Teams
Exploratory vs.
Test Cases
Testers vs.
Engineers
In-House vs.
Outsourced
Offshore vs.
Nearshore
|
Mob Mentality
BUT crowds often look (and act) like unruly mobs
14
9/19/2013
8
|
From Mob to Community
And mobs don’t work in every category
Example: a skilled service like software testing or
requires an orderly and professional “community”
capable of consistently delivering the desired results
15
|
The challenge is to turn this –
16
9/19/2013
9
|
Into this
17
The Challenge
|| 18
The Crowd:
Testing In-the-Wild
9/19/2013
10
|
Community Management 101
• What Good CM Isn’t
– It’s not customer service
– It’s not marketing or social media
• What Good CM Is
– 1st gen � reactive & narrowcast (customer service)
– 2nd gen � proactive & broadcast (marketing)
– 3rd gen � independent & self-serve (program management)
– 4th gen � interdependent & vetting, teaching and policing one
another (community management)
19
|
• CM goals – to build relationships, develop loyalty and nurture desired behavior
– Get beyond 1:1 discussions
– Get beyond reactive corrections
• CM Challenges – to do the following at scale
– Recruiting � Vetting � Engaging � Incenting � Managing
• Tool Kit
– Forums & message boards > email responses
– Help topics & tutorials > repeatedly answering same questions
– Webinars > phone calls
– Profiles
– Matching
– Ratings
– Reputations
– Payments
20
Community Management Tool Kit
9/19/2013
11
|
• Top c-sourcing firms use two forms of compensation
– Monetary
– Reputation
• Performance-based ratings based upon a dozen factors:
– Participation Level:
- Lifetime & recent participation: # active test cycles, # reported issues
– Quality of Participation
- Approval percentage for bugs, test cases and usability surveys
- Accuracy of bug type & severity classifications
- Ability to write test cases & create automated test scripts
21
Rating & Reputation System
The Challenge
||
uTest Case Study:
USA Today
9/19/2013
12
|
• Timeline of USA Today Pioneering
– 1982: 1st nat’l paper to use full color
– 1996: Early adopter of web
– 2008: 1st major news brand with native apps
- iOS, Android, Windows, Kindle, web TVs, gaming consoles
– 2012: Re-designed & re-launched entirely new UI/UX across all
- Print + website + 14 native apps
- First web exceeded print; now apps exceed web
At the Forefront of Innovation
|
• Great for USA Today’s readers
• Great for USA Today’s sales & marketing
• Vast pressure on tech execs to deliver quality across:
– Device makers & models
– OSes & browsers
– Carriers
– Languages (localized content)
– Locations (geo-based intelligence)
Impact on Testing
9/19/2013
13
|
USA Today’s Testing Portfolio
In-House
Manual
Testing
Outsourced
Manual
Testing
In-House
Test
Automation
Outsourced
Test
Automation
In-The-Lab
Testing
The Challenge
||
The Goal:
High Bar For App Quality
9/19/2013
14
|
• Maintain Quality
– Launch nothing less than a 4.5 star app
– Each new version, across 14 native apps
• Increase Test Coverage
– Real-world conditions
– Cost-effective
– Scalable
• Meet Tight Deadlines
– Must have apps ready to win
- For each new platform that gains traction
- For each new device that gains traction
The Goals
The Challenge
||
The Approach:
Achieve In-the-Wild Testing
9/19/2013
15
|
• Traditional Outsourcing
– Lives in a lab
- Further removed from users
- Geo + demo
– Cheaper vs. in-house
– Hidden costs
- Time zones & language
- Scaling concerns and costs
– High noise-to-signal ratio
- No transparency
Two Distinct Choices
• Crowdsourcing & expertsourcing
– Lives beyond the lab, in the wild
- Professional testers + real-world conditions
– Mirrors USA Today’s user base
- Real-world conditions
- Tech + geo + demo
– Blended team
- Recurring testers (incl on-site) + crowd testers
- Dedicated uTest project manager
– Scales to match shifting needs
|
• In-the-wild testing solution for USA Today
– Provide on-demand complement to in-house QA team
– Reduce escape rate by testing beyond the lab
- Across OS, browsers, devices & carriers
- Across locations, languages, industries & hobbies
– Blended test team built into USA Today’s dev & test lifecycle
- Dedicated uTest project manager
– Tester management
– Triaging services
- Team of dedicated testers (remote or on-site) + large team of crowd testers
Their Choice: Partner with uTest
9/19/2013
16
The Challenge
||
Results:
Apps That Win
|
• Reduced escape rate
– ID’d and documented 1,000+ bugs across 6 apps in first 90 days
• Improved usability
– Several iterations of usability testing & analysis
• Crowd details
– Testers from US, Canada, Brazil, UK, France, Germany, China & Japan
– Some projects (eg: brand new iPad app) involved 80+ testers
– Others with tighter scope required smaller teams of 10-12 testers
• Hit the ground running
– Launching Kindle Fire app same day the device launched
– Hand-picked testers pre-purchased various models of device
– Tested their app with all over the world, almost instantly
• Bottom line: better apps
– Enabled USA Today to keep innovating, while maintaining superior quality
– USA Today able to consistently meet 4.5 star app-store rating goal
– 2012 Mobile Excellence Award – Best User Experience
Results
9/19/2013
17
|
Customer Satisfaction
The Final Word
“uTest gives our organization a much better sense of where we
stand in terms of application stability and performance. For a group
that iterates as often as we do, that type of insight is critical”
-- Tim Carlson,
Director of Mobile Product Development, USA Today
The Challenge
||
Crowdsourced Testing
Specific Takeaways
9/19/2013
18
|
Key Takeaways: Crowd Testing
• Apps universe has forever changed testing
– Exponentially more diverse user environments
– Devices & configurations
– Locations
– User demographics
• In-the-wild testing is a necessity
– Some testing must be done beyond the lab
– In-the-wild where app users live, work & play
• Agility is crucial to app success
– Test team size and tester location � enterprise-friendly mobile testing
• Partner selection is vital
– Not all created equal
– Alignment of capabilities, needs and philosophy
– Track record of relevant enterprise success
The Challenge
||
Crowdsourcing
OverarchingTakeaways
9/19/2013
19
|
Understand The Job To Be Done
The Challenge
|| 38
Questions?
Answers
Matt Johnston | CMO @ [email protected] | @matjohnston
