Upload
techwellpresentations
View
294
Download
1
Embed Size (px)
DESCRIPTION
In today’s market, global outreach, quick time to release, and a feature rich design are the major factors that determine a product’s success. Organizations are constantly on the lookout for innovative testing techniques to match these driving forces. Crowdsourced testing is a paradigm increasing in popularity because it addresses these factors through its scale, flexibility, cost effectiveness, and fast turnaround. Join Rajini Padmanaban as she describes what it takes to implement a crowdsourced testing effort including its definition, models, relevance to today’s development world, and challenges and mitigation strategies. Rajini shares the facts and myths about crowdsourced testing. She spans a range of theory and practice including case studies of real-life experiences and exercises to illustrate her message, and explains what it takes to maximize the benefits of a crowdsourced test implementation.
Citation preview
MH AM Tutorial
4/29/13 8:30AM
Implementing Crowdsourced Testing
Presented by:
Rajini Padmanaban
QA InfoTech
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Rajini Padmanaban
As director of engagement at QA InfoTech, Rajini Padmanaban leads the engagement and relationship management for some of QA InfoTech's largest and most strategic accounts. Rajini has more than eleven years of professional experience, primarily in the software quality assurance area. She actively advocates software quality assurance through evangelistic activities; writes on test trends, technologies, and best practices; speaks at conferences including STAREAST 2012 and STARWEST 2012; and provides insights on software testing to analyst firms such as Gartner and IDC. Read Rajini’s official blog and reach her at [email protected].
Corp04 1
Your Software Testing PartnerWe help you build better software
Implementing Crowd Sourced Testing
Rajini PadmanabanDirector of Testing Engagements
Agenda (1 of 2)
Slide 2
Topic Time (in
minutes)
A Peek into Software Quality 10
Crowd Sourced Testing - Defined 10
Understanding Varied forms for Crowd Sourcing 15
Let’s be the Crowd – Exercise Time 20
Crowd Sourced Testing Relevance in Current
Scenario
10
Limitations of Crowd Sourced Testing 15
Corp04 2
Agenda (2 of 2)
Slide 3
Topic Time (in
minutes)
Practices for a Successful Crowd Sourced Test
Effort
30
Know What Not to Crowd Source 15
Getting Stake Holder Buy-in 15
Case Study & Examples 30
Myths and Facts 15
Conclusion and Q&A 15
Requirements
A device to connect to the internet for one of our exercises (laptop, smart phones, tablets )
Slide 4
Corp04 3
Test is responsible for
QualityFocused on creating a quality deliverable
Ensure we don’t sacrifice quality for the sake of schedule
Empower the rest of the team to partake in improving quality
A Peek into Software Quality
Slide 5
Defining Quality
• ISO 9000: Degree to which a set of inherent characteristics fulfill requirements
• Six Sigma: Number of defects per million opportunities
• Philip B. Crosby: Conformance to requirements
• Joseph M. Juran: Fitness for use by the customer
• Gerald M. Weinberg: Value to some person
• Robert Pirsig: The result of care
• American Society for Quality: A subjective term for which each person has his or her own definition. In technical usage, quality can have two meanings:
a. The characteristics of a product or service that bear on its ability to satisfy stated or implied needs
b. A product or service free of deficiencies
Slide 6
Corp04 4
Software Development
Slide 7
Attacking the Inverse Exponential
TestabilityFault InjectionStress TestingEnd-to-EndExploratoryCode InspectionLeverage the Beta Test Crowd
Slide 8
Corp04 5
Portfolio Selection Theory
Diversification in investing tells us that risk lessens as the number of investments in the portfolio increases
10%10% 10%
10%
10%
10%
10%
10%10%
10%
Slide 9
Audience Interaction Time=
What Does Quality Mean to you ?
Slide 10
Corp04 6
What is Crowd Sourced Testing?
Dailycrowdsource.com
Crowd sourcing is the process of getting work, usually online, from a crowd of people. The word is a
combination of the words 'crowd' and 'outsourcing'. The idea is to take work and outsource it to a crowd
of workers
CrowdSourcing.org (similar definition on Wikipedia)
Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that
exploits the benefits, effectiveness, and efficiency of crowd sourcing and the cloud platform towards
software quality assurance and control. With this new form of software testing, the product is put to test
under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all,
bug-free
Crowd Source Testing.com
Crowd sourcing your software testing consists of delegating onto a number of internet users the task of
testing your web or software project while in development to ensure that it contains no defects, referred
to as bugsSlide 11
I prefer the
word engaging
rather than
delegating
What is Crowd Sourced Testing?
.so what really is Crowd Sourced Testing?
• Is it all about a pool of testers leveraged and paid per valid bug? No, not just that
• Think:• Sourcing relevant people from within your company
• Across disciplines and levels• Sourcing end users from various disciplines – e.g.
teachers, students, nurses, bankers• Partnerships with universities, organizations leveraging
domain knowledge• In essence, think of the community at large to test your
productSlide 12
Corp04 7
What is Crowd Sourced Testing?
Slide 13
Services
CompaniesProduct
Companies
External
Crowd
Sourcing
Internal
Crowd
Sourcing
Technologies DomainsCompany Scale
Understand Factors that Motivate the Crowd
Portfolio Selection Theory
Crowd Sourced Testing: Portfolio Selection Theory applied in QA
10%10% 10%
10%
10%
10%
10%
10%10%
10%
Slide 14
Corp04 8
A Practical Solution to Leverage
Slide 15
Set # 1
• Crowd Sourcing:• Is not restricted to any single company, technology,
domain• Has a much larger yet simpler meaning than what it is
often portrayed to be
Slide 16
Corp04 9
Types of Crowd Sourcing - Explained
Slide 17
Crowd Creation
Crowd Voting
Crowd Wisdom
Crowd Funding
Types of Crowd Sourcing - Explained
• Crowd Creation
• Invite crowd to create subject content• Common usage areas: software
development, translations, photos repository• Content usage by crowd or by organizations• Typical crowd motivators – Money, fun,
community involvement, brand loyalty• Well known examples – Linux, iStockPhoto,
99 designs
Slide 18
Corp04 10
Types of Crowd Sourcing - Explained
Slide 19
Types of Crowd Sourcing - Explained
• Crowd Voting
• Leverage crowd’s judgement to organize, filter, stack rank content
• Common usage areas: retail, media, simple yet powerful decisions across domains
• Results used by an organization• Typical crowd motivators – Fun, community
involvement, brand loyalty• Well known examples – American Idol,
Threadless.com
Most popular of crowd
sourced versions
(1:10:89 rule)
1% create
10% vote and rate
89% consume-Jeff Howe, Author of CrowdSourcing
Slide 20
Corp04 11
Types of Crowd Sourcing - Explained
Slide 21
Types of Crowd Sourcing - Explained
• Crowd Wisdom
• Harnesses crowd’s knowledge to solve problems, predict future outcomes
• Common usage areas: reality shows, quality assurance, exchanges markets
• Results used by an organization, crowd• Typical crowd motivators – Money, fun,
product transparency, brand loyalty• Well known examples – Who wants to be a
Millionaire
Slide 22
Corp04 12
Types of Crowd Sourcing - Explained
Did You Know?
The crowd’s answer in
“Who Wants to be a
Millionaire” was right 91%
of the time compared to
“Ask an Expert”, which was
right 65% of the time
Source: The Wisdom of
Crowds, James Surowiecki
Slide 23
Types of Crowd Sourcing - Explained
• Crowd Funding
• Leverages crowd to finance individuals or groups that might otherwise be denied credit or opportunity
• Common usage areas: ideas in developing nations, educational domain
• Results have far reaching impact• Typical crowd motivators – Money, social
causes / community involvement• Several interesting examples at:
http://www.alumnifutures.com/2012/07/crowdsourced.html
Slide 24
Corp04 13
Set # 2
• Crowd Sourcing Options are:• plentiful• available for a diverse set of community needs – both
commercial and not-for-profit• to be customized based on results needed and the
crowd motivating factors
Slide 25
Exercise Time – Let’s Crowd Source
Slide 26
Who has won the most oscars?
1.Walt Disney
2.Elizabeth Taylor
3.Meryl Streep
4.Jack Nicholson
Walt Disney – he won 26
oscars including 4 in a
same year
Corp04 14
Exercise Time – Let’s Crowd Source
Slide 27
On which national flag is there an eagle and
a snake?
1.China
2.Mexico
3.Greece
4.Spain
Mexico derived from an Aztec
legend that a city would be built
where they spot an eagle eating a
serpent, which is now Mexico City
Exercise Time – Let’s Crowd Source
Slide 28
Which American state produces most
potatoes?
1. Oregon
2. California
3.Idaho
4.Washington
Idaho, followed by Washington
and Oregon (in 2011)Source:
http://www.potatopro.com/Newsletters/20111110.htm
Corp04 15
Did You Know?
The crowd’s answer in
“Who Wants to be a
Millionaire” was right 91%
of the time compared to
“Ask an Expert”, which was
right 65% of the time
Source: The Wisdom of
Crowds, James Surowiecki
Set # 3
Collective wisdom of the crowd often surpasses that of an expert
• Please take about 10 minutes to test www.amazon.com from usability and accessibility standpoints
• Need tools - screen readers, magnifiers?• http://en.wikipedia.org/wiki/List_of_screen_readers• http://mediaaccess.org.au/digital-technology/assistive-
tech/screen-magnifiers/ - e.g. Magnifier on Windows, Zoom
for Mac
Slide 30
Exercise Time – Let’s Crowd Source
Corp04 16
• Discuss test results / feedback –• Goal of our study: demonstrating richness
and diversity of crowd’s feedback as end users
• Feedback from our visually challenged crowd
Slide 31
Exercise Time – Let’s Crowd Source
Amazon
Accessibility Issues
• Shrinking release cycles• Close scrutiny on overall spend• Collective ownership of quality• Need to:
• Focus on product domain knowledge; not just disciplinary knowledge
• Understand competing products• Creatively emulate end user scenarios• Mimic user environments via lab and simulations• Consider global product distribution effects
Slide 32
Product Quality Scenario – as it stands today
Corp04 17
A Practical Solution to Leverage
• Distribute quality effort • Enhance productivity through global solutions• Pool in end users into quality implementation• Use live environments to test• Analyze potential partnerships for SME• Flexible, selective and cost effective testing
.........Bring in the Crowd
Slide 33
When does Crowd Sourced Testing
Succeed?
• Diversity of knowledge, background, experience
• Independence in testing process
• Wide spread domain background for product requiring SMEs
• End user scenarios and environments difficult to simulate in-house
• Work aligns with factors that motivate crowd
Slide 34
Corp04 18
Inherent Challenges
Slide 35
• Random test efforts do not fit into quality strategy• Choosing, sustaining a crowd sourced team
• Ongoing motivation, floating crowd• Keeping everyone in sync on product dynamics• Communication challenges• Management overhead including logistics• Stakeholder buy in• Securing product IP before release
Any Challenges from
your experience that
you would like to share?
Perceived Limitation
Slide 36
Quality of product adversely impacted by an amateur crowd
- Solidify your implementation plans- Use instrumentation wherever possible- Refer case studies – understand successes and failures
Corp04 19
Set # 4
• Current product scenario demands creative solutions to test within existing constraints
• Crowd Sourcing is not a no-brainer solution to all problems• Not even a stand-alone solution in most cases• Understand its strengths and challenges in customizing
it to your needs• Understand “What, When and How to Crowd Source in
Testing”
Slide 37
Practices for Successful Crowd Testing
Customize your
practices mindful of
your constraints
Slide 38
Corp04 20
Slide 39
• When to Crowd Test:• Product works reasonably well E2E • Ready to incorporate crowd’s feedback• No time or resources for formal testing .try to
avoid this situation
• Source content files are ready• Ongoing feedback from a chosen SME team
at specific stages
“What, When and How” of Crowd Testing
Slide 40
Examples
Started off with informal testing due to
lack of timeOngoing MVP programs at Microsoft
Product ready to be tested E2E and
feedback incorporated
Corp04 21
Slide 41
• What to Crowd Test:• User facing features• Areas where external team feedback is
important – e.g. design, feature set, performance
• Specialized areas of test:• localization (context based verification)• performance• compatibility, devices testing
• Content testing – valuable SME knowledge
• Align crowd’s focus areas into test strategy• Know what not to crowd source
“What, When and How” of Crowd Testing
Slide 42
Examples
Exam Grading
Content Testing using SME /
Localization
Corp04 22
Slide 43
• How to Crowd Test:• Pick the right areas, team and time
• Minimize duplication, overhead of sifting throughknown issues
• Clear internal ownership:• Communication, technical query resolution• Prompt follow up and responses• Team up-to-date on product changes
• Use of collaborative tools• Think about interactions amongst testers
“What, When and How” of Crowd Testing
Slide 44
• How to Crowd Test:• Leverage cloud, VPC for ease, secured access• Identify crowd motivators
• Use management theories – Maslow’s hierarchy of needs?
• Identify tasks that align with crowd motivators• Work on stakeholders buy-in
“What, When and How” of Crowd Testing
Self Actualization – Pursue Inner Talent,
Creativity, Fulfillment
Self Esteem – Achievement, Mastery,
Recognition, Respect
Belonging – Friends, Family, Spouse, Lover
Safety – Security, Stability, Freedom from Fear
Psychological – Food, Water, Shelter, Warmth
Corp04 23
Set # 5
• Understand when a crowd test effort succeeds:• Diversity of knowledge, background, experience• Independence in the testing process• Wide spread domain background for product requiring SMEs• End user scenarios and environments difficult to simulate in-house• Work aligns with factors that motivate the crowd
Any Examples, Best
Practices from your
experience that you
would like to share?
Slide 45
Slide 46
• Features with moving pieces, demanding close collaboration
• Sensitive IP• Environment specific complex testing• Tasks requiring immediate and regular turnaround
- BVTs• Core testing activities
• Test automation, TDD scripts• Regression testing • First round of performance, security, integration testing• Mundane testing tasks that don’t need diversity
Know What Not to Crowd Test
Corp04 24
Slide 47
• Simple performance tests such as PLTs – use tools• Understand pros and cons specific to your scenario –
study by University of Texas, Austin* on Crowd Sourcing for Usability Testing vs. Lab Usability Testing on a college website
Some Examples
Lab Usability Test Crowd Sourced Usability Test
Participants 5 55 (14 spammers)
Participant Demographics Students Crowdworkers
Age 24 to 33 19 to 51
Education level Bachelor’s degree and Master’s degree All levels
Experience with similar
websites
Yes: 100% Yes: 77%
No: 23%
Speed Approximately 30 min. per session. Less than 4 hours total.
Participant
Costs
None $2.92 for pilot test
$23.41 for final test
(Avg: $0.48/tester)
* - http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf
Slide 48
Some Examples
Major Problems Identified Lab Usability Test Crowd Sourcing Usability Test
Font size too small
Out-of-date information
Menu overlap
Irrelevant picture
Invisible tools
Information not cross-linked
Lack of sort function
Navigation unclear
Search box difficult to locate
Corp04 25
Slide 49
Some Examples
Advantages Disadvantages
More participants Lower Quality Feedback
High Speed Less Interaction
Low Cost Spammers
Various Backgrounds Less Focused User Groups
In Conclusion:
1. In this scenario, usability testing would be better off done by students of the college website;
choosing target users is very important
2. If crowd sourced testing is relevant based on your user profile, design the effort with care; using
same tests and questions as that of lab testers for crowd testers may not yield great results (see
section 4.1.2 on Test Redesign)
Slide 50
Stake holder Buy In
• Stakeholder resistance to Crowd Sourcing largely inline with model’s challenges:• Product IP, privacy issues• Internal team motivation• Quality of test effort and product• Additional overhead in effort management• Randomized test effort – tactical in nature
Corp04 26
Slide 51
Stake holder Buy In
Identify stakeholders; prioritize the team;
engage early
Walk through the test strategy including crowd
sourcing plans; understand their
concerns
Explain Solutions Undertake Pilot if
needed
Ongoing Communication
1. Supplemental test technique
2. Acknowledge problems
3. Explain the technique as
applicable to your product; don’t
assume their know-how
1. Educated decision
2. Explain implementation plans –
what & what not, when, how,
3. Define checks and balances for
internal team and CS team roles
4. Map solutions to address each
concern identified earlier
1. Practical demonstration of the
model; adds to your confidence too!
2. Additional overhead at start but
pays off in longer run
3. Run a pilot like a regular CS test
program but of smaller scale
1. Show steady progress
2. Stick to pre-defined
communication protocols
3. Communicate the good and the
bad
4. Be on top of new stakeholders
or new concerns
Set # 6
• Acknowledging what not to crowd source will:• Fasten stakeholder approval process• Help not impact internal team motivation
Slide 52
Corp04 27
Core
Work Skills
Unique
Work Skills
Future
Work Skills
In-Job
Behaviors
Organizational
Citizenship
Behaviors
Successful Work Patterns
Slide 53
Core
Work
Skills
Unique
Work
Skills
Future
Work
Skills
In-Job
Behaviors
Organizational
Citizenship
Behaviors
Successful Work Patterns
Slide 54
Corp04 28
Localization Testing
• Hard, Large-Scale Problem
• Windows 7 ships in 100
languages
• Thousands of strings and screens per release
• Traditional model of localization testers expensive and difficult to find
Slide 55
link
Let’s Play
Slide 56
Corp04 29
The Language Quality Game
Slide 57
Results
Significant Quality Improvements for Windows 7
Positive Impact on Ship Schedule
Team Morale and Subsidiary Engagement
Internal sourcing alleviates security and access issues
Total Screens Reviewed: Over 500,000
Total Number of Reviewers:
Over 4,500
Screens per Reviewer: Average 119
Slide 58
Corp04 30
Amplify Skill with Volume
Individual dialects,
nuances, hard to
detect with a single
vendor – crowd does
a better job
Slide 59
Reduce Cost with Discovery,
Instrumentation
No need to install
Telemetry to direct
effort
Slide 60
Corp04 31
Reduce Risk with Diversity
Portfolio theory in
Quality Assurance
Slide 61
Trust and Transparency Increase
Effectiveness
Inclusion = Trust, Trust = Enthusiasm
Inclusion = Trust, Trust = Enthusiasm
Subsidiary
engagement with
Language Quality
Game
Momentum for Win 7
Slide 62
Corp04 32
Players earn points for disaster relief agencies
Microsoft donates $ based on leader board
Individual players can sponsor tasks or scenarios
Slide 63
Results
Significant Quality Improvements for Communicator “14”
Positive Impact on Ship Schedule
Team Morale and Dog-food User Engagement
Players Over 1,000
Feedback increase: > 16x
Feedback received: 10,000
Slide 64Source : Ross Smith, Director of Test, Microsoft Corporation
Corp04 33
Other Examples
Slide 65
• Working with Blind Relief Association to bring in real users into accessibility testing
• Working with universities for content grading• Mobile application testing across devices within
the company
Other Examples
Slide 66
Listen to Ross Smith, Director of Test at Microsoft,
on his thoughts on Crowd Sourced Testing
Corp04 34
Crowd Sourced Testing – Myths and Facts
Slide 67
Crowd Sourced Testing impacts core testing team adversely; threatens their positioning
in the product team
Really?
1. Crowd Sourced Testing - supplemental technique, not stand
alone
2. Does not work in all situations; areas of niche to be reserved for
internal testing
3. Internal team to build a sense of empowerment that the crowd
adds to product quality
# 1
Crowd Sourced Testing – Myths and Facts
Slide 68
Crowd Sourcing is only for software testing. Development is a very technical and
specialized area to leverage the crowd for
Really?
1. Think of Open Source Software….Linux - popular example of
open software collaboration
2. Crowd Sourcing easier for testing than development as crowd
often represents user base
3. Security, IP easier to manage in testing – no access to source
code
4. Practices discussed extendible to development as well
5. Top Coder.com another good example
# 2
Corp04 35
Crowd Sourced Testing – Myths and Facts
Slide 69
Management overhead is significantly higher in Crowd Sourced Testing. Given the short
project deadlines, we do not have time or resources to manage a crowd sourced test
team
Partly True…
1. Management overhead slightly more...but this is inevitable in
current day global development models
2. Crowd is a smart and self – sufficient group; YOU DON’T WANT
TO MICRO-MANAGE
3. Best practices of what, when, how to crowd source will make
the effort streamlined and not chaotic
# 3
Crowd Sourced Testing – Myths and Facts
Slide 70
Crowd Testers can be ramped up or down at very short notice, giving great head count
flexibility
Partly True…
1. Scale potential of resources is huge
2. However, getting right resources at right time is challenging
3. Maintain a pool / common database and engage with the crowd
on ongoing basis even in lean periods
# 4
Corp04 36
In Conclusion
Slide 71
Let’s:• Pictorially walkthrough Crowd Sourcing
• Revisit Take-Aways• Look at Call to Action
Crowd Sourcing – Pictorial Walkthrough
Slide 72
Corp04 37
Crowd Sourcing – Pictorial Walkthrough
Slide 73
Crowd Sourcing – Pictorial Walkthrough
Slide 74
Corp04 38
Take-Aways revisited
• CS used across companies, domains, technologies
• Understand varied manifestations, crowd motivators, to customize your implementation
• Educated decision of what, when, how to crowd source
• Acknowledge model’s challenges; be transparent in seeking stakeholder approval
Slide 75
Call to Action
• Evaluate programs in your group; gradually build on them
• Start small• Try a Pilot / Proof of Concept• Register to be a crowd
sourced tester – internally, externally -http://www.qainfotech.com/CS_Reg_Step1.php
Slide 76
Corp04 39
About QA InfoTech
• An independent software quality assurance and testing company, founded in 2003, currently employing 600 people
• Five testing “Centers of Excellence” across the USA and India
• World-class testing labs
• Experience working with clients across various domains
• Bagged the “Top 100 places to work for in India*” award, three years in a row
• Focus on the right balance of people, processes, technology • CMMi III, ISO 9001:2008, 20000-1:2005 certified
QA InfoTech facilities in IndiaSlide 77
* Study conducted by Great Places to
Work Institute, India
Q&A – Let’s find answers together!
Slide 78
Corp04 40
References
• http://outsideinmarketing.wordpress.com/2010/05/08/four-types-of-crowdsourcing/
• http://explore2win.blogspot.in/2012/10/what-is-crowdsourcing.html
• http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf
Slide 79
Thank You
For more information, please:
• Contact us at [email protected]
• Visit us at www.qainfotech.com
• Read our blog at www.qainfotech.com/blog
• Follow us on Twitter at www.twitter.com/qainfotech
USA
Office
International
Headquarters
NoidaUttar Pradesh, India Phone: +91-120-4292222(Three additional testing facilities in India)
Farmington Hills Michigan, U.S.A. Phone: +1-248-719-3409
Slide 80