Upload
doanh
View
220
Download
1
Embed Size (px)
Citation preview
Test and Verification Solutions
Crowd Testing
Mike Bartley - CEO
Delivering Tailored Solutions for
Hardware Verification and Software Testing
Copyright TVS Limited | Private & Confidential | Page 2
TVS - Global Leaders in Test and Verification
India - 2011
UK - 2008
Germany - 2011
France - 2012
Singapore - 2014
China
South Korea
Continuous
geographical
expansion…
USA - 2014
0
50
100
150
200
Q3-13 Q4-13 Q1-14 Q2-14 Q3-14 Q4-14 Q1-15 Q2-15 Q3-15 Q4-15 Q1-16 Q2-16(Est.)
Q3-16(Est.)
Q4-16(Est.)
EM
PL
OY
EE
S
CALENDAR YEAR
Number of Employees by quarter
Copyright TVS Limited | Private & Confidential | Page 3
The rise of the crowd
Users rate a service or products • Airbnb, Amazon, travel websites, ….
Using the Crowd as an Innovation Partner • Apple use large numbers of users and developers around the
world to create apps and podcasts.
Crowdfunding • In 1997 American fans of Rock band Marillion used the internet
to raise $60,000 so they could play in the US.
• In 2012, there were over 500 crowdfunding platforms
• Global crowdfunding market to reach $34.4B in 2015 (Massolution 2014 prediction
The crown as testers • uTest incorporated in August 2007 with $2.3M in seed funding
Copyright TVS Limited | Private & Confidential | Page 4
99 Tests
Partnership deal with T&VS
Over 15K testers
Over 50 countries
Over 200 clients worldwide
Over 85K bugs found
Copyright TVS Limited | Private & Confidential | Page 5
Crowd Testing Basics
“No matter who you are, most of the smartest people work for someone else”
Joy’s Law – by Sun Microsystems co-founder Bill Joy
“Given enough eyeballs, all bugs are shallow”
Linus Law – by Eric S Raymond in “The Cathedral and the Bazaar”
Copyright TVS Limited | Private & Confidential | Page 6
Crowd Testing Basics
Testing by anonymous ‘crowd’ of testers
Testers are rewarded by identifying bugs quickly, first to find the bug gets paid.
Testing can be limited by device, or open to any platform
Testers are keen to work out of hours 24/7
Limit testing by geographical region, or global
Pay for the bugs only
Copyright TVS Limited | Private & Confidential | Page 7
Crowd Testing Advantages - Speed
Test quickly and conveniently, 24/7 if required
Drive down testing times
Ramp up and down test teams quickly
Have test teams the right size for each stage of development
Copyright TVS Limited | Private & Confidential | Page 8
Crowd Testing Advantages - Costs
Reduce your test costs
Improve Cash Flow
Ramp up and down test teams quickly
Have test teams the right size for each stage of development
No need to invest in different platforms and devices
Copyright TVS Limited | Private & Confidential | Page 9
Crowd Testing Advantages - Quality
Improves Test platform coverage
Testing by real users in the real world
Testing in multiple languages and cultures
Pay Testers for Testing, Developers for Developing
Standard bug reporting
Remove internal bias
Copyright TVS Limited | Private & Confidential | Page 10
Crowd Testing Disadvantages
Mainly for customer facing applications
Limited control of staff experience
Not for Unit or Integration testing
May need manuals to be ready earlier
May need support teams in place
Control of Piracy
Management of the Services
Copyright TVS Limited | Private & Confidential | Page 11
Crowd Testing Hidden Costs
Management of the Crowd Test Process
Control of builds
Regression testing
Earlier ‘deployment’ Support teams
Better documentation may be required
Copyright TVS Limited | Private & Confidential | Page 12
Some questions to consider
Do you need Unit Testing?
Do you need Integration Testing?
Do you need a Strategic Test Plan?
How will you manage retesting?
How bug-free does it have to be?
What will you do with Test Results?
Copyright TVS Limited | Private & Confidential | Page 13
Test Results
Testing Approach • To test the web application, exploratory testing approach was used.
• Testers from various locations accessed the web application and tested the whole application using different exploratory approaches.
To ensure coverage of the major features • We shared the critical workflows of the website
Copyright TVS Limited | Private & Confidential | Page 14
Test Results
Copyright TVS Limited | Private & Confidential | Page 15
Test Results
Copyright TVS Limited | Private & Confidential | Page 16
Test Results
Copyright TVS Limited | Private & Confidential | Page 17
Enterprise v Small companies
• Flex testing teams without
having to employee
additional staff
• Reach out to a broader
cross section of users
• Reduce the testing cycle
• Test Apps and Web Products on
every conceivable device in
days
• No need to divert development
teams to carry out User Testing
• Add independence to the
testing cycle
• Control testing costs
Enterprise Small/Startups
Copyright TVS Limited | Private & Confidential | Page 18
Major Companies are using it
Crowdtesting has already been used by likes of eBay, Amazon, GE, Microsoft, Google and Facebook to develop user-centric products.
Google regularly deploys crowd testing for 14 of their major product lines.
Copyright TVS Limited | Private & Confidential | Page 19
Typical questions to answer
Combination of devices i.e. OS & Browser versions, mobile platforms, Tablets to be tested.
Testing timeframe, number of days per cycle or total duration for testing being allocated.
Number of crowd sourced testers would you like testing your site, egg 25 – 50, 50 – 100
Estimate the number of testing hours you would like to achieve in the days allocated
Estimated number of defects you believe will be found as a result of testing
List bugs that you are already aware of
19
Copyright TVS Limited | Private & Confidential | Page 20
Typical questions to answer
Do you require a Test Plan to be written?
Will you provide Test Cases to be executed?
Would you like the crowd to provide you with: • Fully Detailed Steps - screen shots of each tests performed by Crowd (Screenshots / Video
capture of tests)
• Screen Capture - attached to defects only
Select specialist testers with a specific product / domain skills and expertise?
Confidentiality agreement / NDA with the Crowd testers required ?
Do you require Usability testing of your site? • Formal – Crowd actively is testing for usability and raising defects
• Basic – a Questionnaire is filled in at the end of the cycle to gauge testers feeling about usability Defects will not be raised.
20
Copyright TVS Limited | Private & Confidential | Page 21
Case Study of a Large MNC: Intuit
Intuit provides accountancy packages for small
companies, mainly its flagship application
Quickbooks
Intuit has lots of Small Teams building software
products. Each team is given limited resources. i.e.
their critical resource for them is time.
So, each team is tasked with creating innovative
software product that can potentially reach millions
of customers.
Crowd Testing frees up time for these teams. Thus
running a Crowd Testing campaign gives them
ability to compress their release cycles & also
improve the quality of their application.
21
Copyright TVS Limited | Private & Confidential | Page 22
Case Study of a medium MNC: Teligenz
Teligenz is a successful technology company that provides a range of creative services from web designing to App development.
The successful delivery of their projects depends greatly on the quality of the insights gained through App testing.
We leveraged on our talent pool of professional crowd testers from across the globe to implement effective functional testing for their Apps. The test cycles created were executed by professional crowd testers who were assigned to various projects after matching their skill set with the project requirements.
The efficient strategy enabled the entire test cycle for an App to be covered in just 3-5 days. The quick turnaround time helped the development team at Teligenz to make real-time decisions and focus product development in areas that needed their attention. They were able to improve their App quality exponentially.
22
Copyright TVS Limited | Private & Confidential | Page 23
Case Study of a small MNC: Reverie
Reverie’s cloud-based language-as-a-service platform enables businesses to connect more easily with their target audience by delivering dynamic and static content for apps in real-time in the languages that they need.
Testing the usability for language feature is not an easy task, especially when dealing with highly user-centric E-commerce Apps.
Reverie wanted to test the usability of the App for specific target demographics comprising of Tier 2 and Tier 3 Indian cities whose native language was Hindi.
We provided Reverie with testers across different demographics factoring in parameters like age, city etc. from all over India to gain a real life, end user perspective.
The focus on the human approach, helped developers gain valuable insights on how their target demographics would react to the App.
23
Copyright TVS Limited | Private & Confidential | Page 24
Case studies in progress
MovieR – a UK startup providing a social media multi-user video experience. iOS application being tested by 50 users over a 48 hour window
Pogo – a UK startup providing a photo comparison monitor for social networking. iOS application being tested by 50 users over a 48 hour window
24
Test and Verification Solutions
Questions?
Delivering Tailored Solutions for
Hardware Verification and Software Testing