Upload
anthony-chung
View
174
Download
2
Tags:
Embed Size (px)
DESCRIPTION
The QA process overview for testing Firefox OS in 2014
Citation preview
2014QA Process Overview
5.14.2014
These Wonderful People!Who is your QA?
Roles QA Plays• Product Coverage
• Platform Coverage
• Stability and Performance
• Community
Roles QA Plays (cont..)• Assist development with Automation and Tree maintenance
• Active bug triage
• Device Flashing and Maintenance
• Partner and User support
• Internal Testing programs (Beta)
Product Coverage• QA representative per functional team
• Building and running Test cases on features based on user stories
• Daily coverage
• Bug analysis across branches (triage, qawanted, regression hunting)
• Automated and manual smoketests on device
• Hitting acceptance criteria
Device Automation
• Setting up 30 flame devices for lab automation
• Executing various tests: Gaia functional, power measurement, b2g performance (fps and cold launch), and marketplace
• Reporting to Jenkins server short term, long term Task cluster by way of Tree Herder
User story planning Acceptance defined
U definedTestcase creation Regression tests
cleanup
Partial Feature Testrun Regression TR1
Bug Bash Feature review FL Acceptance
Regression TR2 L10n TR1
MTBF Features exploratory Regression TR3
Bug Bash FC acceptance
Daily smoke tests / Ongoing Bugwork
!L10n TR2
Partner CS/ IOT stuff Internal certifications
CC acceptance
Testrun Metrics• 1.3 Case study
! •! % test coverage completed! ! ◦! run1 = 87% completed (12/09/2013)! ! ▪! 3360 passed, 413 failed, 218 invalid, 154 skipped = 4145! !! ◦! run2 = 98% completed! ! ▪! 3721 passed, 323 failed, 96 invalid, 0 skipped = 4104! !! ◦! run3 - 99% completed! ! ▪! 4034 passed, 262 failed, 121 invalid, - skipped = 4415! !! •! # of blockers found in test run (triaged for 1.3+)! ! ◦! run1 = 27 ! ! ◦! run2 = 20 ! ! ◦! run3 = 7! !! •! Test Failures / (#of tests * %completed) = Test Failure %! ! ◦! run1 = 413 / (4145 * 0.87) = 11.45%! ! ◦! run2 = 323 / (4140 * 0.98) = 7.96%! ! ◦! run3 = 262 / (4415 * 0.99) = 6.00%!
Testrun Metrics (cont)• 1.3 Exploratory
! •! # of blockers in exploratory run [search whiteboard = dogfood1.3]! ! ◦! 41 bugs (http://goo.gl/3dNUfl)! ! •! # of man hours on exploratory! ! ◦! 2 Weeks of 1/27 - ~11 testers! ! ◦! 40 hour week! ! •! Blockers per man hours = (# of testers * # hours / total # of blockers)! ! ◦! (11* 80 hours) / 41 blockers = 2.15 blockers! !!
Acceptance Criteria
• Product coverage Criteria listed for FL, FC, and CC milestones
• Platform and automation milestones are ongoing
• Tracked in: https://wiki.mozilla.org/Release_Management/FirefoxOS/Release_Milestones
• Working with different platform teams for backlog of test coverage (ie. WebRTC, graphics)
• Building out existing Mochitest coverage for Gfx / JS / Dom / Layout / Web Api
• Building test apps on device for Product QA to execute
Platform Coverage
• Q2 Goal: 72 hours uptime per device, no crash/ no hangs
• Running Hamachi (1.4) and Flame (2.0) [10 hamachis in parallel]
• Latest results (April 29th, 2014)
• Buri - 12 hours uptime per device
• Flame - 50 hours uptime per device
• Executed once a week, starting after FL
Stability (MTBF)
• Perf Metrics. establish a in-house baseline. analyzing results and building more automated tests
• Working draft: https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance
• Results through Datazilla and Eideticker tools
• Security testing? Seccomp builds? B2G fuzzing? working on these things in the future
Perf and Security
• With Flame, working with Foxtrot and other teams on focus areas
• Posting entry projects for device and automation tasks on public pages (oneanddone, badges)
• Bug bashes, local meetups
• Mentoring / code reviewing process
• More transparency with tests, reports, bugs, and testing opportunities
Community
Find us!• in your meetings, workweeks, offices, bugs, right behind you.
• https://wiki.mozilla.org/B2G/QA
• irc #fxosqa