Upload
linaro
View
142
Download
1
Embed Size (px)
DESCRIPTION
LCU14 509- Overview of QA Reports --------------------------------------------------- Speaker: Milosz Wasilewski Date: September 19, 2014 --------------------------------------------------- ★ Session Summary ★ The session will cover: Means to collect the data from Jenkins and LAVA End-to-end result presentation (build status, build warnings, automated testing, manual testing) Reporting of weekly/monthly testing results The session is meant to show the currently implemented solution and collect feedback/requests from current and potential users. --------------------------------------------------- ★ Resources ★ Zerista: http://lcu14.zerista.com/event/member/137797 Google Event: https://plus.google.com/u/0/events/cvpv1fhs0pp56eafnph9dc3ovls Video: https://www.youtube.com/watch?v=D-1hkuroiYQ&list=UUIVqQKxCyQLJS6xvSmfndLA Etherpad: http://pad.linaro.org/p/lcu14-509 --------------------------------------------------- ★ Event Details ★ Linaro Connect USA - #LCU14 September 15-19th, 2014 Hyatt Regency San Francisco Airport --------------------------------------------------- http://www.linaro.org http://connect.linaro.org
Citation preview
LCU14 BURLINGAME
Milosz Wasilewski, LCU14
LCU14-509: Overview of QA reports
● Why are we doing it?● single place to check build and test status● manual and automated tests in one place● simple weekly/monthly reporting● bugs/known issues together with test results
QA Reports - overview
If the following conditions are met, the project is ‘green’:● build(s) successful● tests submitted to LAVA● all submitted tests produce results● all tests pass
Single build&test status
Single build&test status cont.
● Manual testing covers some areas that are hard to automate (UI, USB hotplug etc.)
● Till now:● Automated results were detached from automated● Reporting to MLs contained only manual test results● Changing test plan was painful
● From now:● Manual tests tied to build with clickable link● Manual and automated tests in one view● Test plan can be updated whenever there is a need (i.e. “won’t fix” bug appears)
Manual and automated test in one place
Manual and automated test in one place cont.
manual test results
automated test results
● So far reporting was manual● Reporting covered only manual test results● With qa-reports:
● reporting covers automated and manual tests● defects can be added to failed tests● reports can span across builds/devices● report is just an aggregated view of all results● additional comments can be added to the report (i.e. explaining the purpose of the
report or builds used)
Simple reporting
Simple reporting cont.
report description
automated test results
Simple reporting cont.
known issues extracted from all results
manual test results
● It’s possible to compare results between boards/builds
Result comparison
● Amount of data is huge● There is a lot of data duplication
● automated test results copied from LAVA● test definition versions copied from git repositories
● UI design isn’t perfect● Missing handy features:
● email notifications● more complex comparison● tracking of test case changes● measurement display● measurement graphs● ...
Issues
More about Linaro Connect: connect.linaro.org Linaro members: www.linaro.org/membersMore about Linaro: www.linaro.org/about/