Upload
vijayanreddy
View
231
Download
0
Embed Size (px)
Citation preview
8/14/2019 Planning and Building Scalable Test Infrastructure
1/22
PLANNING &PLANNING &
BUILDING SCALABLEBUILDING SCALABLE
TESTTESTINFRASTRUCTUREINFRASTRUCTURE
Vijayan Reddy
Engineering Manager
Adobe Systems
8/14/2019 Planning and Building Scalable Test Infrastructure
2/22
Why Test Infrastructure
Where are we missing? How can we save?
How much of your test project ismanual efforts
How much of your test project istest execution
Projected Time slices
8/14/2019 Planning and Building Scalable Test Infrastructure
3/22
In Scope
This is no Rocket Science
Approach from a typical testlifecycle End to End tasksfor shipping a project
A blue-print for scalable testinfrastructure
Giving tips on cross-linking ofthe tools
8/14/2019 Planning and Building Scalable Test Infrastructure
4/22
Test Infrastructure
An interface/glue that connects every single task and providesplug-ability for new tasks
All Material Used Tools,Methods, Scripts
Platform that hosts &integrates tools and scripts
Commercial tools dont offercomplete support of alltasks
8/14/2019 Planning and Building Scalable Test Infrastructure
5/22
How To Build
Available off-the shelf? Probably not
Buy/Build/Customize is the mantragreat commercial software
great open source frameworksgreat engineers in your team
8/14/2019 Planning and Building Scalable Test Infrastructure
6/22
Top Opportunities
Test Execution Biggestopportunity
Build SystemsBVTPIT
Defect AnalysisLinked Test Cases creation
& Maintenance
Test EnvironmentsMaintenanceSetup / clean up after
executions
Test Reporting / Archiving /Data Mining
8/14/2019 Planning and Building Scalable Test Infrastructure
7/22
Scale with Meta Controller
8/14/2019 Planning and Building Scalable Test Infrastructure
8/22
Meta Controller -Requirements
http://staf.sourceforge.net
Able to work p2p, or client server
Able to work across platforms
Able to invoke, monitor tasks, Remotely Trigger Execution, eg.
Able to transfer files Builds, Installers Test Results, Logs, Thread dumps
Able to parallelize across nodes
http://staf.sourceforge.net/http://staf.sourceforge.net/http://staf.sourceforge.net/8/14/2019 Planning and Building Scalable Test Infrastructure
9/22
Optimization @ Test Setup
Time on Test Setup, Test Clean upClean Systems
Imaging / ghosting can helpAcross different platforms?
Build the software stack - Eg: Application Server, Java/.NET run time
Database
Deployment of your applications
Deployment of test suites & Setup
Multiply savings by the number of builds, andMachines
8/14/2019 Planning and Building Scalable Test Infrastructure
10/22
Meta Controller
Control the test automation framework(s)
Glue the controller and tool(s), Script(s)APIs exposed
Command line options / Batch filesParameterize the launch:
Platform / machine agnostic scripting
specific details of each test machine abstracted
in configuration files
Enable Metrics Collection Code Coverage,Profiling
8/14/2019 Planning and Building Scalable Test Infrastructure
11/22
Scale with Helpful Interfacing
Test cases managed under a system (Manual &Automated tests) (Eg.http://testlink.sourceforge.net)
Why do we need a system, not DOC, PDF, HTMLTo assure test coverage / traceability
Cross linking with Feature Tracking
Cross linking with Bug tracker
Cross linking with Source Control
Help track estimates as well, against test cases/cycles
Interface SCM and Bug Tracker
8/14/2019 Planning and Building Scalable Test Infrastructure
12/22
Scale in Defect Analysis:Facilitators
Traceability LinkingFeature -> Test Case ->
Changed Files,versions -> Bug ->dependent bugs
8/14/2019 Planning and Building Scalable Test Infrastructure
13/22
SCM Bug tracker Linking
One spot update of bugs & SCMWhen fix is checked in
Use a templateFill description of fixFill areas affectedFill Recommended tests
Auto - Update Bug trackerAreas AffectedFiles Affected and versions of these files
Useful in Regression testing & Injections check Search up other bugs which had fix in same files and
retest.
8/14/2019 Planning and Building Scalable Test Infrastructure
14/22
Scale with Early Warnings
Static Code Analyzers
Catch bugs even before thecompilation is done, letalone built and tested
Standard Code violations,Uncaught exceptions,Un-freed memory etc.
Code Security audits forscript injections
8/14/2019 Planning and Building Scalable Test Infrastructure
15/22
Automated Builds
Check-in Triggered Builds,Scheduled Builds
Build System to accessbuilds, logs, changedfiles, changed versions
Promotion to QA onSuccess, Auto Deploy
Tagging / Merging onSuccess on SCM
8/14/2019 Planning and Building Scalable Test Infrastructure
16/22
Scale with BVT
Catch the Bugs early with BVT
Automate triggering of BVT onbuilds
Use your Automated Sanity /
Acceptance TestsLink with Build Setup to Bless
builds for QA promotion Saves QA cycles
Automate failure mails Send
with exact test log, pick thecode changes from the build
Auto-log a bug
8/14/2019 Planning and Building Scalable Test Infrastructure
17/22
Pre Integration Test Setup
PIT saves pitfalls Huge development teams, central builds
Check-in to a PIT branch (that is Sync with Trunk)
Kick offAuto Build
BVT
If Success, auto-Merge change into Trunk
If not, follow BVT failure handling and auto- revertfrom PIT Branch
8/14/2019 Planning and Building Scalable Test Infrastructure
18/22
Infrastructure for Reports
Common results repository Different test cases / suites /
tools output differently
Build interfaces to import,export
Define common resultsformat
Preferable formatDatabase
XML
8/14/2019 Planning and Building Scalable Test Infrastructure
19/22
How to Export
Commercial tools export in XMLs
If that helps, standardize other tools / scripts forthat
Use XSLT to transform to common formatDefine Performance results schema
S l th h i t lli t
8/14/2019 Planning and Building Scalable Test Infrastructure
20/22
Scale through intelligentreporting Easier access for data mining
Easier reporting, trends along builds, alongfeatures, along platforms, locales
Drill down ability on any individual test case, andsee history along builds to isolate injections
Easier collation of all test results
Charting for Non functional parameters
improvements Cross Link to Feature / Test Case tracker
8/14/2019 Planning and Building Scalable Test Infrastructure
21/22
Take Away
Identify these bridges thatwork for you No onesize fits All theory
Create an integrated
platformBuildBuyCustomize
Available athttp://[email protected]
http://qualinfra.blogspot.com/mailto:[email protected]:[email protected]://qualinfra.blogspot.com/8/14/2019 Planning and Building Scalable Test Infrastructure
22/22
Questions