Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
Eff ti fEffective use of Automated GUI Interactions on
System Verification
LeRoy Dunn & y
Michael Mullins
2 March 2016Copyright © 2014 Raytheon Company. All rights reserved.
Customer Success Is Our Mission is a registered trademark of Raytheon Company.
Overview
This presentation seeks to provide an insight to the use of automation of data entry and interaction via a Graphical User Interface (GUI) to provide the input required to conduct integration and verification efforts of a large system.
2 March 2016 2
Topics
Why to automate and expected benefits
Ho to incorporate a tomationHow to incorporate automation
Practical lessons learned from application
Summary
2 March 2016 3
Congressional and Customer Desire
Annually addressed in House Reports for D.O.D. budgetsg
Navy NAVSEA Instruction 4215.1 authored in 2010 to provide guidance for implementation for2010 to provide guidance for implementation for Automated Test and Re-Test (ATRT)
From the House of Representatives Report for National Defense Authorization Act for FY2016
The committee believes that the valuable lessons from such activities as ATRT should be more widely leveraged across the Navy and the rest of the Department of Defense
2 March 2016 4
rest of the Department of Defense.
Return on investment
Return on investment–Direct cost savings for test execution–Direct cost savings for test execution Automating an existing test procedure
– First Procedure – investment resulted in a break even point of 8.6 test executions after automation
– Subsequent Procedures – investment resulted in a break even point of 6.1 test executions after automation
Automating a test without developing a test procedure– Automation resulted in 30% increase in development cost and 80%
reduction in execution costreduction in execution cost
– Break even point realized in 4 test executions
–Numerous indirect savingsg Vary based on test environment, customer relations, etc.
2 March 2016 5
Technical Benefits (1 of 3)
Increased test run count provides broader coverage of test objectivescoverage of test objectives–Execute automated script hundreds of times in a
Monte Carlo approachMonte Carlo approach
–Allows for wider range of test inputs
Provides better opportunity to witness intermittent–Provides better opportunity to witness intermittent failures Tool may even help provide characteristics that support debugTool may even help provide characteristics that support debug
of the failure
2 March 2016 6
Technical Benefits (2 of 3)
Broader exercise of operator entered values for data entry pointsfor data entry points–Random selection of enumerations rather than specific
entries dictated by test procedure for manualentries dictated by test procedure for manual execution
– Increased breadth of invalid entries to move away fromIncreased breadth of invalid entries to move away from testing boundary conditions
Manual Test Points
0 99 255Special Characters za
Manual Test Points
2 March 2016 7
Automated Test Points
Technical Benefits (3 of 3)
Immediate determination of Pass/Fail evaluations rather than post-execution data analysisp y
Screen captures of Pass/Fail evaluationsCreates evidence of findings for what was typically a–Creates evidence of findings for what was typically a Demonstration test procedure
–Consistent capture points–Consistent capture points
Verifying exact measurement that are difficult to conduct in real timeconduct in real-time–Stringent timing requirements
C l i i d d–Color mappings to requirements or standards
2 March 2016 8
Why to automate and expected benefits
Ho to incorporate a tomationHow to incorporate automation
Practical lessons learned from application
Summary
2 March 2016 9
Evaluating for Implementation
Review your Contract and/or Statement of Work– Incentives/Contractor Performance Assessment ReportIncentives/Contractor Performance Assessment Report
–May require updates to Master Test Plan and/or Test Procedures which require customer approvalq pp
Dozens of options based on desired approachp pp–More than 30 tools have developed in this area and
some are already obsolete
–Wide range of characteristics drive tool selection
–Understand both your corporate tool providers and the y p ptools available to your customer
2 March 2016 10
Gaining Customer Acceptance
Determine if tool selected requires customer certification for use in program qualificationp g q–Plan for collaboration as automated testing is likely to
be new to the customer as well as the contractor
–Understand limitations of the selected tool
Issue concerning Method of Verification (MOV)–Many tests of an HMI or heavily utilizing an HMI areMany tests of an HMI or heavily utilizing an HMI are
Demonstration, but automated testing replaces the human witness with a computer witness (i.e., a tool)
–May have ramifications for an existing specification2 March 2016 11
Incorporation into Formal Test Events
Customer acceptance/reluctance will vary greatly
One approach that has gained concurrenceOne approach that has gained concurrence –For converted tests, verify identical execution of an
automated test script and paper procedure during theautomated test script and paper procedure during the same witnessed test event Identical outcomes allows for future automated execution only y
–For automated only tests, allow the customer to observe the execution of the automated test procedure Demonstrate against specification or approved use case
–Address use of automation during Test Readiness Review and gain the board’s approval to proceed
2 March 2016 12
Why to automate and expected benefits
Ho to incorporate a tomationHow to incorporate automation
Practical lessons learned from application
Summary
2 March 2016 13
Execution Challenges (1 of 2)
Test automation infrastructure–Hardware upgradesHardware upgrades
–Lab support
–Scripting software and supportScripting software and support
–Documentation
Personnel training–Personnel training
–Security Plans
2 March 2016 14
Execution Challenges (2 of 2)Test maintenance cost
–Evolving System Under Test (SUT)g y ( )–Evolving test tool GUIs–Evolving lab environment
Starting the implementation of test automation either too early or too late in the development process –Too early: “spinning wheels” during initial development
i d d ROIperiods reduces ROI–Too late: Not enough test execution runs to recoup
investment losses (reduced ROI)investment losses (reduced ROI)
2 March 2016 15
Scripting Tradeoffs
Go path testing vs failure and error handling–Challenge of handling test step failure and errorChallenge of handling test step failure and error
exception sprawl
–Determine the appropriate level of test robustness and pp preliability for each automation case
Test step3 Test step4 Test step5 Test step6pass pass pass pass
Log step3 failure
Execute errorException3_1
Terminatetest and logtest results
Log step4
Execute errorException4_1
Execute errorException5_2
Execute errorException4_2
Execute error
Execute errorException5_1
err err
err
err
err
errfail
fail
cont contcont
RandomizationHuman in the loop introduces inherent variability vs
Log step4 failure
Log step5 critical failure
Execute errorException5_3fail
term
term
–Human in the loop introduces inherent variability vs scripting introduces repeatability and repetitiveness
2 March 2016 16
Graphics Comparison and OCR Tuning
For graphics comparison or Optical Character Recognition
Click (ImageName: “CloseWindow”, SearchType: Tolerant, HotSpot: (45 15) WaitFor: 10 Tolerance:p g
(OCR), start with default settings, then depending on
(45, 15), WaitFor: 10, Tolerance: 10, Discrepancy: 5, Pulsing: false, SearchRectangle: (180, 250, 240, 300))g , p g
test performance tune searches as needed for the specific GUI and test goals ReadText ((300, 200, 350, 220),
DPI: 80, Contrast: true, ContrastColor: black
Tuning can be done for individual searches or
ContrastColor: black, ContrastTolerance: 50, ValidCharacters: “A0123456789”, IgnoreSpaces: true)
globally (all searches)2 March 2016 17*eggPlant code snippets. For eggPlant documentation see http://docs.testplant.com/
Execution Limitations
Human-In-The-Loop (HITL) limitations–Non-GUI system controlsNon GUI system controls
Safety limitationsRadar ranges–Radar ranges
–High power equipment
S i li i i!
Security limitations–Screen locks, Password control, VNC servers
–Prevention of disclosure of classified capabilities
Consult SMEs as needed 0100010010010101001010110101011010101010001010001010 10101010100
–Test Engineers, Lab Manager, Security, etc.2 March 2016 18
010001010 10101010100010101000 00100100
Why to automate and expected benefits
Ho to incorporate a tomationHow to incorporate automation
Practical lessons learned from application
Summary
2 March 2016 19
Summary
Infrastructure– HW, SW, support, training
Customer Satisfaction
Value , , pp , gMaintenance
– SUT, GUIs, lab environmentD l t
– Direct long-term cost savings
– Indirect efficiencies Development
– Scripting, tuning Limitations
Improved Technical Approach– Thoroughness Limitations
– HITL, Safety, Cybersecurity– Robustness
– Expanded test capabilities
ROI– Frequency
2 March 2016 20
Consider ALL benefits, costs and drawbacks in the ROI analysis
Go Forward Plan
Start early–Use ROI analysis to drive plans
ROI analysis
Use ROI analysis to drive plans
–Get customer buy-in on approach
Start conservative
Buy In
Implementation
Execution
Start conservative–Better off with realistic ROI estimate
S ll
p
Start small–Focus on a small set (1-3) of near term test cases
Start easy–Automate the “low hanging fruit”
Reevaluate2 March 2016 21
Biographies Mike Mullins is a Senior Principal Systems Engineer at Raytheon Company,
Integrated Defense Systems. He has 20+ years of experience in real time communications, networking systems, and track fusion, including algorithmcommunications, networking systems, and track fusion, including algorithm development and IV&V. In his current assignment, Mike is the Integrated Product Team Lead for a Combat System Element. Mike has a Bachelors Degree in Computer Science from Florida State University.eg ee Co pute Sc e ce o o da State U e s ty
LeRoy Dunn Jr. is a Principal Systems Engineer at Raytheon Company, Integrated Defense System. He has 10+ years of experience in mechanical design of test equipment and radar electronics as well as 10+ years ofdesign of test equipment and radar electronics, as well as 10+ years of experience in systems requirements development, architecture, IV&V and data analysis. In his current assignment, LeRoy is a data analyst for a Combat System Element. LeRoy has a Bachelor’s Degree in Mechanical EngineeringSystem Element. LeRoy has a Bachelor s Degree in Mechanical Engineering from University of Massachusetts at Amherst, a Master’s Degree in Mechanical Engineering from Northeastern University, and a Master’s Degree in Computer Science from Worcester Polytechnic Institute.Degree in Computer Science from Worcester Polytechnic Institute.
2 March 2016 22