Test Automation: A Roadmap For Sucesss

2008-01-20

© SQS Group Limited | Presentation

title | July 2008 | page ‹Nr.› 1

SQS Group Limited

Scott Armstrong

Belfast, February 2010

Test Automation

A Roadmap For Success

Agenda

� Steps for Success

� Automation Tool Selection

� Test Selection and Prioritisation

� Implementing the Strategy

� SQS Case Studies

� Questions

2© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010

2008-01-20



Steps for Success

Approaching Test AutomationPlanning the Framework

© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010

Approaching Test AutomationSteps for Success

AUTOMATION PROJECT

DESIGN

Build maintainable tests

Target and prioritise test areas

TOOL

Select with care

Pilot tool

Manage Implementation

TEST MANAGEMENT

Defect Management

Config Management

Information and Reporting

Requirements

TEAM

Code reuse and collaboration

Partnership between test and dev

PLAN

Treat as a Development Project

Manage Expectations

Commitment at all levels


2008-01-20



Planning the FrameworkSteps for Success

� What are the Management Information Requirements

� Contents

� Format

� What are the Execution Requirements?

� On-demand

� Automated Nightly Builds

� What technologies are we testing?

� Embedded

� Operating System

� What resources are available?

� Hardware

� Technical Expertise

� Are there cultural or organisational considerations?

� Outsourced IT Management – no admin access

� Current partnership with tool vendor


Automation Tool Selection

Selection ProcessIdentify RequirementsBuy, Open Source or Build


2008-01-20



Selection ProcessAutomation Tool Selection

Define Problem

Consider automation

as a solution

Make business

case

Define required features

Shortlist

Evaluate

Demo

Trial

Decide

Constraints


Identify RequirementsAutomation Tool Selection

� Start by evaluating your requirements, not by investigating the tool market.

� Look at the current problems – what has prompted the search for an automated solution:

� Manual testing problems (e.g. too time consuming, error prone);

� No time for regression testing when small changes are made to the software;

� Set-up of test data or test cases is error prone;

� Inadequate test documentation;

� Don’t know how much of the software has been tested;

� Testing is ineffective.

� Not all of these problems are best solved by automation.

� Rank your problems by importance or current cost to the organisation.


2008-01-20



Buy, Open Source or BuildAutomation Tool Selection

• Quickly provides a high level of features

• Well supported in terms of documentation, help and training

• Often an attractive project to be involved with

• Added cost of license fees

Commercial Tool

• No license fee cost

• Limited to support from community forums

• Defects in tool may not be addressed

• Tool may lack a clear roadmap; may not be future-proof

Open Source Tool

• No license fee cost

• Development and maintenance can prove expensive

• Designed to meet project requirements

• Lacks full range of features

Build Bespoke Tool


Test Selection and Prioritisation

Test SelectionTest Prioritisation


2008-01-20



Test SelectionTest Selection and Prioritisation

� Some tests cannot be automated.

� Some tests are not worth automating.

� May take more time to automate tests than to run them manually

� You may not have to automate everything to get significant benefits

� 10% of the tests may be run 90% of the time.

� Some types of tests are amenable to automation, some not:

� Functional tests are often the most straightforward

� Performance tests are notoriously difficult to carry out manually. These are a good candidate for automation.

� Non-functional tests can cover maintainability, portablility, testability, usability, etc, and can be inappropriate for automation.


Test PrioritisationTest Selection and Prioritisation

� Do not automate too much too soon.

� Best practices take time to develop.

� Go for ‘quick wins’ – the areas where automation will have the largest impact most quickly.

� Prioritisation factors:

� A set of breadth tests (sample each system area);

� Tests for the most important functions;

� Tests that are easiest to automate;

� Tests that will give the quickest payback;

� Tests that are run the most often.


2008-01-20



Implementing the Strategy

Automation Team FitMaintenance Model


Automation Team FitImplementing the Strategy

Automation Team

Specialist Group/Tool

Support

Subject Matter Expert

Management Buy-in

Development Team

Test Team


2008-01-20



Maintenance ModelsImplementing the Strategy

• Client representative with technical background is trained during automation project

• Maintenance of current scripts can be supported independently for future product releases

• Future new tests can be automated in-house and added to the regression pack

Knowledge Transfer

• Client does not have to be involved in the details of automation development, execution or maintenance

• The necessary technical skills do not need to exist in-house

• Assured best practice automation and quality

Managed Service


SQS Case Studies

Lloyds TSB Wholesale BankingLocal Utility Company


2008-01-20



Lloyds TSB Wholesale BankingSQS Case Studies

Manual Testing Challenges

• 600+ Tests and growing

• 200 man days per release

• Concerns over coverage

• Regression pack growth impacted by limited resources

Our Engagement

• Innovative test automation solution implementation

• Highly maintainable and extensible solution

• Successful solution handover to client test team

Customer Success Return on Investment

• Increased test execution efficiency by 85%

• Reduction in deployed defects >50%

• Enabled daily smoke test execution

• Overall ROI of £1.9m over 6 years

Lloyds TSB

“Without

automation, we

would not have

gone live”

“The automated

regression suite

made my job

more interesting”


Local Utility CompanySQS Case Studies

Manual Testing Challenges

• 100 test scripts with multiple complex data-driven iterations

• 120 man days regression testing per year

• Low test coverage – limited to ‘spot checks’

• Testing of new functionality impacted by large regression packs

Our Engagement

• Collaborative design of framework specific to client requirements

• Full knowledge transfer to existing test team for ongoing maintenance and development

• Modular framework allows new tests to be built with minimal effort

Customer Success

• SQS are now engaged on further automation projects with the parent company

• The existing test team continue to expand and maintain the framework to increase their ROI

Return on Investment

• Saving of 96 man days of execution per year

• Data-driven scripts increased test coverage by a factor of approximately 200

• Automation solution was ported to a further 2 business units, multiplying the ROI

Local Utility


2008-01-20



Thank you for your attentionThank you for your attention

SQS Group Limited

The Concourse | Northern Ireland Science Park

Queen‘s Road | Queen‘s Island | Belfast BT3 9DT

Phone: +44 (0) 2890 022 660

E-Mail: [email protected]

Internet: www.sqs-uk.com


Documents

Test Automation: A Roadmap For Sucesss