TTM methodologyTTM methodologyThe right way to manage your Testing Automation The right way to manage your Testing Automation

projectsprojects

Testing Automation projects - life cycleCommon life cycle

1Purchase the tool 2Recruit developers 3 Understand ROI 4Fail to implement

1Analyze the systems Purchase the tools 2 Recruit developers

and managers3 Understand ROI 4 Follow TTM

methodology

TTM – life cycle

Testing Tools Management Methodology

Why Do We Fail?

We have the best people and the best tools but still many projects are failing

management expectations

What is our management expectation

•Testing Automation ROI

•Coverage

•Resource saving

•Shortening execution period

•Testing Automation usage

Age of the product

Releases planned for a year

Required regression cycle per year

Number of existing regression test cases

Number of configurations to be tested

Cost of the testing resources

Infrastructure cost

Test automation tool cost

Percentage of test cases that can be considered for automation

Time required for building reusable components, automation frameworks and batch scripts

Hourly cost per test automation resource

How to build Testing Automation ROI -input parameters

Testing Automation ROI – Points of benefit 3

Testing Automation ROI – Calculator

Microsoft Excel 97-2003 Worksheet

3

Testing Automation Readiness • Developers

• Stable environment

• Understand business process

• Tools readiness

• Testing Automation environment readiness

• Developers understand the project scope

• Set expectation

4

• Sanity SET

– Set of ~50 QTP scripts, verify that new build (version) ready for massive testing process

• Quick ROI Developments

– Packages can retrieve the investment immediately (e.g. API, data driven, data inflation)

• Regression by modules

So important to make sure NOT to develop all regression tests in one SET, “Before moving forward with the development stage, we need to analyze our system and divide it by modules, for each module to develop his own automation SET”

• Data Inflation

– Using our GUI interface can help with massive data changes

Testing Automation Stages4

How to succeed with your testing automation project using the TTM

Why Do We Fail con’t

• Understand the application and the development process

• Management support (not enough)

• High expectations for immediate ROI– Underestimating the time, cost and effort for the initial introduction and implementation of a tool (e.g. training, external expertise and changes to the testing process).

– Underestimating the effort required to maintain the test scripts and tool updates

4

• Define the project scope

• Follow up

• Testing Automation methodology

• Summary

Testing Tools Management Progress4

• High Level Design (Test Plan)

– The document contains the Set we are about to automate, the concept of automation, verification points, parameters, etc…

Define the Project Scope

4

• Design (Test cases QC)

– Test cases developed in QC cover all required area in the application

Define the Project Scope (Con’t)4

• Estimation module

– A tool can assist to understand how many working days we need for the project, include all required steps:

– Testing priority

– All Automation steps

– Risk factor

Define the Project Scope (Con’t)4

• Work plan

– Following the estimation module solution the exact work plan by days

Define the Project Scope (Con’t)4

Follow Up• Test Cases list

4

Follow Up (Con’t)- Work Plan

4

Testing Automation Methodology• QTP Standard and Guidelines

– Documents introduces basic guidelines and code methods for scripting via QTP. The document describes all acceptable practices to develop in QTP:

4

• Code inspection

– All of the inspectors to reach consensus on a work

– Product and approve it for use in the project

Testing Automation Methodology (Con’t)

4

Execution Daily reports

TA SET No. of

TC’s

No. of errors/failure Status Executer

Estimated time to

complete SET Log PATH Defect # Comments

BI 182 FAIL Dotan 2 WD Link to BI Report Till today covered 50 TC’s API Interaction 18 3 PASS Tali

3 WH Link to API Interaction report Yesterday Configuration issues defects close

API Application 19 1 PASS Irmi 1WH Link to Application API Testing automation errors

API Application 49 1 PASS Irmi 1WH Link to Application API_1 Ticketing 46 2 PASS Dotan 1 WH Link to Ticketing report Analyze by Dotan

EMT 18 2 PASS Dotan 1 WH Link to EMT report

A2A 10 PASS Dotan 30 Min Link to A2A report Voice 12 1 PASS Tali 3 H Link to Voice Report Testing automation errors

Sanity 14 1 PASS Lior 45 Min Link to Sanity report JT 17 8 FAIL Tali 1 WH Link to JT Report

Link to JT Report_1 Analyzed by Dotan

Reports 53 17 PASS Tali 1 WH Link to Reports log Data issues need to be fixed!!!

4

Execution Daily reports

Module# of

written TC's

# % # % # %BI 182 20 11 10 50 10 50API Interaction 18 18 100 18 100 0 0API Application 68 68 100 66 97 2 3Ticketing 46 46 100 40 87 6 13EMT 28 18 64 15 83 13 72A2A 10 10 100 10 100 0 0Voice 11 11 100 11 100 0 0Sanity 14 14 100 12 86 2 14JT 17 17 100 17 100 0 0Report 53 53 100 40 75 13 25

BIAPI InteractionAPI ApplicationTicketingEMTA2AVoiceSanity JTReport

FAILBIAPI InteractionAPI ApplicationTicketingEMTA2AVoiceSanity JTReport

Testing coverage # of pass TC's # of fail TC's

0

20

40

60

80

100

120

140

160

180

200

# o

f TC

's

Coverage

# of written TC's

Testing coverage

0

20

40

60

80

100

120

140

160

180

200

# o

f TC

's

FAIL TC's

# of written TC's

# of fail TC's

0

20

40

60

80

100

120

140

160

180

200#

of T

C's

PASS TC's

# of written TC's

# of pass TC's

Summary Development Stage• Final Report

• A document contains all project steps include:

• Test case list

• Testing automation coverage

• Functions

• Folder where script

• How to execute the SET

• Result example

4Follow TTM methodology

• Code Freeze

– A process to save all developed code under one place. (With a limited access)

Summary Development Stage (Con’t) m

Thank you All!!!

Any Questions?