Test ManagementIVS-TRAINING

“© 2005 Infosys Technologies Limited.  All rights reserved.  The information provided in this presentation is intended for the sole use of the recipient and is for educational purposes only. No part of this presentation may be reproduced or transmitted in any form or by any means, including photocopying and recording, without written permission.  Permission must also be obtained before any part of this presentation is stored in a retrieval system in any nature.  No responsibility can be accepted by Infosys Technologies Limited, the Editorial Board or contributors for action taken as a result of information contained in this presentation.  The views expressed in this presentation by the presenter are not necessarily those of the Editorial Board or Infosys Technologies Limited.”

-2-

Session Objectives Project management overview Testing lifecycle

Initiation Test strategy / plan Estimation Scheduling Support activities

Execution Test execution Status reporting Defect management

Closure Test process analysis

-3-

Session Objectives (Contd…)

General management topics Risk management Test metrics Stakeholder management Test management challenges Test management best practices

Summary

-4-

Project management overview

Project Management

QualityManagement

TimeManagement

TeamManagement

CommunicationManagement

ScopeManagement

RiskManagement

CostManagement

IntegrationManagement

Vendor/SubcontractorManagement

-5-

Testing Lifecycle

InitiationExecution

Closure

Test Process

It is a combination of different lifecycle stages of a testing project. Each lifecycle stage is characterised by a definite Entry criteria, activities, necessary checks, deliverables, exit criteria Control and management of critical information can be handled by applying different tools and techniques

At high level there are 3 Life cycle stages

In some projects initiation and execution stages can be iterative

Initiation

-7-

Initiation

This stage includes the Requirement Analysis, Test Planning, Test case development phases of the Software Testing Lifecycle

Activities Understand contract, proposal, application and objective of all stakeholders Prepare Requirement Traceability Matrix (RTM) in the initial stages to keep track of every requirement Deliverables (Strategy/plan, test scenarios, test cases, project plan), commitments, estimates approved by client Define critical success factors for the project to plan the milestones Test data & Test environment set up Standardize templates, checklists, process to have consistency in usage of artifacts Structured formal communication Automation feasibility analysis and tool selection in case of test automation

-8-

Initiation (Contd…)

Deliverables RTM Test Strategy / plan Test scenarios and test cases Configuration Management Plan Training Plan, Quality Plan Estimation Automation feasibility analysis report

Initiation

Note: Open the file to detail the initiation process

Requirement traceability matrix

-10-

Requirement traceability matrixRequirement Traceability Matrix (RTM)

Maps business requirements to the corresponding test scenarios and test cases

As a whole helps estimate effort as it provides a sense of the testing effort required in terms of test scenarios / cases to be executed

Plays a critical role during test execution as the amount of testing required and the scenarios to be tested can be determined based on the available time and the relative criticality and priority of the remaining test cases

Test strategy / plan

-12-

Test strategy / planTest strategy / plan is the ‘Approach’ applied to handle a testing project.

It helps the Testing team get answers to following questions

What is the objective, scope, timeline and cost of testing?

What are the various types of testing to be performed?

What are the tools to be used for functional and performance test automation, test management, defect management, review, configuration management, estimation?

How do Testing and Production environments differ?

When should the other stake holders (User, data set up team, Environment set up team, development team for fixing defects etc.) be involved?

-13-

Test strategy / plan (Contd..) What is the technical architecture of the application/product?

What testing methodology should be adopted depending on the methodology used for development of the application/product?

What should be the skills and training needs of the test team members?

What are the roles and responsibilities of test team members?

Who are all the stakeholders and what is the role and importance of each stakeholder?

Is there a need to submit a proof of concept?

What is the estimation methodology used?

-14-

Test strategy / plan - Activities

Scope and Planning

Identify the scope & types of testing, dependencies and extent of testing and number of iterations

Identify the timelines, test team members, risks

Prepare training plan, quality plan and configuration management plan

Determine Test Suspension / Resumption criteria To plan the suspension and resumption of testing during

execution

-15-

Test strategy / plan – Activities (Contd…)

Test environment specification

Hardware/software, operating system, tools, connectivity, interfaces, sign off of the environment

Understanding the test environment and test data with respect to production environment

Release Management

Get acceptance criteria from the client

Prepare release checklist and use it before every release

Get the Sign-off from the designated stakeholder for the release

-16-

Test approach/methodology

Identify the lifecycle (LC) stages for the project and for each LC stage identify the

Entry & exit criteria Activities & Deliverables

Schedules and timelines

List down the timelines of each type and phase of testing List down the specific milestones throughout the duration of the

project

Identifying and documenting Assumptions & Constraints

Test strategy / plan – Activities (Contd…)

-17-

Pradeep Yadlapati (GTM) says “There is a lot a manager can do by understanding the value that the project can add to the customer”

Expert speaks

-18-

Case study

Test Strategy

For the scenario given in the file,1. As a Test Manager, how will you approach the Test strategy / plan?2. Prepare staffing plan3. Decide on mile-stones and deliverables

-19-

Case study – Answer

Key points Test strategy should focus on the 1. Types of testing to be done 2. Deliverables 3. Release road map 4. Test team composition 5. Risks – knowledge of POS 6. Test environment 7. Training needs 8. Automation needs 9. Assumptions 10. Schedule and timelines

-20-

Change Management

Plan for managing requirement changes (Requirements traceability)

Assess impact of changes on testing

Code fixes

Log New defects

Next Test Cycle

CCB Impact

AnalysisRTM

based Test Case

Impact Analysis

Prioritize CRs

Build next Release Version

Add, Modify,

Delete test cases &

Test data Test Team

Dev Team

CCB = Change Control Board

CR = Change Request

Test strategy / plan – Activities (Contd…)

-21-

Case study

Prepare an approach to test the change in scope specified in the example

May 12th: Arjun is a Test Manager working for a Insurance client. He has a team of 3 TAs and 15 TEs. At present they are into the execution mode. The testing is expected to be completed by 30th June.

The system has 6 modules with 2 modules (M1, M2) released for testing from the client development team and 4 interfaces (I1, I2, I3, I4) from other vendors. On 3rd June client decides to replace I2 with a component being developed in house.

The development is expected to be completed on 11th June. I2 interacts with I3, M1 and M2. Client wants the testing to be completed by 8th July.

-22-

Case study - Answer

Key points

1. Impact analysis of scope change and obtaining client sign off 2. Prioritization 3. Using RTM

For details of the resolved case study refer to the attached document

Answer - Scope

Support activities

-24-

Support Activities – Configuration Management

Configuration Management

Identification of ConfigurationManagementtool

Prepare CM plan

Track status of configurable items

Regular CM audits

-25-

Support Activities – Quality Assurance

Quality Assurance

Reviews of testartifacts and periodic audits to verify process compliance

Customizing theprocess to the project’s need and implementingthe same

Defect preventionactivities

Test coverage using theTraceability matrix

-26-

Support Activities – Infrastructure set up

Infrastructure Set up

Test data setup as per the need

Data security for sensitive data

Software installation or deployment

Hardware and software (tools, OS, DB etc) Setup as specified for the test environment

Estimation

-28-

Test Estimation

Effective software test estimation helps track and control cost/effort-overrun

Estimation is a continuous process that should be completed at the beginning of a project, but reviewed throughout the lifecycle

Test Estimation should be based on Technology, Architecture, Domain, and complexity

Thumb rule - The bigger and more complex the code is, the more likely the possibility that errors exist in it!

-29-

Test Estimation (Contd…)

Consists estimation of the following for carrying out the project The project size, cost and effort The number of H/W, S/W, human resources

The Industry wide used effort estimation are

Phase Requirements analysis + design

Development Testing (Unit to System)

Effort 25% to 40% 20% to 30% 25% to 40%

-30-

Business requirements for each function are identified and classified as Simple (S), Medium (M), or Complex (C)

Define the SMC criteria. Example Test Case with less than 5 steps – Simple Test case with 5 -10 steps – Medium Test Case with more than 10 steps – Complex

The average implementation effort on S/M/C can be got from the baseline if it exists

SMC method

-31-

SMC method

If a project-specific baseline does not exist, the project type, technology are used to get the test effort from similar project

If the technology/domain is unknown, then the average testing effort for S/M/C is derived from past experiences

General activities like Test Environment & Data Setup form input parameters to calculate the total effort estimates

Limitations of this method

Level of complexity may differ from project to project, based on combination of domain, technology used. Hence SMC values cannot be generalised

Does not consider the expertise level of test team members Is person dependent

-32-

Test Unit Model - Introduction

Aimed at helping in estimating the effort for the LC stages of Functional Manual Testing,

Built on an abstract unit called - Test Unit (TU)

TU is used as the unit of measurement of the Size of the project

There is no direct relationship between TU and the Test cases

The Model considers Test case specific inputs and adjustment factors based on Team specific attributes

Instead of leaving the decision of identifying the complexity of a test case with PL/PMs, the Model equips them in quantitatively justifying the complexity

-33-

Test Unit Model System specific Parameters

# of Steps # of input Parameter # of validation checks # of Links # of Numeric validations # of UI checks # of Database validations

Each of these parameters contribute to the complexity of the Test case and the same is measured in terms of TU

Scheduling

-35-

Scheduling Refers to the timelines for activities during the life cycle Determined from the effort estimate & release roadmap Involves identifying major milestones with detailed tasks Is an iterative process

Factors affecting schedule Scope Testing team Third party tools Interfaces Dependencies

Execution

-37-

Test executionActivities

Verification of Entry & Exit criteria to understand whether the necessary criteria are met to start and end the project respectively

Verification of Suspension & Resumption criteria to understand when should testing stop and resume

Execution of test cases, test scripts and updates to test artifacts Tracking testing progress through milestone and metrics analysis

-38-

Test execution (Contd…)Activities

Regular reviews & audits to keep the project in good health

Issue resolution

Regular status tracking & reporting and communication with all stake holders to keep them updated about the project

Defect management

-39-

Test execution (Contd…)

Deliverables

Updated test plan

Updated test schedule

Test cases

Review report

Test log

Execution

Defect report

Status report

Milestone analysis report

Metrics report

Note: Open the file to detail the execution process

Status reporting

-41-

Status reportingStatus report is used to update the stakeholders regarding the

Current status of the project Forthcoming releases and milestones Challenges faced Challenges ahead Best Practices followed Customer appreciation Customer complaints

Depending on the

Role of the stakeholder in the project, the status data is distributed

Complexity and need, the frequency (Weekly, Monthly) and type of status is reported

-42-

Status reporting (Contd…)Status report contains details regarding Planned Vs Actual data w.r.t. deliverables, schedule, cost, quality

What were the risks encountered?

How much testing is complete? What are the results of the completed tests? Who were involved in testing this? When did the testing complete? Number of defects raised in each iteration per build? What is the requirements coverage percentage?

What are the dependencies?

Does the current product meet requirements that were originally set?

Is the product ready for release?

Defect management

-44-

Defect management

Classification based on priority, the urgency at which the defect fix is needed and on severity, the impact the defect is causing on the application

Identifying defect life cycle. This depends on tool, but may need customization. Movement of defect from one stage to other and who are involved in it.

Metrics collection like defect density, defect detection rate, defect fix rate etc

Check to see if there exists mapping between defects and failed test cases

Closure

-46-

Test cycle closure process The test cycle can be closed when

Test cases for the cycle are executed and all outstanding defects are resolved

All deliverables are accepted by customer Quality goal of the product is achieved

Closure activities Conduct closure meeting – To discuss closure report data Perform closure analysis - Prepare Closure report and obtain a

sign off Collect feedback from all stakeholders – To understand the

satisfaction level Archive project artifacts – For future reference and use Handover the client assets (access cards, H/W, S/W resources)

and release test team members Submit knowledge assets – For reference in future

-47-

Closure Report Contains detailed information regarding

Process Details Risks Size and Schedule Effort Defects Test Metrics Best Practices Customer Appreciations / Complaints

Closure

Note: Open the file to detail the Closure process

Test process analysis

-49-

Test Process Analysis

Analyse To

Gaps between goals set and goals met

Implement techniques to narrow the gap

Lessons learnt during the test lifecycle

Derive best practices & identify areas of

improvement

Compliance with internal & customer specific processes

Changes in requirementsCheck the impact on cost, schedule & effort

Improve quality of the product & Capture metrics

General management topics

Risk management

-52-

What is a Risk?

A possible event or situation, potentially having either positive or negative impact on the project.

Example – Your company is the only Testing service provider for your client. For a new service, this client requests for a proposal from you and two of your competitors (Risk to your relationship with the client). You do a thorough due diligence and arrive at a solution package that impresses the customer and you get a go ahead from the customer to carry out this project.

Risk Opportunity Benefit

-53-

What is a Risk?

Example – You get a new service request from your client. You plan to include fresh test engineers in to the project and prepare their training plan. Few weeks into the project, due to lack of expertise among the testers you start seeing problems related to test case development, understanding of test process etc. which could potentially lead to Crisis

Risk Problem Crisis

-54-

Risk management

Assess risks Classify them Identify the impact on testing Review the assessment in the light

of changing circumstances and risk management actions

Risk AssessmentRisk

IdentificationRisk

Prioritization

Risk ControlRisk

MonitoringResponse Planning

Identify risks Proactively at the

beginning of the project Throughout the testing

life cycle

Risk management is an iterative process of identifying, assessing and responding to risk

-55-

Contingency planning The idea behind contingency planning is to reduce the impact of a risk

by creating alternatives

Assumes that the worst will occur and tries to figure out methods so the project can continue

Allows the testing team to address the unforeseen challenges during life cycle

Will focus on areas such as: Sanity testing prior to full blown testing Risk based testing Bringing in an external domain expert to focus testing on highly

critical areas of the application Increased testing effort, at the cost of redundancy in order to increase test coverage

-56-

Sample risks in testing

Risk to the deliverables - Defects found in production

High test management overhead due to dependency on other phases, projects, interfaces, development team

Connectivity failure to test environment

Issue resolution from client not happening on time

Requirements change affecting testing

Increased turn around time in defect fix

• Lack of data to test some specific cases

-57-

Sample risks in testing (Contd…)

Incomplete Test Planning

Incomplete / Absence of traceability

Improper Testing, Test data usage

Insufficient knowledge of the application, tools and processes

Testing time shrinks but deadline does not change

-58-

Case study

Identify the risks in the scenario given and prepare contingency plan

The project is about a system of Mortgages. The development PM Saroj and testing TA Vijay are at onsite doing a system study. They are expected to complete estimation for this. The system is to be implemented using Microsoft Visual C++ and a Sybase RDBMS. The system would interface with an existing legacy system to maintain audit trail. The company for which Saroj and Vijay work is a reputed solution provider. Saroj and Vijay decide to prepare their respective estimation and then review it. End of same day they decide to meet. When Vijay presents his estimation sheet to Saroj, she does not agree to that. She agrees on the accuracy but her idea is, to sustain competition, the estimation given should be less than what the competitor could have given. Saroj has previously managed projects with such tough timelines with members in her team slogging. She quotes comparatively very less time for testing and Vijay is completely against this. Since she is the POC for client, she presents her case to Peter (client contact) and Peter is very happy to note the estimation. Peter thinks 'so much work in such a short duration? The company is anyway reputed they will deliver the goods'. Saroj's team gets the project and Vijay has to handle testing.

-59-

Case study - AnswerKey points 1. Common risks like – connectivity failure, attrition, issue resolution not happening on time etc 2. Specific risks – Dominating development manager, testing team morale, Stringent timelines

For details of the resolved case study refer to the attached document

Answer - Risks

Test metrics

-61-

Metrics collection and usageThe objectives of metric collection are to Determine the quality and productivity at which each project is

operating

Determine if project is meeting the Service Level Agreements within the defined bounds and look for areas of improvement

Check the health of the project

Analyse strengths and weaknesses

Set goals for future projects based on past data

Provide inputs for estimation of future projects

-62-

Metrics collection and usage (Contd…)

Business Goal Measurement Objective Metrics

Improvement in productivity

• Ensure that the project operates at the desired productivity levels • Measure process performance

Size, effort, review effectiveness, rework effort percentage

Improvement in delivered quality

• Ensure that the software produced is at a desired level of quality• Meeting SLAs• Ensure User Acceptance Level satisfaction

Delivered defects, defects detected at various stages, Turn around time, Service levels (as applicable)

Adherence to schedule

• On time delivery• Meeting SLAs

Elapsed days, schedule adherence percentage

Return on Investment (ROI)

• Validate if the total cost of ownership has been lowered

Cost of Quality, cost of defects detected / undetected, cost savings over multiple releases, cost benefits of off shoring

Following table lists the Metrics collected for a Business goal to achieve a stated Objective

Stakeholder management

-64-

Stake holder management

I am 6 months old in this project and have sufficient experience to take up additional responsibilities. That would help me move to a higher role as well

Tester

Vendor company is CMMi5 certifiedI needn’t worry about quality and schedule of deliverables

Client

Success of this project will line up more new projects with good operating margin

Management

The testing team has been loading us with lot of rework by raising too many defects. Let me deliver my module late for the next release

Developer

Challenge – Meet expectations of all Stakeholders

Challenged TM

Stakeholder management - Customer

-66-

Customer managementUnderstand the customer’s objective of executing the project

Obtain sign off for the deliverables

Communicate regularly to Update the status Resolve issues Understand any scope changes

Mode of communication could be face to face meeting, conference call, email, video conference

Capture periodic feedback to know customer satisfaction level and work on the areas needing improvement

-67-

Kapil Saxena (GTM) says “Client management plays a very important part in the business”

Expert speaks

Stakeholder management – Test team

-69-

Test team Test team Identification

Willingness to learn new technologies, domain and be part of the overall testing team

Constructive attitude towards quality

Destructive creativity

Probing skills

Flexibility to adapt to frequent changes

A skeptical, yet not hostile attitude

Ability to appreciate the users’ perspective

-70-

Test team

Benefits of effective test team identification Minimal training effort and costs

Lower Cost of Quality

Fewer people management issues

Reduced risk to the project

Controlled costs

Winning Client confidence

-71-

Training

Identify the skills required for short term and long term

Conduct training

Evaluate feedback

Periodically assess the relevance and benefits of training programs through metrics

Training requirements could be in the following areas Testing process Quality process System/ functional knowledge Technology Business domain Testing Tools Soft skills

-72-

Training (Contd…)

Benefits of effective training

Higher effectiveness and productivity of testing effort

Better test coverage of the requirements with Shortened cycles

Higher motivation for testing team

Reduced risk to the project leading to lesser rework effort in testing

-73-

Some facts which can motivate a testing team Using tools to test an application

Identifying performance bottlenecks

Testing multiple interfaces

Being responsible for quality

Working on complex domains

Working on multiple levels of testing

Motivating testing team

-74-

Rajneesh Malviya (DM) says “Challenging the testing team to own their task makes them contribute their best”

Expert speaks

-75-

Match strengths and work assignment

Empower the team with responsibility

Make work more challenging

Equitable allocation of work

Personal attention to each team member

Regular communication

Recognize achievements and celebrate every success

Have fun always - even under stress

Best practices in team management

-76-

Case study

Resolve the case study and prepare a communication strategy for the given scenario

Sesha is appointed as a manager to handle a testing project with 8 testers for aRETAIL client. Sesha has expertise in System Integration testing only. The scopeof testing in the new engagement is manual, System Integration, Usability,compatibility and the project has two parallel tracks spanned across 3 geographiclocations. Team composition is 2 experienced testers at Australia 1 in USA 4 freshers + 1 experienced tester at offshore. Sesha is at offshore The client is at USALooking at the tough timelines, Sesha decides to help his team with execution.Sesha needs help to devise a strategy for communication with the team and client

-77-

Case study - Answer

Key points 1. Two parallel tracks of the project 2. 3 geographic locations and 4 freshers at offshore 3. Plan for Knowledge Transition Phase / Pilot Phase 4. Plan for Steady Phase / Long term Phase

For details of the resolved case study refer to the attached document

Answer - Communication

Stakeholder management – Development team

-79-

Development team Diagram below shows development and testing phases in parallel

Regular communication with developers to bridge the gaps (same version of documents being used in development and testing, defect resolution, build management etc.)

-80-

Role play

Vishal is a very good Test Engineer. He is testing a web application. The defectslogged by him are generally accepted by the development team and fixed.

While doing an internationalization testing, Vishal logs about 10Defects which are related, and developer Lead Anil fights with Vishalsaying he should have logged one defect and said ‘similarly in location x,y, z…’. Vishal is of the opinion that it is easy to track when there areDifferent defects.

Anil says that the quality of the product will not meet the set criteria if 10Defects are raised and wants Vishal to reduce the number of defects.

What should Vishal do?Role play: 2 members one as Vishal and other one as Anil

-81-

Role play - Answer

For details of the resolved case study refer to the attached document Dev Vs. Test

Test management challenges

-83-

Other Challenges Training on the Technology, Domain, Testing, Tools

Prioritization of activities

Manual Testing (Removing monotony)

Making time for testing

Communication with multiple stakeholders

Dependency on external factors like interfaces, development teams to plan the testing activity and

Motivating the testing team

Building good relationship with development team

Multitasking (Project, Team, Communication etc..)

-84-

Test Management Best Practices Knowledge Management – for smooth running of project even when

people move in/out of project

Team motivation to get the best from the them

Prioritization of tasks - With best combination of priority and severity

Regular communication with stakeholders

Timely escalation to avoid emergency situations

Timely audits to check process compliance

Use of RTM

-85-

In a RTM each requirement is mapped to

Business criticality - Helps determine the depth of coverage required for the requirements. Higher the criticality, greater the number of test cases to test that requirement

Market priority - Helps decide the prioritization of the testing effort for the test cases written to validate that requirement

Risk of not testing - Captures in qualitative / quantitative terms the impact of not testing that particular requirement

Main data captured in an RTM are listed below

Business Functionality

Test Scenario

Test Case

Test Data requirements

Complexity Business Criticality

Market Priority

Risk of not testing

Best Practices - Risk based testing using RTM

-86-

Indira R (GTM) says “How quickly can, one scale up to meet client expectations is an important factor in client relationship management”

Expert speaks

-87-

Case study

Resolve the case study and prioritize the tasks

Two testers Rekha - well experienced in functional test automation and Nirmala -well experienced in performance testing have been working together for 2 yearsnow in the same project along with their manager Nisha. Both of them areexcellent performers and have never missed the timelines.

Nisha in addition to her current project (test management of 15 members doingmanual test execution), has to present a proposal to the client for a very goodopportunity, involving both functional automation and performance testing within aweek. Nisha decided to involve Rekha and Nirmala.

Off late Nisha started noticing that Nirmala is not showing any interest in theproject now. For any task she wants to approach Rekha to do the work. Rekha hasbeen very busy with her deliverables but always obliges Nirmala.

-88-

Case study (Contd…)Few days past Nirmala stops coming for work without any notice. When contactedon phone she says she will join after a week. Nisha asks Rekha to take overNirmala's work immediately and Rekha disagrees saying she is alreadyoverloaded and she cannot take anymore. Nisha accuses saying had Nirmalagiven this work to her she would not have fought like this. Rekha does not like it.

Background - Rekha has got a very good offer and plans to resign after 10 days.She had told Nirmala at lunch 3 days ago and since then Nirmala is disturbed.Nirmala is on leave to search for a job as she feels, without Rekha she cannotwork in the present company.

Rekha decides to submit her resignation immediately.

What would Nisha do now? Her proposal work, current project and two excellentmembers are at stake. Prioritize the tasks and suggest what best done by Nishawould keep all the three activities out of trouble.

-89-

Case study - Answer

Key points 1. Avoiding person dependency 2. Motivating the team 3. Maintaining good rapport between manager and team members 4. Prioritization of tasks

For details of the resolved case study refer to the attached document

Answer - Task Prioritization

Summary

-91-

SummaryInitiation Thorough understanding of the contract, proposal, application,

customer, objective of the project

Deliverables, commitments, estimates, plan and strategy approved by client

Understand the project value addition to each stakeholder

Define critical success factors for the project

Standardize templates, checklists, process

Structured formal communication

-92-

Summary – Strategy / Plan

Identify required Testing types

Identify test Data requirement and

process to identify the same

Decide tools and monitoring Parameters

Assign roles and

responsibilities

Decide if a Proof of Concept (POC)

test is required

Decide on Test Environment

Setup

Test Strategy

/Plan

Decide on entry/exit criteria

and finalize scope

Effort estimation and planning

-93-

Summary (Contd…)Execution Request acknowledgement and signoff for deliverables

Regular milestone analysis and release to get intermediate feedback and process improvement

Defect management

Closure Identify the re-usable artifacts

Publish the best practices and lessons learnt

Provide feedback to team on their performance

Questions?

Further Reading *PMElitePRIDEQIVS

* Open in the slide show to access the links

Thank YouIVS-TRAINING

Any doubts or suggestions for improvement can be forwarded to: IVS_TRAINING@infosys.com