Fundamentals of Test Execution V1.0

8/3/2019 Fundamentals of Test Execution V1.0

http://slidepdf.com/reader/full/fundamentals-of-test-execution-v10 1/67

© 2007, Cognizant Technology Solutions. All Rights Reserved.The information contained herein is subject to change without notice. C3: ProtectedC3: Protected

Fundamentals of Test Execution

Session 1: Fundamentals of Test Execution



2

About the Author

Created By: Rajarajan(225558); Balasubramanian(226466);

Elangovan(208633); Venkatakrishnan(102083);Rohit(158977);Vishnu Priya(179360); Mathuram(105686)

CredentialInformation:

3-12 years of Experience in Software Testing

Version and

Date:

Version 5.0 and 22 Feb 2011



3

Icons Used

Questions

Contacts

Reference

Try it Out

Hands onExercise

CodingStandards

Test YourUnderstanding

Tools

AWelcomeBreak



4

Fundamentals of Test Execution Session[1]:Overview

Introduction: This session explains thevarious activities performed during the testexecution phase and the process to log thetest results and the defect managementmethodology



5

Fundamentals of Test Execution Session[1]:Objective

Objective:After completing this chapter, you will be able to:

» Describe various activities involved during TestExecution

» Describe and interpret a Test Execution Plan

» Describe Test Environment and the need for it

» Explain Build Management and Verification Process

» Describe the process of Test Execution

» Describe Triaging and Defect Management

» Differentiate Test Execution for Fresh developmentand Maintenance Testing



6

Test Execution: An Introduction

Fig: Test Execution Process

Planning

Activities

for

TestExecution

The Test Execution involves:

» Initial Planning for TestExecution

» Build Verification Process

» Test Case Execution

» Test Log Creation

» Defect Reporting and Tracking» Defect Triaging

» Updating Traceability Matrix

» Re Testing of Defects

» Regression Testing

» Test Execution Status reporting



Follows the Test Design phase in STLC

Process of executing a series of test cases asdefined in the test plan to validate if theapplication meets the requirements

The application is tested and the defects arereported to the development team

The fixes provided by the development teamare retested and ensured that they do notrecur, the fix does not introduce any new

defects Can be manual or automated using a tool

Test Execution: AnIntroduction(Contd.)



Below activities need to be taken care beforethe start of Test execution» Test Cases Review and Sign Off

• All the test cases that are identified for the specific level andtype of testing need to be approved and signed off by therespective stake holders

» Test Cycles• The number of rounds the testing will be conducted and the

scope of testing at each level need to be identified in the Testplan and agreed upon by all the stake holders

Planning for Test Execution



» Test Lab set up

• The methodology for deploying the build in the respective testenvironment need to be devised and approved by thedevelopment, testing teams and the customer

• Test Lab set up also includes creation of a test suite with the testcases which need to be executed in a specific sequence

– Creation of Cycles for each level of testing

– Identification of Smoke/Build verification test cases– Identification of Test Suite(collection of test cases) per cycle

– Frequency of Build delivery per cycle

» Task allocation / Run plan

• The allocation of the test cases to be executed need to be donebased on the skill set and the availability of the testers

Planning for TestExecution(Contd.)



Planning for Test Execution:Execution Run Plan

Execution Run Plan



Sample Execution Run Plan

Execution Run Plan - Sample

Execution Run

Plan Sample



» Risks and Mitigation Process

• The potential risks that are identified for Test execution duringthe course of the project are revalidated and the availability of the mitigation and contingency plan for each of these risksneed to be ensured

» Escalation Process

• The escalation process if an unforeseen event occurs during

test execution need to be arrived and agreed upon by thestake holders

Planning for TestExecution(Contd.)



Testing an application hosted in the desktops/Servers

used by the developers for coding purposes is notrecommended for the below reasons

» Both testers and developers will have access to the code

» Behavior of the application may be unstable if thedevelopers work on the code while the testers are testing

» The developers may have used simulated programs duringcoding and the testers (being unaware of this) may alsouse the same for testing. This may result in huge failurewhen the application is deployed in the productionenvironment

» Parallel or overlapping data update by various teams

Need for a Test Environment



Mirror the production environment as closely as possible

The availability of the test environment exclusively forvarious types and levels of testing is always recommendedbut is subject to every individual project like

» Entire test environment is exclusive

» Database is the same but the hardware used is different

» A sub set of the data will keep changing for every cycle andthe master data remains the same

Need for a Test Environment(Contd.)



Configuring of Test Environment can includesetting up of the following:» Software required(Ex. Tools like Eclipse, Crystal Reports)

» Hardware required(Ex. Special interfaces like HandheldDevice, Touch Screen)

» Server to host the application in the desired configuration» Database setup/Server Setup to run the application

» Appropriate level of access for test team to accessDatabase, server and application

» Appropriate level of access to external application if thetesting scope covers for the same

Test Environment Configuration



» Application to be tested

» Test management tool set up

» Access to the remote desktops and the connectivity tothem

» Contact Details of the customer’s helpdesk should beavailable during the testing if the environment is slow

» Environment availability – All the environments may notbe available full time during test execution. There may beexceptions like

• An application will be running only during US business hours

• An application will be down for maintenance during US publicholidays

Test Environment Configuration(Contd.)



Test Management Tool is a software used to plan and

execute testing activities like capturing requirementsand test cases, managing defects, collecting metricsand analyzing the reports

Version of the tool or the instance agreed upon in theTest Plan needs to be installed in the test environment

desktops Project has to be created in the tool to manage the test

activities and right level of access should be given to alltest and development users

All requirements should be uploaded into the tool by

Business Analyst / Testing team

Test Environment: Test ManagementTool Set Up



18

Cycles for execution needs to be created in the tool

under the project» Cycles agreed upon in the Test Plan is considered and may

vary during the test execution

» Cycle period or dates should be mentioned and it shouldbe mapped with the test cases that we plan to execute

The tool may also be integrated with the third party tool

Test Environment: Test ManagementTool Set Up (Contd.)



19

The test environment should be as close to the production

environment in terms of » Database(DB) set up

» Levels of users defined and the no. of users defined for eachlevel

» Volume of data in the database

» Type of data Detailed Environment Requirement has to be addressed and

tracked in a separate document like

If there are multiple test environments, the abovementioned points need to be ensured in all theenvironments

Test Environment: Database SetUp



20

Test Users set up

» The Appropriate roles identified as test data for testingneed to be created and obtained from the customer for allthe instances of test environments

» Once the test data is received, it is always better to ensureif the credentials provided works and has the appropriate

levels of access» In some cases, the credentials provided may be valid only

for a limited period of time. This may require us toreschedule the run plan for execution

Batch Job dependencies (if there are any) need to beidentified and planned accordingly

Test Environment: Database andTest User Set Up



21

Test Environment: Test User set up

21

Testers

Rights to accessDBServer

Application

Server

ExternalApplication

h



Test Data Set Up in the TestEnvironment

22

Primary / Secondary Data

» Primary Data is generally referred to the master data and is obtained fromcustomer. Example for Primary data tables are Employee Data, Countriesapplicable for the application

• Secondary / Transaction data

» Secondary data can also be obtained from the customer if the application(previous version) already exists in Production

» If it is a new version of application, Secondary data can be created and

customized during testing» Test team is responsible for ensuring the test data availability before test

execution



23

Test Data Management in TestEnvironment

23

Test data should be available before the test

execution

The data used for every cycle of testing mayvary like» The entire database is refreshed

» A subset of data is retained and reused

» Only the master data is retained

Tools can also be used for creating test data

Batch queries should be created for insertion/

Updation / Deletion of data in the tables Ownership of data has to be clearly defined

among and within the teams

T D M i T



24

Test Data Management in TestEnvironment (Contd.)

24

Sample Queries for cleaning up and inserting

the data» Delete from empdetail

» Delete from emp

» Insert into emp(table columns)(table Values)

» Insert into empdetail(table columns)(table Values)» Update can be done as required

Note: While doing deletion, secondary/transaction data should bedeleted first and then primary but it is vice versa during insertion



25

For every cycle of test execution, the

development team shares a controlled versionof software (Build) which needs to be deployedin the respective test environment and tested

Each build is associated with a version

number and a release note attached The release notes of the build primarily

includes information of the features of thesoftware included in the build and the

deployment details

Build Management



26

Frequency of build delivery to the project team

is as per the Test plan and in consensus withthe stake holders of the project

The files/DB files/Batch Programs used togenerate a build are controlled by a

configuration tool/ team assigned for it Build Manager is responsible for

communicating to the stake holders on therelease of the build to test with release notes

(In Some Projects, Program Manager) The version of the build is incremented for

every change in the build

Build Management (Contd.)



27

Build Verification Process

Build Manager

Test Team

BVT Failed

BVT Pass

Release Notes

DevelopmentTeam

Build the compiledcode into software

Adds the releasenotes

BuildVerification

usingSmoke /

Sanity Test

Test ExecutionTest Team



28

The units of software developed is integrated, compiled

and built as a single unit by the developers

The Build Manager will add the release notes to thisversion of the software build and release it for testing

The Testing team runs the identified Smoke and Sanitytest cases on the build, once it is ready

If the smoke/sanity test cases pass with the desiredresults, build will be accepted and the testing team willproceed with the Test Execution

If the smoke/sanity test fails with critical defects that

the team is not able to proceed with testing, the buildwill be rejected

Build Verification Process (Contd.)



29

Release notes includes details on» Version of the build

» Features included / not included in the build

» Defects that are fixed / not addressed in the build

Testers need to test only those features thatare mentioned clearly in the release notes

Importance of Release Notes



30

Test Your Understanding

Test YourUnderstanding



Questions on Build Management

BVT means ?

a) Smoke b) Regression c) SIT

What is the percentage of BVT test caseshave to be passed ?

a) 85% b) 90% c) Depends on the project

Release notes should be sent by ?a) Build Manager b) Program Managerc) Test Manager

31



32

Test Execution

Test Case Test Log document to log actual results

Identification of test cases assigned to each tester forthe day

Does theApplicationbehave asdesired ?

Log Test Results withproof of execution

Tester understands the dependency between testcases and identifies the sequence among them

Tester ensures the prerequisite of each test case ismet before execution

Executes the test case / Retest if the defect is fixed

Log defect / Updatestatus of defect

Update TraceabilityMatrix

Tester uses the test data identified for the test caseduring test execution

Test Case Execution and Logging



33

Test Case Execution and LoggingResults

Execute Step 1 of a test case

Does theApplication behave

as desired ?

Execute next test case

Log defect / Updatestatus of defect

Execute the next step of the same test case

Log Test Results for the test step withproof of execution

Is the next test stepavailable and

executable with thedefect raised for theprevious step in this

test case?




34

Each test case is executed step by step

After executing the test step, the actual behavior of theapplication is logged in the “Actual result” section of thecorresponding test step

If the actual result is the same as the “Expected result”, theapplication behaves as desired and hence the test step ismarked as “Pass”

If the actual result is the not same as the “Expected result”, theapplication is not behaving as desired and hence the test stepis marked as “Fail”

While logging the “Actual result” the behavior of theapplication, needs to be mentioned clearly. This will help the

following:» Other testers in the team to understand the impact of failure of this

test case on the test cases that they would execute

» Development team to identify the exact issue and addressing it

» Retrospection of the cause to introduce this defect

Test Case Execution and LoggingResults




35

If a particular test step fails, a defect needs to be logged in the

defect log and it is recommended to execute the consecutivetest steps if possible

This may not be possible in all cases, then the tester mayproceed to execute the next test case

A test case is considered to be pass only if all the test steps inthat particular test case pass

Test Case Execution and LoggingResults (Contd.)

ExampleIn the Leave Management system, While applying leave, if the “Type of Leave” combo box does not have the label name as expected, a defect needs to beraised and the testing can be continued for the consecutive test steps of thesame test case

ExampleIn the Leave Management system, While applying leave, if the “From” and “To” dates are not editable

Logging of the Actual Results in the



36

The actual result of a test case and the details of execution are

logged in a document called Test Log/ Test Management tool,irrespective of the behavior of the application

Test log sheet is a copy of test case document with additionaldetails like Actual results, Pass/Fail, Defect ID and the Testername who executed it

Test Log is a chronological record of the all the informationabout the test case execution

» What are the test cases executed

» In What order it was executed

» Who executed those test cases(Tester)

» Status of the each step of the test cases (PASS/FAIL)

» This can be created manually or using test management tool For some specific applications it is mandatory to log the test

results physically in the test case document, to be compliantwith the regulations

Logging of the Actual Results in theTest Log



37

Sample Test Log

Sample Test Log Sheet



38

Sample of the Test Case run status

Sample Test Case Run Status

Test Case Run Status



39

Testers raise defects when the expected result

and the actual behavior of the application donot match

The defect management methodology definedfor every testing project would be different

The testers need to follow the one defined inthe Test Plan of that particular project

The defect logging can be done in a simpleexcel document or any defect management /

test management tool adopted by the team

Logging a Defect



40

Key Elements of a defect includes:

» Defect Id• A unique Id is assigned to each defect raised during testing.

Usually, a naming convention decided by the testing team isfollowed. This can be auto generated if a test management /defect management tool is used

» Defect Description

• A textual description provided to describe the defect to theother stakeholders of the project

» Steps to reproduce the defect

• The list of steps to be followed to reproduce the defect

» Test Case Id

•The Id of the test case executed to find the defect is providedfor reference

• The Test data used to reproduce the defect may also beprovided here for reference

Logging a Defect (Contd.)



41

» Assigned to

• Each defect will be assigned to a developer to fix, the name of the developer is provided here. This is usually assigned duringthe defect triage meeting

» Module Name

• The module of the application in which the defect was found ismentioned here




42

Key Elements of a defect include

» Severity

• The impact of the defect in the software is called Severity

• Every project will define the severity levels of defects in the Test Plan

• Severity of a defect is generally set by the Testing team while logging defects

• A few samples of the severity levels of a defect are


Examples for Defects–Severity:



Examples for Defects–Severity:Critical

43

All pages areblank whilelogged into

theapplication



Examples for Defects–Severity: High

44

“Half a Day” option isnot available for

“Leave Start Date”

Examples for Defects–Severity:



Examples for Defects Severity:Medium

45

The calendaricon does not

allow to click



Examples for Defects–Severity: Low

46

The label isincorrect



47

Key Elements of a defect include

» Priority• Priority of a defect is generally determined by the

delivery manager of the project as defined in the TestPlan based on the following

– Severity of the defect

– Capacity management of the development team– Time required to fix a defect

– Impact of fixing a defect on the other defects –sometimes, fixing one defect may rectify many of therelated defects

• Each of these priority levels will have a time associated

with it, to enable the developers to fix within that time-– For Example, Critical defect may have a target resolution

time of 2-6 hrs, less critical defects can have the targetresolution time anywhere between 8-48 hrs

Logging a Defect



48

» Capture screenshot of the defect

• Though the details of the defect are documented, there may be cases

where the defect is not repeatable• A clear screenshot of the defect with the relevant area highlighted will

help the development team and the business to understand the defectbetter

• It is always better to highlight the area where the defect had occurredin the screen shot

» Defect Status

• The status of a defect would change from stage to stage during thedefect life cycle

• The defect life cycle is defined in the Test Plan for a project

» Test data used to identify the defect

• This will help the other stake holders to

– Reproduce the defect

– Identify if the issue is with choosing the test data

» Version of the build in which the defect is identified

» Test Environment details in which the defect occurred




49

Sample Defect Life Cycle

49

Sample Defect Life Cycle (Defect Status and the process may vary from Project to Project as per Test Plan)

Is ValidDefect?

NEW

OPEN

FIXED

CLOSED

REJECTED /CANCELLED

REOPEN

Dev Team

Test Team

If Defectfound?

Fix of defects

TestExecution

Fix in

thisrelease?

DEFERRED

Defectfound inRetest?

TriageMeeting

Best Practices While Logging



Best Practices While LoggingDefects

Log defects with:

» Clear description of the defect – as is in the application» Appropriate tone of the language

» Screenshot(s) with

• Clear marking of the defect area

• Call outs

• Timestamp

• Sequence of screen shots for each step if required

» Highlight if there are any similar/ related defects earlier in theremarks

» Mention the other areas/features of testing affected due to thisdefect in the remarks

» Even if the defect is observed once and is not reproducible, log it

and the development team and the business will decide toconsider it as a defect or just an observation

» In the above case, log the defect with more details on when thedefect occurred and when it is not reproducible

50

d h b l



51

The traceability matrix created during the

Requirements Analysis and the Test Designphase is updated with the defect details

The defect id is mapped against thecorresponding failed test case id

This will help in tracking which are therequirements that are misinterpreted /misunderstood by the teams

Only the defect id is mapped here and the

details of the defects are obtained from thedefect log

Updating the Traceability Matrix

R i f d f



52

Defects that are fixed by the development team are

retested by the testing team to ensure that theapplication behaves as desired after the fix

The status of the defect is changed to Re-Open and isassigned to the developer if the application does notbehave as desired, even after the fix

Otherwise, the defect status is changed to closed andthe test log is updated appropriately

In some cases, fixing of a defect would introduce newdefects in the same module or any other related moduleof the application. Hence the related modules also need

to be retested

Re-testing of defects

R i T ti



53

If the application under test is a maintenance

application, the enhancement to theapplication may introduce defects in theexisting functionality of the application

Needs to be done to ensure that the

application is in tact and behaves as beforeeven after the enhancement/ defect fix

There will generally be a set of regression testcases identified or automated test suite

already available

Regression Testing

Best Practices During Test



Best Practices During TestExecution

Identify the business critical/ error prone areas and

focus on those. These can be identified from/using:» Knowledge of the application

» Business knowledge

» Programming knowledge

» End user experiences

Test the application as an end user

Try and relate to similar scenarios while testing

Collaborate with other testers/leads/SMEs in the projectto understand

» Implications of the defects on other modules» Behavior of the defect if tested with other credentials

» Behavior of the defect in various environments – Entiretest environment/hardware/software

54

T t E ti R t



55

Types of Reports

» Daily Execution Report – End of Day» Test Summary Report (No / No Go Decision) – End of

Execution

These reports will be circulated to the stakeholdersincluding

» Onsite Test Manager» Test Leads

» Dev Manager

» Program Manager

» Onsite Dev Managers

» Dev Module Leads» Client

Data for reports can be prepared manually or taken fromthe tool if a test management tool is used

Test Execution: Reports

Test Execution Status – Daily



56

The Status of test execution needs to be communicated

to the stakeholders of the project in the agreedfrequency as per the Test Plan

Below are some of the key information which will beshared with the stakeholders during test execution

» % of test cases executed

» No. of test cases passed Vs No. of test cases executed» No. of test cases failed Vs. No. of test cases executed

» Defect report• By Status

• By Severity

• By Priority

• By Module• By age of the defects

Test Execution Status DailyExecution Report

E d f T t R t



57

At the end of every test cycle, the testing team will sharethe end of test report with the stake holders

This will help the other stake holders of the project tounderstand

» Status of test execution

» Summary of defects found grouped by• Severity

• Status

• Age of defects

• Modules of the application in which the defects occur

» Summary of the modules that are impacted due to the fixesmade to the defects

» Changes introduced in the business due to the defects found

by the testing team (if any)

» Stability of the application

» Risks involved in moving to the next cycles of testing or UATwith the current state of the application

End of Test Report

Sample Daily Execution Status



a p a y u o a uReport

Daily Execution Report consists of Defect Status, Execution

Status, Defect List and the Downtime Tracker

58

Daily Execution

Status Report Sample

Test Execution: Comparison ‘App



p ppDev’ Vs ‘App Maintenance’

59

Test Environment Test Data Tools & Process Test Execution

Application

Development

The Test Environment has to be setafresh

Test Data has to be identified for all the test cases, as applicable

Test Management tool needs to be set up for the project

Defect Management related setup needs to be done afresh

Focus is more on SystemTesting, System Integration

Testing and retesting of defects

ApplicationMaintenance

The server set up for the Test

Environment would generally beavailable Only the new build of the applicationhas to be deployed in the existingenvironmentIf there are any specific changes

based on the enhancement, those needsto be taken careThe access to the DB, App server andother external interfaces may beavailable already

Only those that are additionallyrequired need to be obtained beforeexecution

Test data may be identified from

the existing production environmentOnly the additional need for thetest data required need to beidentified and obtained beforeexecution

Project would have been

already set up in the TestManagement tool New folders and other essentials need to be set for thecurrent release onlyDefect Management and the

process is already set and hence just need to be followed

Focus is more on

Regression testing of theapplication to ensure theenhancement does notintroduce any defects intothe existing functionality





When the Test Summary Report will besend?

a) End of SIT b) End of Week c) Both

Which tracker is used for down times ?

a) Down time b) Environment c) Defect

Who will take decision to go for UAT ?

a) Dev Manager b) Client c) TestManager

60

Fundamentals of Test Execution Session



61

1: Summary

Process of executing a series of test cases as defined in

the test plan to validate if the application meets therequirements

Can be manual or automated using a tool

The following need to be planned before the TestExecution

» Test cases review / Sign Off

» Test cycles

» Test Lab set up

» Task Allocation / Run plan

» Risks and mitigation process» Escalation process




62

1: Summary (Contd.)

Configuring of Test Environment includes the following:

» Software required(Ex. Tools like Eclipse, Crystal Reports)

» Hardware required(Ex. Special interfaces like HandheldDevice, Touch Screen)

» Server to host the application in the desired configuration

» Database setup/Server Setup to run the application

» Appropriate level of access for test team to accessDatabase, server and application

» Appropriate level of access to external application if thetesting scope covers for the same

» Identifying and accommodating the batch job

dependencies




63

1: Summary (Contd.)

The release notes of the build primarily includes information of

the features of the software included in the build and thedeployment details

Build verification Test validates if the build is good to proceedwith testing and does not have any major showstoppers

Test Log is a chronological record of the all the informationabout the test case execution

» What are the test cases executed

» In What order it was executed

» Who executed those test cases(Tester)

» Status of the each step of the test cases (PASS/FAIL)

» This can be created manually or using test management tool




64

1: Summary (Contd.)

Testers raise defects when the expected result and the actual

behavior of the application do not match The defect management methodology defined for every

testing project would be different

The testers need to follow the one defined in the Test Plan of that particular project

The defect logging can be done in a simple excel document orany defect management / test management tool adopted bythe team




65

1: Summary (Contd.)

Key Elements of a defect includes:» Defect Id

» Defect Description

» Steps to reproduce the defect

» Test Case Id

» Assigned To

» Module Name

» Severity

» Priority

» Screenshot of the defect

» Defect Status

» Test data used to identify the defect

» Version of the build in which the defect is identified

» Test Environment details in which the defect occurred

Traceability Matrix has to be updated with the defect details post the test execution

Defects that are fixed by the development team are retested by the testing team toensure that the application behaves as desired after the fix

Regression Testing needs to be done to ensure that the application is in tact andbehaves as before even after the enhancement/ defect fix

During and post the Test execution, various reports are generated to shareinformation on the status of the execution and the application defects with the stakeholders of the project.




66

1: Source

Disclaimer: Parts of the content of this course is based on the materials available from the Web sites

and books listed above. The materials that can be accessed from linked sites are not maintained by

Cognizant Academy and we are not responsible for the contents thereof. All trademarks, service marks,

and trade names in this course are the marks of the respective owner(s).

CSTE CBoK

ISTQB

Project experience



You have completedthe Session 1

Fundamentals of TestExecution .

Documents

Fundamentals of Test Execution V1.0