Integrating Six Sigma and Software Testing Process for ...crestechglobal.com/wp-content/uploads/2013/10/WhitePaper_SixSigma… · and Software Testing Process for Removal of ... Based

Visit us at: www.crestechglobal.com

White Paper

Author – Ankit Tuteja

[email protected]

Integrating Six Sigma and Software Testing Process for Removal of Wastage & Optimizing Resource Utilization

24 October 2013 With resources working for extended hours and in a

pressurized environment, there comes a situation where

knowingly or unknowingly we have to compromise with the

quality. The reason behind this is that everyday resources

have to spend time on doing repetitive tasks which consumes

a lot of time and effort thus impacting the project timelines,

budget and increasing associated with the quality.


1. Introduction

In the past few years we have found that every IT organization whether it is a Service based industry or a product

based industry, are emphasizing a lot on QA & Testing in their projects. Either they are hiring testers from outside to

build there in house testing teams or they are exponentially increasing the size of their existing teams or they are

outsourcing it to other testing companies. But in all the cases there is one major problem that can be seen, even with

the increased size of these teams often there comes a situation in which we are not able to meet the deadlines. With

resources working for extended hours and in a pressurized environment, there comes a situation where knowingly or

unknowingly we have to compromise with the quality. The reason behind this is that everyday resources have to

spend time on doing repetitive tasks which consumes a lot of time and effort thus impacting the project timelines,

budget and increasing associated with the quality.

Current Scenario

Taking an example of the QA team of a reputed travel product based company, on an average each resource is

expected to work for 48.5 hours weekly (9.5 x 5 hours (weekdays) + 1 hour (average working time of 1

resource every Sunday)). This is the ideal condition that should happen but this is not the case in reality. As per the

data collected from the timesheets of the team members, the working hours came out to be 63 hours weekly (12 x 5

hours (weekdays) + 3 hours (average working time of 1 resource every Sunday)) which is 13.5 hours

overhead of the actual working time of single resource every week. But even working for these extended hours the

team is not able to meet its deliverables on time & in the budget allocated to it with the risk of compromising with

the quality.

This is an example of one organization but this is the story of almost every other IT industry which makes this topic

a good study to implement Six Sigma techniques so as to improve the overall process thus resulting in the effective

Resource Utilization and Optimizing the costs related to the project delays. With the help of Six Sigma

Concepts and using the DMAIC approach in our projects we can actually overcome this problem to a great extent by

minimizing wastage of time in doing repetitive tasks and by automating various manual activities which can actually

increase the daily productivity of a resource.

Problem Statement

The purpose of this paper is to establish Process Improvement Techniques to rapidly address concerns that

sometimes why even working for extended hours in a project the testing teams are not able to deliver modules on

time and have to risk the quality. We will be closely analyzing the existing performance reports, examining source

data, identifying inconsistencies in existing processes, developing standard control procedures, and implementing

improvements to ensure optimum utilization of all the resources and delivering modules with quality.

Goal

To design and develop a complete set of solutions to address the root causes behind the wastage of time on daily

repetitive tasks which are not required that the QA team have to work on. We will now determine the full extent of

the problem through data analysis, interviews, and other tests and develop solutions for improving the processes and

monitor the results of the implemented solutions.

Business Impact

No matter how small the project is, its timelines & quality matters a lot. On the basis of present projects, the

timelines, budgets and pipelines of the future projects are decided. If the current process is broken then there are

material distortions to the number of hours that should be spent on a project than the actual number of hours that are


currently being spent. This study will attempt to fix this broken process by suggesting various improvement

techniques through which we can identify major critical problems that impacts our deliveries in terms of resource

utilization and then carefully designing and implementing solutions to overcome these problems hence optimizing

overall resource utilization. This study will also give management the correct picture of the total man hours actually

spent on a project so that they can accordingly plan the upcoming projects in terms of Cost, Time & Quality.

2. What is Six Sigma?

The term Six Sigma originated from terminology associated with manufacturing, specific terms associated with

statistical modeling of manufacturing processes. The maturity of a manufacturing process can be described by

a sigma rating indicating its yield or the percentage of defect-free products it creates. A six sigma process is one in

which 99.99966% of the products manufactured are statistically expected to be free of defects i.e. 3.4 defects per

million, although, this defect level corresponds to only a 4.5 sigma level. Motorola set a goal of "six sigma" for all

of its manufacturing operations, and this goal became a byword for the management and engineering practices used

to achieve it.

Figure 1

Six Sigma – Tools & Techniques

As shown below, Six Sigma is based on a DMAIC approach that comprises of 5 sequential phases which are

Define, Measure, Analyze, Improve & Control which leads to an improved process at the end.

Figure 2

The term "Six Sigma Process" comes from the notion that if one has six standard deviations between the

process mean and the nearest specification limit, as shown in the graph below, then practically no items will fail to

meet the specifications. This is based on the calculation method employed in process capability studies.


Capability studies measure the number of standard deviations between the process mean and the nearest

specification limit in sigma units, represented by the Greek letter σ (sigma). As the process standard deviation goes

up, or the mean of the process moves away from the center of the tolerance, fewer standard deviations will fit

between the mean and the nearest specification limit, thus decreasing the sigma number and increasing the

likelihood of items outside specification.

Figure 3

There are various methods or the tools & the techniques that are used in the DMAIC approach for e.g. SIPOC

diagram, CTQ characteristics, Pareto analysis, etc. We will be covering them in details in the coming phases.

3. Integrating Six Sigma with Software Testing process

In this study, we are using a word ‘Defect’. This term ‘defect’ is not related to the one in testing. It is a term used in

Six Sigma which means cause of errors which affects our process.

Below is the diagram which shows us that how can we incorporate the 5 phases of Six Sigma in our testing projects

so as to identify the Critical problems that impact our projects, propose their solutions and can continuously monitor

them.


Figure 4

3.1 Define Phase

Consider below SIPOC diagram applicable to all software testing lifecycles. This is a very high level diagram to

explain the characteristic flow of STLC.

Figure 5

To go into further details, below is a detailed functional flow diagram to understand the system deeply.


Figure 6

This diagram describes the standard software testing process in any organization i.e. how a developer gets

requirements from the user, how the QA gets the files for testing from the developer, how developer & QA interacts

in a bug lifecycle, how UAT is done and finally how the product is released in the market.

Critical to Quality Characteristics

CTQs are specific factors or attributes that are associated with the products, processes, and services that customers

consider extremely important. This is often identified by listening to the customers through surveys, interviews,

complaints, and other ways for fully understanding the requirements of the customers. Six Sigma professionals refer

to this as Voice of the Customer or VOC. Once you have captured VOC, you can then translate this information into

a CTQ tree diagram. Considering the e.g. of a travel product based company again, we had a discussion with the

entire QA team (who are our end users in this case) to capture there problem areas. On further research it is found

that these problems are not limited to this organization but almost to every other organization who is in the process

of software development & testing. On the basis of that a standard CTQ tree is derived as shown below.


Figure 7

3.2 Measure Phase

Pareto Analysis

Few projects from different domains like travel, CMS and telecom are sampled to perform some analysis. After

doing brainstorming sessions with QA teams of these projects, we have shortlisted 5 repetitive tasks which if

minimized will reduce the overall time spent on a project.

These tasks are:

T1 Manual sanity of the build after every release

T2 Deployment on testing servers

T3 Inadequate product knowledge

T4 Rework due to requirement not clear

T5 Test Case authoring/scripting

Table 1

Based on the data collected from the sample projects which is the average number of hours spent by each resource

on the above tasks, we have prepared a Pareto Chart as shown below:

Figure 8

Contributor # of Hours (per day)

% Cum

35.1

21.6

16.213.5 13.5

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Manual sanity of the build after every releaseDeployment on testing servers Test Case authoring/scripting Inadequate product knowledgeRework due to requirement not clear


Manual sanity of the build after every release

6.5 35.1

Deployment on testing servers 4 21.6

Test Case authoring/scripting 3 16.2

Inadequate product knowledge 2.5 13.5

Rework due to requirement not clear 2.5 13.5

Table 2

The results of the Pareto analysis help us to prioritize our problems (P1, P2, P3, P4, P5) and if the time spent on

these problems can be minimized then this will result in overall process improvement in our testing projects.

Data Collection Procedures

Data is the most important part of this study and we have used various tools specific to our sampled projects like

Bugzilla, Clarizen, etc. to generate various reports so as to collect data at resource level and then using various

techniques like attribute MSA, gauge R&R, etc. for analysis of this data.

For our study, we have taken the data of 3 months (Nove’12-Jan’13) of travel product company that we have

discussed above. This data shows that how much time a team of 12 QA resources have spent on the problems P1-P5

for 3 months working on 179 different projects. Below is the pivot chart representing that data:

Figure 9

Also, below is a pie chart that shows us how much % of time each QA resource have spent out of his/her total

productive time on working on problems P1-P5.


Figure 10

Calculations

The calculation of a Sigma level, is based on the number of defects per million opportunities (DPMO).

In order to calculate the DPMO, three distinct pieces of information are required:

a) The number of units produced

b) The number of defect opportunities per unit

c) The number of defects

Using the formula:

DPMO = (Number of defects X 1,000,000)

(Number of Defects Opportunities per Unit X Number of Units)

Where,

Total number of projects examined or Number of Units 179

Number of projects in which working in extended hours required

or Number of Defects 97

Total number of resources or Number of defects opportunities per unit 12

Hence, DPMO is 45158.28678.

Using the Sigma table below, the current Six Sigma value will come out to be 3.19.

Six Sigma Table:

Value DPMO 1 690,000

2 308,000

3 66,800

4 6,210

5 320

6 3.4

Table 3

After examining other sampled projects we found that there sigma rating is also around this value itself.

Summarize Test Results


To find the critical defects that need our immediate attention, we again carry out Pareto Analysis. Below are the

results for the same.

Type of Defect or Error Estimated Impact of Defect (hours) % of Total

P1 1512 32% < Critical Defect

P2 1116 23%

P3 900 19%

P4 630 13%

P5 630 13%

Total 4788

Table 4

Figure 11

The above Pareto analysis shows that if we work on our critical problem P1 then this will overall increase our

performance to a great extent (based on 80-20 principle).

Histogram

To go into further details of problem P1, we prepared a histogram based on the data collected from travel product

based organization. Twelve observations were made over 12 QA resources regarding the time spent by each QA

resource on the task P1 (Value) which is considered as the Critical Defect.

Observation Value 1 179 2 51 3 93 4 173 5 140 6 122 7 52 8 85

0

500

1000

1500

2000

Impact of Number of Defects

P1 P2 P3 P4 P5


9 185

Maximum 185.4

10 173

Minimum 51.2

11 152

Range 134.2

12 119 Table 5

Number of Intervals based on the Square Root of Observations = 3.464101615 rounded off to 3

Minimum intervals required = 5

Width of each interval based on Range / (No. of Intervals - 1) = 33.56 rounding off to 34

Interval Description Frequency

1 Less than or Equal to 51 1

2 Between 52 and 85 2




Table 6

Figure 12

In the above Histogram we can see that we have a defined peak which further proves that working on problem P1

will definitely enhance our overall process.

3.3 Analyze Phase

In this phase we will analyze the root cause of our problem P1 i.e. time spent on doing manual sanity of build after

every release.

Fish Bone Diagram

Discussing our Critical Defect with experienced QA team members and using various brainstorming techniques,

below are major causes that we’ve listed responsible for spending more than the required number of hours on doing

sanity of the build after every release:

1. Manually executing test Cases which can be automated

2. Not enough knowledge of new features that impact our sanity

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1 2 3 4 5

Fre

qu

en

cy

Intervals

Frequency Distribution

Freq


3. Improper KTs in case of new resources

4. Server Slow

5. Ambiguity in requirements from Dev end

Doing a fish bone analysis of these problems we get a diagram as mentioned below:

Figure 13

From this diagram we can explore the causes of our problem P1 and can provide a solution to it by discussing with

SMEs or doing some brainstorming sessions. Other analytical techniques like FMEA, 5 Why analysis, etc. can also

be used to perform root cause analysis of our problem P1.

Scatter Plot

A scatter plot is used when a variable exists that is under the control of the experimenter. In our scatter plot, we will

use time spent on problem P1 as a variable (y-axis) and total time spent on project as a constant (x-axis) and

will show a relationship between these two.

Step 1 - Enter the X and Y data that we want to plot. This is a simple regression analysis i.e. we want to see if a

change in the X value correlates in some way to a change in the Y value.

X Values Y Values 19 6

140 57 60 21 57 18 20 6 For total number of hours spent on a project,

22 6 how does it correlate with the

14 3 number of hours spent on P!

12 3 22 6 30 9


125 45 Table 7

Step 2 - Insert a Scatter Chart with Trendline:

Figure 14

From the scatter plot above we can clearly see that the points are equally scattered around the line. This shows a

strong dependency of our variable which is the number of hours spent on problem P1 on our constant which is total

number of hours on manual sanity spent for a project.

3.4 Improve Phase

After all the measurement and analysis, now it is time to actually implement our observations in the practical

scenarios, measure the changes and monitor the progress that occurs by implementing 6 Sigma methodologies.

Solutions Rating Matrix

It is a 5 step process in which we jot down all the possible solutions, rate them on various characteristics and then on

the basis of these ratings we decide which solution is best to implement and will give effective results.

Step 1: Establish Criteria for Rating Your Solutions

Step 2: Describe each rating from lowest to highest for each of the criteria in Step 1

Table 8

Step 3: Describe each solution and assign a rating as defined in Steps 1 & 2


Table 9

Step 4: If you want to weight the criteria so that some criteria are more heavily considered than others, then assign

weights and recalculate the ratings

Table 10

Step 5: Place each solution into the following Cost Benefit Matrix to quickly assess its return

Cost - All resources to implement, such as funding, personnel, time, etc.

Benefit - Both tangible and intangible benefits such as fewer errors, reduces time, improves customer satisfaction,

lowers costs, etc.

Figure 15

Benchmarking Report

We need to benchmark the points for the existing process which we will use for comparison after the process has

been improved. Again considering our travel company example, taking 4 major projects, we will identify the

number of hours spent on the tasks P1-P5 and will take the minimum time as Best in Class.

Project ID P1 P2 P3 P4 P5

64864 25% 21% 21% 16% 10%

64880 9% 7% 7% 5% 3%

66414 21% 16% 16% 12% 8%

68782 20% 15% 15% 11% 7%

Best in Class 9% 7% 7% 5% 3%


Average Performance 19% 15% 15% 11% 7% Table 11

Figure 16

To ensure that there is no leakage in the improvement methods that we have suggested, we can perform error

proofing of our Critical Process. This can be a simple Q&A process in which we can have some questions like

What is a defect, What are sources of documents, What positions are responsible for running this process, etc. and

can have a discussion on these questions with SMEs, QA team and other relevant stakeholders.

After this is required a corrective action plan stating our immediate action items and long term plans.

3.5 Control Phase

This phase will suggest various control methods & techniques to make sure that the solution that we have provided

will be implemented more effectively.

Control Plan for Critical Process

Below is a sample control plan which can be used to monitor the entire process. It will be continuously updated

based on the coming updates and changes related to this project.


Table 12

Reaction Plan

To avoid all the breach in the process, monitor defects and take appropriate actions wherever leakage occurs, a

reaction plan is designed. This reaction plan is shared with all the relevant stakeholders. Below is a reaction plan that

we have designed to cater to our solution proposed for the problem P1.


Figure 17

Before – After Trend Chart

According to our analysis, if this solution is implemented as described then we can see some good changes in the

overall testing process of a project. Below is a trend chart that we plotted to show the difference between the

performance of the current process and the expected process for the travel product based org. For current

performance we used the data available for the time period of November, 2012 to January, 2013 and predicted the

result for consecutive 3 months of the new solution implementation.

Table 14

Figure 18

Conclusion

This is a study which is carried on a sample of standard testing projects of various domains like Travel, CMS, etc. In

some testing projects, the defects may be different but the overall solution to the problem using the methods of

DMAIC as suggested in this paper will remain the same.

Considering again our travel product based organization’s project, as per our predicted results, we can see the total

number of defects coming down from 45% to 21% as per the above trend chart. This means that our critical defect

that is the number of hours spent on sanity will reduce from 42 hours per QA resource per month to 28 hours per

QA resource per month.

Also, if everything goes as expected then we can achieve new sigma value of 3.54 from the current sigma value of

3.19 which means a reduction of approx. 24,209 defects per million.

References

Six Sigma Overview: http://en.wikipedia.org/wiki/Six_Sigma

About the Author

Ankit Tuteja, a technology & quality enthusiast is currently working as an Associate Team Lead – Testing at

Crestech Software Systems Pvt. Ltd. He is an ISTQB certified software tester and has achieved Green Belt in Six

Sigma from Indian Statistical Institute, New Delhi. He is having more than 3 years of experience in Testing &

Quality Analysis.

http://en.wikipedia.org/wiki/Six_Sigma

Documents

Integrating Six Sigma and Software Testing Process for ...crestechglobal.com/wp-content/uploads/2013/10/WhitePaper_SixSigma… · and Software Testing Process for Removal of ... Based