A Regression Analysis Approach for Building a Prediction Model for

System Testing Defects(Paper No: S1-3)

Muhammad Dhiauddin bin Mohamed SuffianFaculty of Computer Science & Information System

mdhiauddin2@live.utm.my

AP Dr. Suhaimi IbrahimAdvanced Informatics School

suhaimiibrahim@utm.my

Presentation Outline

• Introduction• Related Works• Research Methodology• Findings and Discussion• Conclusion and Recommendation

Introduction

• Defect prediction is very significant to the independent testing team– ensure all potential field defects could be successfully contained within

system testing phase– defects could be prevented from escaping to the end-users– achieve the target of zero known post release defects for the software

delivered to end-users

• Common understanding of defect forecast defects in software [1][2]

• Defect prediction for system testing– predict failures in system testing instead of defects [3]– predict remaining defects in software release as part of test process

simulation [4]

Introduction (cont.)• Motivation to undertake this research effort:

Assigning Assigning appropriate appropriate number of test number of test engineers across engineers across multiple test multiple test projectsprojects

• Right test Right test scenarios to scenarios to capture capture predicted predicted defectsdefects

• Better root Better root cause analysiscause analysis

• Re-align test Re-align test execution to execution to meet deadlinemeet deadline

• Action plan Action plan when actual ≠ when actual ≠ predictionprediction

• Decision by Decision by management on management on software releasesoftware release

• Stability of Stability of whole whole development development processprocess

Introduction (cont.)

• Objectives:– To analyze existing techniques of building

prediction model for system testing defects– To build prediction model for system testing

defects using statistical approach– To evaluate the proposed prediction model based

on specified acceptance criteria.

Related Works

• Defect terminology:– any flaw in the system or even in the system’s

components could cause the system to malfunction [5]

– deviation from its specification: physical software or work products [6]

– imperfection in the process as well as work product besides software [7]

– defect generated from V & V activities

Related Works (cont.)• Approaches to defect prediction:

– Term is used interchangeably with defect estimation to describe the proactive process of characterizing defects found in software in producing high quality product [8]

– size and complexity metrics: McCabe’s cyclomatic complexity as well as lines of code (LOC) e.g. Defect = 4.86 + 0.018 Lines of Code [6]

– categorized into project management, work product assessment and process improvement [7]

– used Rayleigh Model to predict defect density at different phases of project life cycle ]9]– combination of product and project metrics via regression analysis [10]– used mathematical distributions as quality prediction model as part of software fault

prediction techniques [11]– Used development information as important factor for the prediction and model quality

[12]– applying object-oriented metrics for predicting faults in open source software [13]– several inputs can be used to simulate system test phase in SDLC [14]– applied statistical method in Six Sigma to predict defect density [15]– used defect decay model to predict remaining defects in on-going testing process [16]

Related Works (cont.)• Issues:

– Strength: Easy to use, efficient, effective and able to indicate the process performances; Weakness: need to have sampling, require stable process and does not account for changes [7]

– Critiques [6]: • unknown relationship between defect and failures• problems with multivariate statistical approach,• problems of using size and complexity metrics as sole predictors of defects• problems in statistical methodology and data quality• false claims about software decomposition

• Measuring the success:– measuring the percent of faults found in the identified files [17]– help in maintenance resource planning as well as software insurance [18]

Research Methodology

Source of data:ONE applied

R&D organization

V-shaped process model

Metrics from V&V:•requirement review•design review•test plan review•test cases review•code inspection & unit testing•system testing

Software type:•Web-based•Component-based

Language:•PHP•.NET•Java

Findings and DiscussionPrediction of

System Testing Defects

Findings and Discussion(cont.)

Initial data set

Initial regression analysis result

Findings and Discussion (cont.)Revised data set

Findings and Discussion (cont.)

Findings and Discussion (cont.)Verification result

Functional Defects Functional Defects = 4.00 - 0.204 Requirement Error - 0.631 Coding Error += 4.00 - 0.204 Requirement Error - 0.631 Coding Error + 1.90 KLOC – 0.140 Requirement Page + 0.125 Design Page 1.90 KLOC – 0.140 Requirement Page + 0.125 Design Page

– – 0.169 Total Test Cases + 0.221Total Effort Days0.169 Total Test Cases + 0.221Total Effort Days

Selected prediction model for initial implementation

Conclusion and Recommendation• Achievement:

– Demonstrated the successful construction of prediction model for system testing defects by applying regression analysis approach

– Demonstrated the ability to predict defects for system testing by using metrics in requirement, design and coding phase

• Future works:– To predict non-functional defects such as performance, security and usability

defects– To predict defects based on severity of defects i.e. critical, major and minor

defects– To incorporate more factors in building similar model such as function point,

programming languages, and number of classes– To develop software tool that could dynamically generate the latest prediction

equation in real time and assist in prediction activity

THANK YOU