A Case History Analysis of Software Error Cause-Effect

830 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 17, NO. 8, AUGUST 1991

A Case History Analysis of Software ErrorCause-Effect Relationships

Takeshi Nakajo and Hitoshi Kume

Abstract- Software errors have been studied from variousperspectives; however, most investigations have been limited toan individual section or a partial path of the cause-effect relation-ships of these errors. The bresent study analyzes approximately700 errors in 4 commercial measuring-control software prod-ucts, and then identifies the cause-effect relationships of errorsoccurring during software development. The analysis methodused was: (i) defining appropriate observation points along thepath leading from cause to effect of a software error, followedby gathering the corresponding data by analyzing each errorusing Fault Tree Analysis, and (ii) categorizing each observationpoint’s data, and then summarizing the relationships between twoadjoining points using a cross-indexing table. This paper presentsfour major cause-effect relationships and discusses the effects ofthe Structured Analysis and Structured Design methods on theserelationships.

Index Terms- Software development, cause-effect relation-ships, case-based error analysis, program faults, human errors,process flaws, structured analysis and structured design, FaultTree Analysis.

I . INTRODUCTION

M any problems due to human error occur during softwaredevelopment, yet effectively eliminating root causes

from design processes to prevent their recurrence is difficultunless the types of human errors and their cause-effect rela-tionships are identified [l]. This identification is especially im-portant for long-term improvement of software-developmenttechniques.

Although there have been many quality investigations ofhuman errors in software development (e.g., [2]-[5]), mosthave been limited to examining an individual section or apartial path of the cause-effect relationships of software errors,with no previous report systematically investigating theserelationships.

This paper examines empirical errors in four commercial,measuring-equipment, control software packages, and thenidentifies the cause-effect relationships of these errors. Sec-tion II describes a method used to analyze the cause-effectrelationships based on software-error case data. Applying thiscase-based method, Section III shows that four major error

Manuscript received January 2, 1990; revised March 22, 1991. Recom-mended by M. S. Deutsch.

T. Nakajo was with the Faculty of Engineering, University of Tokyo,Bunkyo-ku, Tokyo 113, Japan. He is now with the Department of Industrialand System Engineering, Chuo University, 1-13-27 Kasuga, Bunkyo-ku,Tokyo 112, Japan.

H. Kume is with the Faculty of Engineering, University of Tokyo, Bunkyo-ku, Tokyo 113, Japan.

IEEE Log Number 9101143.

occurrence mechanisms exist, whereas the effects of StructuredAnalysis and Structured Design methods [6], [7] on thesemechanisms are discussed in Section IV. Finally, Section Vcompares the presented analysis method with the commonempirical methods used to identify software-error cause-effectrelationships.

II. IDENTIFICATION OF CAUSE-EFFECT

RELATIONSHIPS BASED ON ERROR CASE DATA

One method of identifying the cause-effect relationshipsof software errors is to analyze error case data. Fault TreeAnalysis (FTA) accomplishes this by using a logic tree diagramto present each error’s cause-effect relationships, and developsa countermeasure for each error [S]. Although this method canhandle the complex structure of cause-effect relationships, itis difficult for FTA to extract the relationships common toall errors under investigation, because it analyzes each errorindividually.

The presented analysis method basically follows FTA, yetfacilitates the extraction of the cause-effect relationships in-herent in software errors by restricting the logic tree diagram’sstructure using predefined observation points, and presentingthe relationships between two observation points using cross-indexing tables, i.e.,

1) Define observation points along an error’s logical pathfrom cause to effect

2) Gather observation point data by analyzing each errorusing FTA

3) Categorize each observation point’s data4) Identify the relationships between each two adjoining

observation points using a cross-indexing table5) Summarize the major cause-effect relationships based

on a series of the cross-indexing tables.The key elements in the above analysis method are: 1)

observation points, and 2) categorization of each observationpoint’s data.

Fig. 1 shows various types of error information along thepath from cause to effect. The following considerations mustbe incorporated when choosing the observation points:

1) A human error occurring during software development isfirst detected as “system failure”; however, the “systemfailures” vary widely. Tracing back to the source ofa human error reveals that “inappropriate work systemmanagement” was ultimately responsible for the error,although the details surrounding this state vary and arestrongly dependent on the organization’s characteristics.

00985589/91/0800-0830$01.00 0 1991 IEEE

N.4KAJ0 .~ND KuME: ~ALY~IS OF SOFIVARE ERROR CAUSE-EFFECT RELATIONSHIPS 831

Work Systen Flaw

\naPprOPrlate :+ ~ana~eaent i lndivldual F l a w

of Work System ‘\ I 1

System FailureProgram FaultHuman Error

Work SystemFlaw

InappropriateManagement ofWork System

System behavior mismatched with certain system operation specificationsIncorrect program codes defining software behaviorUnintended deviations from work standards or targets caused by the carelessnessof designers or programers, such as forgetfulness, mistakes OP misunderstandingInherent characteristics of methods, workers and environments affectinghuman BTPOI‘ occurrence and deviation. Classified into three types:Process Flaws, Individual Flaws and Environmental FlawInappropriate system managing work system elements, such as methods,workers and environments

Fig. 1. Causwzffect process of software errors

It is therefore important to choose observation pointswhere the most common information can be gathered;i.e., somewhere along the cause-to-result process ratherthan at either the original cause or final result points

2) The final purpose of identifying cause+effect relation-ships is to eliminate the root causes that produce “systemfailure”; thus it is important to choose an observationpoint which indicates the causes to be eliminated. Forexample, to improve not only individual product qual-ity but also product development techniques, “programfaults” and “human errors” as well as “work systemflaws” must be included in the set of observation points

3) In the data analysis case using FTA, the effects and theircauses have a one-to-many correspondence; therefore itis important to focus on a specific part of the causes.For instance, to improve software design techniques,“process flaws” must be focused among three types of“work system flaws” which mutually cause a humanerror.

Another key issue in this analysis method is the cate-gorization of each observation point’s data. The followingconsiderations must be accordingly included:

1) There are two data classification methods: one is basedon predefined criteria, and the other on data similarity.The latter method is called “clustering” and is effectiveeven when there is no theoretical model of the processwhich produces the data, with the Affinity Diagrammethod [9] being a non-numerical data-clusteringmethod

2) When a group of results has theoretically no correlationwith a group of causes, it is appropriate to independentlyclassify the causes corresponding to each group ofresults.

III. CAUSE-EFFECT RELATIONSHIPS IN

MEASURING EQUIPMENT-CONTROL SOFTWARE

The method described in Section II was applied to 670 soft-

ware errors occurring during the software development phaseof 4 commercial measuring equipment control packages. Thisanalysis identified the cause-effect relationships of softwareerrors in order to improve the developmental techniques ofhigh-reliability software.

Two of the four software packages investigated were sub-systems of an IC measuring system, with one being a setof “intrinsic routines” for users’ measurement programs, andthe other an application program using these routines. Theother two were firmware products which control measuringinstruments for electronic components. The four products,respectively referred to as A, B, C, and D, have the followingcommon characteristics:

1) Significantly hardware-specific, since they are designedfor initializing, setting, and measuring hardware

2) Consist of a set of unit functions that can be combinedto produce various required functions

3) Numerous global variables for memorizing hardwareequipment status are required.

The errors used in the analysis were all found during orafter the module testing phase.

The three following observation points were selected:1) Program fault2) Human error3) Process flaw.

The specifics of each error corresponding to these observationpoints were collected through source code investigation, de-sign documents and their change histories, and an interviewwith the designer or programmer who made the error. TheAffinity Diagram method was selected for observation pointdata classification, with Table I showing an example of datacollected.

A. Types of Program Faults

Fig. 2 shows the results of “program fault” data classifica-

832 IEEE TRANSAaIONS ON SOFIWARE ENGINEERING, VOL. 17, NO. 8, AUGUST 1991

TABLE IAN EXAMPLE OF ERROR CASE DATA

Observation Point Data CollectedProgram Fault *Reading data from a file in the wrong orderHuman Error *Misunderstanding the sequence of the data stored

in the file - 1Process Flaw *The global variable structure in which the data is

stored is different from that of the file-The programmer could not easily refer to theinformation about the file structure when he wascoding the program 1

Prog-DramFaults

+ II) Module Interface Faults (289/56.9)

;2.7%

P r o d u c t A Product B( a ) S y s t e m p r o d u c t s

i Internal inconsistencies in a modulef Logic Faults: Logical inconsistency (38/7.5)

Ex) A logical equation that is always true‘- Programming Faults: Program rwle violation (12/2.4)

Ex) Reference to undefined local variables 01‘ labels

Mismatching in the data transference or control1 between a module and its environment, i.e., other

modules, global variables, data files and hardware(Not caused by Fault I)Name Faults: Name mismatch (53/10.4)Ex) Referring to a nonexistent global variable

- Structural Faults: structure, type orconfiguration mismatch (99/19.5)

P r o d u c t C P r o d u c t D

(b) Firmware productsEx) Mismatch between a subprogram’s argument type and

its calling statement’s corresponding variable- Value Faults: Value meaning or possible

variable range mismatch (99/19.5)Ex) Substituting the wrong value for an argument that

switches a sub-module’s function- Procedural Faults: Data transfer OP controlprocedures mismatch (38/1.5) Note: (1) Goodness of fit test [lo] using a 5% significance lwel did not

Ex) Failure to set hardware to a receiving state reject the hypothesis that the proportion of internal errors 1s

before transferring dataconstant for all products.

(2) Goodness of fit test using a 1% significance level rejected thehypothesis that program fault distribution in Products A and Bis the same as that in Products C and D.III) Module Function Faults (169/33.3)

Incorrect operations of a module resulting inunsatisfied upper functionality requirements

k Operating Faults: Operation omission orunnecessary operations (59/11.6)Ex) Failure to set an item to hardware that will be

used in the following commands or modules.Condition Faults: Incorrect operationconditions (79/15.6)Ex) Use of an incorrect limit value in judging

whether or not to reset hardwareL- Behavioral Faults: Incorrect behaviors (31/6.1)

Ex) Displaying terms not conforming to requirements

* fiumbers in parentheses show the totals and ratios of errorsfalling under corresponding categories, obtained by classifying508 program faults in product A and B

Fig. 2. Types of program faults.

tion, where program fault is the observation point nearest tothe end of the cause-effect chain.

Fig. 3 and the numbers in parentheses in Fig. 2 show theproportions and total number of program faults correspondingto each of the categories. From these figures the following canbe seen:

1) Interface faults and module function faults togetherconstitute more than 90% of the total for all products

2) In the system software programs, interface faults oc-curred more frequently than module function faults,whereas the latter occurred more frequently in firmwareproducts than the former

Fig. 3. Proportions of three major types of program faults.

3) In the system software programs, the name, structural,and value faults were the most common types of inter-face faults, with the operating and condition faults beingthe most common types of module function faults.

B. Cause-Eflect Relationships Between HumanError and Program Fault

This section focuses on the two major program fault typesidentified in Section III-A: interface fault and function fault,and describes human-error categorization results, followed byexamining the relationships between human error and programfaults.

An interface fault-mismatching between various softwarecomponents-is closely related to technical information com-munication between the various component’s developmentteams. On the other hand, a module function fault is closelyrelated to designing software which satisfies the users’ needs.Because it is well known that differences in these types ofworks greatly affect the types of human errors which occurduring the work process, 289 human errors causing interfacefaults and 169 human errors causing function faults in systemsoftware products A and B were independently classified. Fig.4 shows the human-error categorization results.

NAKAJO AND KUME: ANALYSIS OF SOFTWARE ERROR CAUSE-EFFECT RELATIONSHIPS 833

““UIUIErrorsC*“S-

L-l

ingInter-faceF*“ItS

HumanE~IYXSCT*“%-

c

ingF”“VtionF*“ltS

Communication Errors within a Development Team

- Misunderstandings of Software Interface Specifications:Omissions and misunderstandings in communicating andrecognizing interface specifications related to thesoftware that the team is responsible for developing

Ex.1) When changing the module name, forgettingto change the relevant calling statements

Ex.2) Misunderstanding the data sequence stored in .?A file

-I Communication Errors between a Development Team and Others

Misunderstandings of Hardware Interface Specifications:Omissions and misunderstandings in communicating andrecognizing interface specifications related to thehardware that the developed saftware controls

Ex.1) Forgetting to communicate a hardwareconfiguration change to the software designers

Ex.2) Misunderstanding hardware data transfer procedures- Misunderstandings of Other Software’s Interface

Specifications:Omissions and misunderstandings in communicating andrecognizing specifications related ta other softwarethat the software being developed exchanges data with

Ex.1) Misunderstanding the calling sequence ofa module developed by another team

(a) Human erw~s causing interface faults

1Errors in Recognizing Requirements

- Misunderstandings of System Functions:Omissions and misunderstandings concerning the systemfunctions required as system external specifications

IEx.1) Forgetting to consider the order in which

measurements shall be displayedEx.2) Overlooking possible misoperations by user

Errors in Deplaying RequirementsJ

- Misunderstandings of Module Functions:Omissions and misunderstandings concerning the modulefunctions and their relationships to system functions

Ex.1) Overlooking relevant environmental conditionswhen calling a module

Ex.2) Forgetting to save a calculated value for globalvariables in a module designed to set hardware

- Misunderstandings of Program Segment Functions:Omissions and misunderstanding concerning the programsegment functions and their relations to module functions

Ex.1) Misunderstanding an algorithm for sart~ng data

(b) Human 811110115 causing functian faults

Fig. 4. Categories of human errors.

Table II summarizes the relationships between human errorand program faults using the categorizations in Figs. 2 and 4.From Table II the following can be seen:

1) More than 50% of interface faults were due to a mis-understanding of software interface specifications; i.e.,communication errors within a development team. Onthe other hand, in the case of communication errorsoccurring between a development team and others, mis-understanding of hardware interface specifications oc-curred more frequently

2) Misunderstanding of software interface specificationscauses primarily name, structural, or value faults, whilemisunderstanding of hardware interface specificationscauses structural, value, or procedural faults

3) Approximately 45% of the function faults were causedby a misunderstanding of system functions (i.e., anerror in recognizing requirements), with about 40% offunction faults being caused by a misunderstanding ofmodule functions

4) System function misunderstandings cause mainly con-dition and behavioral faults, while module functionmisunderstandings cause operating faults more than con-dition faults.

ProgramFaults\HumanErrors

NameFaults

StructuralFaultsValueFaults

ProceduralFaultsTotal

ProgramFaults\HumanErrors

OperatingFaults

ConditionFaults

BehavioralFaultsTotal

Note:

TABLE IIPROGRAM FAULTAND HUMAN ERRORS

Team 1Misunder- 1 Misunder- 1 Misunder-

I I I63 30 6 99

47 42 10 99I I I

4 28 6 38I I I

156(54.0) 106(36.7) 1 27( 9 . 3 ) 289133(46.0)

I) The Relationships Causing Function FaultsRequirement I Requirement Deployment IRecognition I

Total

Misunder- I Misunder- ( Misunder-

48 20 11 79

18 4 9 31I I

76(45.0) 68(40.2) 1 25(15.8) 16993(55.0)

(1) These tables resulted from an error analysis on Products A and B.(2) Goodness of fit test [lo] using a 1% significance level rejected thehypothesis that the proportions of program faults due tomisunderstandings of software interface specifications are the same asdue to misunderstandings of hardware interface specifications, and thehypothesis that the proportions of program faults due tomisunderstandings of system functions are the same as due tomisunderstandings of module functions.

C. Cause-Effect Relationships Between ProcessFlaw and Human Error

This section focuses on the following four major human-error types by describing the process flaw categorizationresults, and then examines the relationships between processflaws and human error.

Error I: Human errors causing interface faults:1) Misunderstanding of software interface specifications2) Misunderstanding of hardware interface specifications.Error II: Human errors causing function faults:1) Misunderstanding of system functions2) Misunderstanding of module functions.Process flaws causing 406 human errors in Products A and

B (falling under the 4 above categories) were classified based


!rrors\Process FlawsHLJI

Misunderstanding ofSoftware Interface

Specifications

MAIHuma

v ERRORS ANO PROCESS FLAWS CAUSING INTERFACEDesign-Principle Flaws

Inappropriate interface definitions prone to bemisunderstood and misdeveloped1) Definitions inconsistent and distributed (113)*

Ex) Parameter structure varies widely formodules when each module is individuallydefined2) Complicated correspondence betweendefinitions (53)

ErrorsCausingInterface

FaultsMisunderstanding ofHardware Interface

Specifications

Ex) Variable structure is different from that ofthe data to be stored3) Insufficient discrimination between defineditems (37)

Ex) Global variables or modules have similarnames4) Ambiguous labels defining items (59) Ex)Flag variable name does not present the systemstate that it implies

* Numbers in parentheses show the total process flaws of Products A and B in each category. Adesign-management flaw are counted for each human error.

Design Principle Flaws

rl

Flaws related to fundamental principles thatPI%- designers OP programers must follow in order tocess define proper and understandable interfaceFlWS or functional structures.

Ex) The global variable structure in which the datais stored is different from that of the file

3) The major design-management flaws that cause the mis-understanding of hardware interface specifications wereunclear documentation of hardware accessing methods

4) Dependent/unsymmetrical system unit or module func-tion definitions were the design-principle flaws thatcause the misunderstanding of system and module func-tions

Design Management Flaws

Flaws related to the methods and proceduresfacilitating design management, i.e., how todocument and communicate information on the inter-faces and functional structures so that designersand programers can utilize them properlyand evaluate their correctness.Ex) The programmer could not easily refer to the

information about the file structure whenhe was coding the program

5) The design-management flaws that cause the misunder-standing of system functions were unsystematic require-ment documentation and insufficient documentation onhow to combine system unit functions to satisfy theserequirements

6) The design-management flaws that cause the misunder-standing of module functions were inappropriate doc-umentation of the relationships between system andmodule functions.

Fig. 5. Categories of process flaws.

on their similarity. Fig. 5 shows the process flaw categorizationresults.

As a result, it is concluded that the error-occurrence mecha-nisms in the development of these products consist of the fourmajor paths shown in Fig. 6.

Tables III and IV summarize the relationships betweenprocess flaws and human errors using the categorizations inFigs. 4 and 5. From these tables the following can be seen:

1) Inconsistent and distributed interface definitions werethe major design-principle flaws that cause the misun-derstanding of software and hardware interface specifi-cations

IV. E FFECTS O F STRUCTURED ANALYSIS AND STRUCTURED

D E S I G N M E T H O D S O N E R R O R -OC C U R R E N C E M E C HA N I S M S

2) The design-management flaws that cause the misun-derstanding of software interface specifications wereinappropriate documentation for module calling methodsand/or file and global variable definitions

This section describes the effects of Structured Analysis(SA) and Structured Design (SD) methods [6], [7] on the previ-ously identified four error-occurrence mechanisms. Althoughthe SA/SD methods are generally used in designing simplystructured software, when used as design-management toolsthey can help suppress these error mechanisms.

A measuring-equipment control system product, similar toProducts A and B, was chosen to evaluate the SA/SD methods’effects on error-occurrence mechanisms. For comparison, six

TABLE IIILILTS

Design-Management FlawsLack of methods for recording and referringsoftware interface definitions1) Inappropriate communication of module

calling information (80)Ex) The necessary operations before calling

modules is not clearly documented2) Inappropriate of communication of globalvariable or file access information (76)

Ex) Global variable value meanings are notclearly documentedLack of communication methods betweensoftware engineers and hardware engineers1) Inappropriate communication of hardwarephysical configurations (24)

Ex) The number of hardware components andthe connection between them are shown only bycircuit diagrams2) Inappropriate communication of hardwareaccess information (82) Ex) Methods ofreading or writing hardware memories and eachmemory’s meanings are unsystematicallydocumented in hardware specifications

.a1 design-principle flaw and a vital


TABLE IVN ERRORSAND PROCESS FLAWS CAUSING FUNCTION .ULTS

Design-Principle FlawsComolicated corresoondence betweenrequirements and the means of realizing them1) Functions defined unsymmetrically (91)*

Ex) Some commands do not make sense andare unexecutable under certain conditions2) Functions defined dependently (53) Ex)Functions related to user interface and equipmentcontrol are included in a module because modulestructures are designed so as to satisfyperformance requirements

Design Management FlawsLack of systematic methods to describeexternal system functions1) Inappropriate documentation of users’requirements and system unit functions (71)

Ex) The conditions where each command canbe executed are documented using naturallanguage2) Inappropriate documentation of relationshipsbetween requirements and system unit functions(5)

HI I,irrors\Process Flaws

__-.Hum:

Misunderstanding of SystemFunctions

ErrorsCausingFunction

Faults

Misunderstanding of ModuleFunctions

* Numbers in the parentheses show the total process flaws of Products A and B in each category. A

Ex) How to combine commands for functionrealization is undocumentedLack of methods describing module functionsand the relationships between module andexternal system functions1) Inappropriate documentation of modulefunctions (15)

Ex) The conditions where each module can beexecuted are not completely documented2) Inappropriate documentation of relationshipsbetween system unit and module functions (53)

Ex) The relationships between commands andmodules are incompletely documentedal design-principle flaw and a vital

design-management flaw are counted for each human error

TABLE VCHARACTERISTICS AND DEVELOPMENT CONDITIONS OF THE COMPARED

SUBSYSTEMSCharac t e r i s t i c 1 Subsystems Subsystems

Process -1Development Condi- Developed using Developed without

tion\Subsvstems %/SD Methods usine &A/SDMithods

Number of 6(including 5 intrinsic 6(including 4 intrinsicSubsystems* subsystems) subsystems)

The Size of Codes 24.9 KNCSS** 27.0 KNCSSThe Size of Reused 1.7 KNCSS 12.0 KNCSS

Flaws +Lack of methods forrecording and referring

+Lack of communicationmethods between

Codes***Program Language

Number ofProgramers

Experience ofProgramers

System SpecificationDevelooment

7z!s%$rl

C C5 persons 6 persons

34 years 34 years

No difference

[Module interface faults 1(Structural, value andprocedural faults)

(b) Mechanism 2(a) Mechanism 1

and the relationshipsbetween module andexternal systemfunctions(Relationships betweenupper functions

Processes* Subsystems of a single system were selected and compared.* * KNCSS = 1000 Noncommented source statements.* * * All codes reused were reviewed and modified if necessary.

Complicated correspond-ence between reauire-merits and the means of

PrOCeSS realizing themFlaws +

Lack of systematicmethod to describeexternal system I

uct were developed without using SA/SD methods (Controlgroup). Table V shows the characteristics and developmentconditions for both groups. All errors discovered during andafter the module testing phase were analyzed and compared.

Fig. 7 shows the occurrence rates and proportions of thethree program fault types for both groups, with Table VI givingthe relationships between human error and program fault forthe SA/SD group. This information, combined with that ofFig. 3 and Table II, yielded the following:

Misunderstandings ofmodule functions

Human Misunderstandings ofErrOr& system functions

Program Module function faultsFaults (Condition)

Module function faults(Operation, Condition)

(d) Mechantsm 1(cl Mechanism 3

Fig. 6. Error-occurrence mechanisms of measuring equipment control soft- 1) The internal fault proportion was constant for the fourware. test groups; i.e., M/SD, Control, and Products A and B

2) The internal fault-occurrence rate in the M/SD groupproduct subsystems were developed using M/SD methods(SA/SD group), and six other subsystems of the same prod-

was the same as in the Control group3) The interface fault proportion in the Control group was


Program Faults\HumanErrors

TABLE VIPROGRAM FAULTS AND HUMAN ERRORS IN THE SUBSYSTEMS DEVELOPED USING S&SD METHODS

(a) The Relationships Causing Interface FaultsCommunication within a Communication between a Development Team and Others

Development teamMisunderstanding of Misunderstanding of Misunderstanding of OtherSoftware Interface Hardware Interface Software’s Interface

Specifications Specifications Specifications

Total

Name Faults 1 0 0 1Structural Faults I 2 1 10

Value Faults 1 2 0 3Procedural Faults 1 0 0 1

Total

Program Faults\HumanErrors

I4(26.7) l( 6.7)

lO(66.7) 5(33.3)(b) The Relationships Causing Function Faults

Requirement Recognition Requirement DeploymentMisunderstanding of System Misunderstanding of Misunderstanding of

Functions Module Functions Program SegmentFunctions

15

Total

Operating FaultsCondition FaultsBehavioral Faults

Total

3 10 0 134 I 1 124 0 0 4

17(58.6) l( 3.5)ll(37.9) UJ(62.1) 29

4)

5)

6)

the same as in Products A and B, whereas its proportionin the SA/SD group was smaller than in Products Aand BThe interface fault-occurrence rate in the SA/SD groupwas about a half of the Control group’sThe function fault-occurrence rate in the &A/SD groupwas approximately 35% smaller than that in the ControlgroupAlthough the amount of data was not sufficient todetermine the relationships between human error andprogram faults, for the SA/SD group the proportion ofstructural faults due to software-interface specificationmisunderstandings increased, and the proportion of con-dition faults due to system-function misunderstandingsdecreased.

It is concluded that in this experiment the M/SD methodssuppressed the four error-occurrence mechanisms, and alsothat the effects on 1 and 2 were greater than those on 3 and 4.

V. D I S C U S S I O N

The metric-based method is a common method of em-pirically analyzing software error cause-effect relationships[5], [ll], [12]. This is an application of the method usedto analyze hardware cause-effect relationships, and primarilyuses correlation coefficients between two metrics measuringresults and their causes. The method’s general outline is:

1) Divide the products under investigation into appropriate-sized units for analyzing causeeeffect relationships; i.e.,systems, subsystems, or modules

2) Define the metrics which measure the unit characteristicsand the unit-producing processes and then gather data

3) Calculate the correlation coefficients between the unit’smetrics and the processes’ metrics and then evaluate

(a) Croup developed us,ng SR/SD methods

“‘% (‘) 4 4 . 3 % (27) 4.0

17.5%

rn 3 . 0z5\ 2 . n2

j 1.0

0 .0Tota, Inter- Inter- Func-

“.3I face *lo”

(b) Gro”p developed s~thout “sing SA/SD methods

(1) Goodness of fit test [lo] usmg a 5X sigmficance level did notreject the hypothesm that internal fault pmport*on is constant forfour test groups. ,.e., %/SD, Control, and Products A and B, thehypothesis that the internal fault occui-rmce rate in SA/SD grout isthe same as in Control group, and the hypothesis that Interfacefault proportum is constant for Control group and Products A and B.

(2) Goodness of fit test usmg a 1% sigmflcance level reJected thehvoothesls that interface fault mm~ort~on 1s constant for S.A/SDgroup and Products A and B, and ihe hypothesis that the interfacefault occurrence rate based on the code size exrludmg reused code,n SA/SD group 1s the same as I” Control group. Even mcludmg thereused code, the test stat ist ic value was near to the 5% signlflCanCPpaint.

(3) The statistic value of goodness of fit test for the hypothesis thatt h e functmn fault occurrenre r a t e ,n %/SD gro”~ IS t h e same a s InControl group was nrar to the 5X significanw pant when excludingrmsed code.

Fig. 7. Effects of W/SD methods on program faults.

The key elements of this metric-based method are the sizeof the units used in Sten 1 and the metric tvpes defined inthem.


Step 2, especially the process characteristic measurements.Method effectiveness is highly dependent upon these ele-ments (e.g., large units decrease the amount of data forcalculating correlation coefficients and do not isolate theeffects of various causes, whereas small units increase datavariance and the respective correlation coefficient variance).Usually trial-and-error techniques determine the appropriateunit sizes, and similarly the process characteristic metrics areprimarily selected based on either specific engineering modelsor experience.

Another approach in determining the cause-effect rela-tionships of software errors is to individually analyze eacherror. FTA is one such method, where unit sizes and processcharacteristic metrics are not required to be determined inadvance. FTA is flexible enough to handle each error’s com-plex cause-effect relationships; however, this flexibility makesit difficult for FTA to extract the relationships common tovarious errors. Additionally, analyzing more than one errorat a time in order to obtain a fault tree diagram presentingcommon cause-effect relationships often results in a subjectiveanalysis unrelated to individual errors. Another FTA methodweakness is the number of man-hours necessary to collectdata. In the metric-based method, data can be automaticallycollected once the metrics are established. Contrastingly, theFTA method is difficult and inappropriate to limit the datarange in advance, and as a result requires more data-collectiontime. This degree of difference significantly depends on thedevelopment environment, especially on the error-trackingsystem.

As in FTA, the presented case-based analysis method doesnot require the unit sizes and process characteristic metricsto be determined in advance and can be applied withoutspecific engineering models of the cause-effect relationships.The case-based method overcomes the FTA weakness by usingboth predefined observation points and a phased approachwhich individually tracks each error’s case data to extract thecommon cause-effect relationships inherent in the data. Onthe other hand, this case-based analysis method still requiresa considerable amount of man-hours, because it basicallyfalls under the category of the FTA method. The man-hoursnecessary for data collection are more critical than thosenecessary for data analysis; however, the data collection timecan be fairly reduced once appropriate observation points areestablished.

VI. CONCLUSIONS

The paper focused on measuring-equipment control soft-ware and identified cause-effect relationships of softwareerrors. Four major error-occurrence mechanisms were iden-tified-two are related to hardware and software interfacespecification misunderstandings, and the other two are relatedto system and module function misunderstandings. The effectsof the Structured Analysis and Structured Design methods onsoftware errors were also evaluated using these mechanisms.The structured methods could suppress the error-occurrencemechanisms, being more effective on hardware and softwareinterface mechanisms than on system and module function

mechanisms. The presented case-based cause-effect relation-ship analysis method can be expected to supplement themetric-based and FTA methods. These conclusions were drawnfrom specific software packages and are thus limited to them.Given the similarities of software development procedures,however, the presented results can be generalized and appliedto the development of other software products.

A C K N O W L E D G M E N T

The authors wish to thank the following employees ofYokogawa-Hewlett-Packard Co. Ltd.: M. Mori (InstrumentDivision Manager) for his beneficial advice, and K. Sasabuchi,T. Akiyama, and I. Azuma (Product Assurance Section), andF. Turuda, M. Tada (Research and Development Sections),and many other software engineers for their help with thecollection and analysis of software-error information.

R E F E R E N C E S

Ill

PI

I31

(41

[51

I61

[71

[*I

[91

[lOI

[Ill

[=I

W. S. Humphrey, Managing the Software Process. Reading, MA:Addison-Wesley, 1988, pp. 363-388.A. Enders, “An analysis of errors and their causes in system programs,”IEEE Trans. Software Eng., vol. SE-l, pp. 14&149, June 1975.N. F. Schneidewind and H. M. Hoffman, “An experiment in softwareerror data collection and analysis,” IEEE Trans. Software Eng., vol.SE-5, pp. 276286, May 1979.E. A. Young, “Human errors in programming,” In?. J. Man-MachineStudies, vol. 6, no. 3, pp. 361-376, May 1974.V. R. Basili and T. Perricone, “Software errors and complexity: anempirical investigation,” Commun. ACM, vol. 27, no. 1, pp. 42-52, Jan.1984.T. DeMarco, StructuredAnalysis and System Specification. EnalewoodCliffs, NJ: Prentice-Hall, 1579. . . 1C. Gane and T. Sarson, Structured System Analysis: Tools and Tech-niques. New York: Improved System Technologies, 1979.N. G. Leveson and R. R. Harvey, “Software fault tree analysis,” J. Syst.Sofhvare, vol. 3, no. 2, pp. 173-182, June 1983.S. Mizuno, Management for Quality Improvement: The 7New QC Tools.Cambridge, MA: Productivity Press, 1988, pp. 115-142.I. Guttman and S. S. Wilks, Introductory Engineering Statistics. NewYork: Wiley, 1965, ch. 12.V. R. Basili and D. Rombach, “Tailoring the software process to projectgoals and environments,”m Proc. 9th Int. Co@ Software Eng., Mar.1987, pp. 345-357.V. R. Basili, R. W. Selby, and D. H. Hutchens, “Experimentationin software engineering,” IEEE Trans. Software Eng., vol. SE-12, pp.733-743, July 1986.

Takeshi Nakajo received the B.S., M.S., and Ph.D.degrees in engineering from the University of Tokyo,Japan, in 1979, 1981, and 1986, respectively.

He was a Research Assistant at the University ofTokyo from 1987 to 1990, and has been a Lecturer atChuo University since 1990, where his special fieldis in quality improvement, especially the preventionof human errors, with his current research interestsbeing in software-error analysis and design andtesting methods. Dr. Nakajo is a member of IEEEComputer Society.


Hitoshi Kume received the B.S., MS., and Ph.D.degrees in engineering from the University of Tokyo,Japan, in 1960, 1962, and 1965, respectively.

He was an Associate Professor at the Universityof Tokyo from 1974 to 1980, and has been aProfessor since 1980, where his special field is inquality control. Since 1960 he has been engagedin consulting work for the improvement of qualitysystems in Japan’s various manufacturing industries.

Dr. Kume is a member of the American Soci-ety for Quality Control and a Japan Delegate to

iagement and Quality Assurance) of the Internationaltdardization.

Documents

A Case History Analysis of Software Error Cause-Effect