1

Fault-Based Analysis: Improving IV&V Through Requirements Risk

Reduction '02

Jane HayesRama BireddyD.N. AmericanSAIC

Department of Computer ScienceUniversity of Kentucky

2

Outline

Research Objective Research Approach Progress to Date Current Plans Future Work

3

Research ObjectiveTo improve how we focus resources for IV&V of Critical/Catastrophic High-Risk (CCHR) software functions,

we use a fault-based analysis method comprised of:

• a requirements fault taxonomy (Phase I)• a method for extending taxonomies (Phase I)• a taxonomy of IV&V techniques (Phase I and II)• what faults the techniques can detect (Phase II+)• a fault-based risk assessment per Class and project (Phase I+)• a cost-benefit analysis of technique effectiveness (validated) (Phase II+)

4

Research Approach (Phase I)

• Task 1 – Select a Known Fault Taxonomy

• Task 2, 3 – PMR 2,3 (Presentation and Milestone Meeting)

• Task 4 – Examine NASA-specific requirements faults

• Task 5 – Build a list of IV&V techniques

• Task 6 – Adopt or build a method for extending the taxonomy

• Task 7 –Implement the method to extend the fault taxonomy

• Task 8,9 - Year-end report and presentation (also PMR 4)

5

Progress to Date

• NUREG/CR-6316 basis of general taxonomy

• Literature survey (55 references) resulted in three additions to taxonomy

• Review of defect data for 3 NASA projects resulted in two additions and re-organization of taxonomy

6

General Taxonomy

Taxonomy figure

7

NASA Requirement Faults• Have proven difficult to obtain• Level of detail varies greatly• Have thus far received and examined

– IV&V “comments” on requirement problems for 3 projects

– Project fault reports related to requirements for 1 project

• Data very useful, resulted in several changes to fault taxonomy and taxonomy extension/tailoring processes

8

Task 6

• Process for extending fault taxonomy split into two parts: Process A and Process B

• Process A - activities to develop a Class-specific taxonomy

• Outputs of Process A are inputs to Process B

• Process B – activities to develop a project-specific taxonomy

9

Processes for Extending Fault Taxonomies

High level process

10

Class-Specific Taxonomy Process

Process

11

Project-Specific Taxonomy Process

Process

12

Current Plans

• Obtain requirement-related fault reports for additional NASA projects

• Perform Process A (Class-specific) for classes for which we have data

• Complete list of IV&V techniques• Continue sharing information and

soliciting feedback from ST-5 project• Prepare final report

13

Future Work (Phase II)• Build taxonomy of IV&V techniques• Survey literature and determine what

techniques have been shown to detect certain types of requirement faults

• Gather expert opinion to fill in gaps of the technique-to-fault matrix

• Design experiments to validate some of the technique-to-fault mappings

• Provide resulting information in an Advanced Risk Reduction Tool (ARRT) friendly format

14

Backup

15

How does this differ from ODC?

• ODC uses a fixed set of trigger and defect types

• Our emphasis is on building tailored taxonomies

• We use historical information about Classes of related projects

• ODC classification strives to be independent of the specifics of a product or organization

16

How does this differ from ODC? (cont’d)

• ODC defect types don’t map well to requirements (function, I/f, checking, assignment, algorithm, etc.)

• We integrate risk analysis in our taxonomy building process

• Our long-term goals include validated cost-benefit information for fault-technique pairs