1

IV&V Facility

NASA Software IV&V Facility

Metrics Data Program

June 5, 2003

Mike Chapman chapman@ivv.nasa.gov 304-367-8341Pat Callis Patrick.E.Callis@nasa.gov 304-367-8309Ken McGill Kenneth.G.McGill@nasa.gov 304-367-8300

2

IV&V FacilityThe Metrics Data Program Goal

Establish a centralized repository that provides consistent, Establish a centralized repository that provides consistent, fully-involved fully-involved software product datasoftware product data across multiple domains: across multiple domains:

•To improve the effectiveness of software assuranceTo improve the effectiveness of software assurance•To improve the effectiveness of software researchTo improve the effectiveness of software research•To improve the ability of projects to predict software errors To improve the ability of projects to predict software errors

early in the lifecycleearly in the lifecycle

3

IV&V Facility

Data, Data Everywhere: The Data Drought

There is very little fully-involved software product data…There is very little fully-involved software product data…

Error data associated to smallest functional unit Error data associated to smallest functional unit

Requirements traced through the design to the smallest Requirements traced through the design to the smallest functional unitfunctional unit

……available to those who need it.available to those who need it.

4

IV&V FacilitySARP PI Survey

To what degree has a lack of software defect data from actual NASA projects impacted your SARP funded research?

-  Greatly (9).  Lack of software defect data from actual NASA project has seriously hampered my research

- Moderately (8).  It would be nice to have more (or some) real project data, but I have found non-NASA data sources or other workarounds.

-  Not significantly (5).  My project has not been impacted because I either have all the data I need or I don't need software defect data

Note: The totals are in parentheses.

5

IV&V Facility

The Lack of Data from a Project Perspective

Error data associated to smallest functional unitError data associated to smallest functional unit•There is little value in this activity for the projectThere is little value in this activity for the project

Requirements traced through the design to the smallest Requirements traced through the design to the smallest functional unitfunctional unit

•Projects only need to trace requirements to the executable Projects only need to trace requirements to the executable (CSC level or above)(CSC level or above)

Fully-involved data made available to the research communityFully-involved data made available to the research community•Vulnerability of the programVulnerability of the program•Proprietary issues Proprietary issues

6

IV&V Facility

The Quest for the Holy Grail

Requirements

Design

Code(smallest functional unit)

Traceability

ProblemReports

dev, test, user, maintenance

Regardless of the development model the following is needed:

Association to the smallest functional unit.

(Many to one issue.)

7

IV&V Facility

Recruitment of Project Data

Existing repository data Existing repository data

Error Data Enhancement ProgramError Data Enhancement Program

Identifier Language LOC Domain Error Data

JM-1 C 315K Real-Time 8 years

KC C++ ~750K Data System 5 years

CM-1 C 20K Instrument 2 years

CM-4 C ~40K Flight N/A

8

IV&V Facility

Error Data Enhancement Program

The goal of the enhancement effort is to successfully recruit projects to work with the MDP to provide fully-involved software product data – the Holy Grail.

The MDP team will provide:Requirements analysis and traceability supportConfiguration Management Support (Error Tracking)Metrics generation and analysisDatabase and web supportMachine learning analysis

9

IV&V Facility

Project Error Data

MDP Web Site

ClearQuestInterface

Firewall

Stored Metrics and Error Data

ClearQuest

Project SecurityProject1………………….Project n

Project Metrics and Error Data

MDP Repository

Queries

Sanitized Data

Sanitized Data

Other Project Data

10

IV&V Facility

Agency benefits:•The improved ability to predict error early in the lifecycle•The improved ability to assess the quality of the software

The research community benefits:•Availability of quality error and metric data•Availability of a support team for data needs

Participating projects benefit:•Additional metrics analysis•Additional error analysis•Problem tracking tool•Other support such as requirements traceability

Benefits

11

IV&V Facility

Where to Find More Information

mdp.ivv.nasa.gov

12

IV&V Facility

Site Metrics

Web Site activity for 3 months:Web Site activity for 3 months:

596 hits596 hits46 accounts46 accounts146 logins146 logins85 downloads of data85 downloads of data

13

IV&V Facility

Special Data Requests

Five time stamps of KC-2 dataFive time stamps of KC-2 dataSanitized activity fields of JM-1 dataSanitized activity fields of JM-1 dataError Reports from CSCI of KCError Reports from CSCI of KC

Note: Five papers have been written from the repository data so far.

14

IV&V Facility

Current Data Request

CM-1 requirements data and associated errors - JPLCM-1 requirements data and associated errors - JPLCM-1 data including metrics – Dolores Wallace SATCCM-1 data including metrics – Dolores Wallace SATCKC semantic metrics generation – Letha Etzkorn UA – HuntsvilleKC semantic metrics generation – Letha Etzkorn UA – HuntsvilleJM-1 time stamp data (five sets) – Tim Menzies WVUJM-1 time stamp data (five sets) – Tim Menzies WVU

15

IV&V Facility

A Study

Metric Combination % of Errors % of Code Probability of Failure (%)

l >= 0.047015525 50% 34% 71%

loc >= 9.197AND l >= 0.066 30% 20% 37%

•v(G) – cyclomatic complexity – independent linear paths•ev(G) – essential complexity – unstructured constructs•e – programming effort – mental effort•l – (2/u1)*(u2/N2)-program level – the level at which a program can be understood

Metric Combination Probability of Module Error Probability of Failure (%)

% of Severity One Errors

v(g)>45.9 & e>=1340053 80% 0% 50% (8/16)

v(g)>45.9 & loc>= 150 73% 0% 33% (23/69)

16

IV&V Facility

Background Slides on Metrics

17

IV&V Facility

v(G): cyclomatic complexity = # path(ish)s = edges-nodes+1

m = # one entry/one exit sub-graphs

ev(G): essential complexity = v(G) – m

iv(G): design complexity (reflects complexity

of calls to other modules)

Halstead: programmers read code.

Too many “words” error

Mccabe: paths between “words”Twisted paths error

McCabe Metrics

18

IV&V Facility

e.g. 2+2+3N1 = 3N2 = 2µ1 = 2µ2 = 2µ1’ = 2(ish)µ2’ = #input parameters

µ1µ2

N1N2

µ = µ1+ µ2N = length = N1+N2V = volume = N*log2(µ)V’ = (2+ µ2’)*log2(2+ µ2’)

Could be found via simple tokenizers

Halstead Metrics

L = level = V’/VD = difficulty = 1/LL’ = 1/DE = effort = V/L’T = time = E/18

19

IV&V Facility

Operators

! % & * + , -

. / ; < > ? ^ |

~ = >= <= == != >>

<< += -= *= /= %= &=

^= |= >>= <<= && || ++

-- -> return size of enum struct delete

continue new break union      

if (…) … if (…) … else

switch (……) default:

case <label>: goto <label>

do … while (…) while (…) …do

for (… ; …; …) this->

[ ] { }

<function name>( ) ( ) in any other cases not covered

… ‘?’ … ‘ ‘: …  

20

IV&V Facility

Operands

Variables and identifiersConstants (numeric literal/string)Function names when used during calls

21

IV&V Facility

Module LOC ER ED* 6 mth 1 yr 2 yrCSC1/mod1 59 6 101.7 3 4 5

CSC2/mod1 190 6 31.58 2 4 5

CSC1/mod9 40 5 125 1 2 3

CSC7/mod27 47 3 63.83

CSC4/mod10 287 2 6.9 1 2 2

CSC6/mod51 482 10 20.8 1 4

*ED = ER/KLOC

Error Metrics

22

IV&V Facility

OO Metrics

Number of Children (NOC) – number of sub-classesNumber of Children (NOC) – number of sub-classesDepth – level of class in the class hierarchyDepth – level of class in the class hierarchyResponse for Class (RFC) – number of local methods plusResponse for Class (RFC) – number of local methods plusthe number of methods called by local methods (>100)the number of methods called by local methods (>100)Weighted Methods per Class (WMC) – sum of the complexitiesWeighted Methods per Class (WMC) – sum of the complexitiesof the methods (>100)of the methods (>100)Coupling Between Object Classes (CBO) – dependency on classesCoupling Between Object Classes (CBO) – dependency on classesoutside the class hierarchy (>5)outside the class hierarchy (>5)Lack of Cohesion of Methods (LOCM) – the use of local Lack of Cohesion of Methods (LOCM) – the use of local instance variable by local methods instance variable by local methods

23

IV&V Facility

ARM Metrics

Weak Phrases (adequate, be able to) – clauses that cause uncertaintyWeak Phrases (adequate, be able to) – clauses that cause uncertaintyIncomplete (TBD, TBR) – Words and phrases that indicate the Incomplete (TBD, TBR) – Words and phrases that indicate the spec may not be fully developedspec may not be fully developedOptions (can, optionally) – Words that give the developer latitudeOptions (can, optionally) – Words that give the developer latitudeImperatives (shall, may, will, should) – Words that are explicitImperatives (shall, may, will, should) – Words that are explicitContinuances (below, as follows, and) Extensive use of continuances Continuances (below, as follows, and) Extensive use of continuances can indicate complex requirements.can indicate complex requirements.Directives (for example, figure, table) – examples or illustrationsDirectives (for example, figure, table) – examples or illustrationsLines of text (size)Lines of text (size)Document structure (levels) Document structure (levels)

24

IV&V Facility

Program Background Slides

25

IV&V Facility

Problem Report Fields

Error Identifier: (Alpha-numeric) Headline: (text- short description)

Submitted-on: (Date yymmdd) Severity: (1 thru 5)

Status: (NVTCDARMB) System Mode: (operations versus test)

Request type: (Problem or enhancement) Problem-type: (requirements, design, source code, COTS, documentation, hardware,etc)

Problem Mode: (PR or Action Item)

Assigned-to: CCB Approval: (Date)

Impacts-csci: (high level design element) Impacts-csc: CSC

Impacts-class/file: class/file Impacts-method/function/module: Method/function/module

Impacts – Requirement Impacts – design element

Resolution: source code, COTS, documentation, not a bug, unreproducible

Problem: (text) Analysis: (text)

Resolution: (text) Closed-on: (Date)

26

IV&V Facility

Project Non-Specific - Universal

Error Identifier: (Alpha-numeric)*Submitted-on: (yymmdd)Severity: (1 thru 5)Status: (NVTCDARMB)Mode: (operations versus test)Request type: (Problem or enhancement)Problem-type: (requirements, design, source code, COTS, documentation,etc)Impacts-csci: (high level software design element)*Documents: (What documents are affected?)CCB Approval: (Date)Resolution: source code, COTS, documentation, not a bug, unreproducibleVerified-on: (Date)Closed-on: (Date)

* May need sanitized.

27

IV&V Facility

Project Non-specific - Expanded

Impact data: Costing Data:Impacts-csc: CSC*Impacts-class/file: class/file*Impacts-method/function/module: Method/function/module*Recommend-change: source code Process data:How-found: (e.g., Acceptance Test)When-found: (e.g., Acceptance Testing)Analysis-due: 020322Assigned-Eval-on: 020322Assigned-Implement-on: 020323Implement-due: 020325Fix-date: 020325Fixed-on: (date)In-Test-on: 020325Test-name: (Numeric id)*Test-system: (hardware)Verify-date:Merge-build-id:Deferred-on:Build-name: (Alpha-numeric identifier)Patch-rel-name: Patch-rel-date: (date)Automated history entries: (Alpha-numeric) * May need sanitized

Cost: (high, medium, low)Est-fix-hours: Est-fix-date: (date)Est-Num-SLOC: Rev-fix-time: Rev-fix-date: (date)SLOC-Type:SLOC-count: Fix-hours:

Miscellaneous Data:Operating-system:Priority: (High, Medium, Low)Enhancement: (Y or N)Workaround: YIteration: (Version bug identified in)

28

IV&V Facility

Project Specific - Universal

Headline: (text – short problem description)Problem: (text – expanded problem description)Analysis: (text)Resolution: (text)Closure: (text)Submitter-id: Assigned-to: Closer-id:

Question: Can Project Specific data be sanitized?

29

IV&V Facility

Project Specific - Universal

Headline: (text – short problem description)Problem: (text – expanded problem description)Analysis: (text)Resolution: (text)Closure: (text)Submitter-id: Assigned-to: Closer-id:

Question: Can Project Specific data be sanitized?