A Product Maturity Model for Industrial · PDF file1 A Product Maturity Model for Industrial Practice Dr. Joachim Wegener [email protected] jw, 09.02.2009 Overview

1

A Product Maturity Model forIndustrial Practice

Dr. Joachim Wegener

[email protected]

jw, 09.02.2009

Overview

• Motivation

• Product Maturity Model

• Criteria System for the Product Maturity Model

• First Experiences

• Summary, Outlook

2

System Development

System Specification

Component

Development

Implementation Unit Test

Integration

Integration Test

Vehicle Test

Production Test

Vehicle Integration

System Test

System Design

Model Test

System Development


Component

Development


Integration

Integration Test

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

System Design

Model Test

3

System Development



Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Component

Development

Component

Development

System Development and QA



Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

4




Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing

Component

Development




Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing

Supporting processes: project mgmt., defect mgmt.

Component

Development

5



Component

Development


Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing


Open defects

t



Component

Development


Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing


Open defects

t

6



Component

Development


Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing


Open defects

t



Component

Development


Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing


Open defects

t

Forecasting product

maturity is difficult

7

Quality relationships according to ISO 9126

Processquality

Internalquality

Externalquality

Quality in use

influences influencesinfluences

Depends on Depends onDepends on

Process Software product Effect of software product

Evaluation of process

capabilities

• CMMI

• SPICE (ISO 15504)



Unit Test

Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing


SPICE-Assessments

Implementation

Component

Development

8

Situation

Process models concentrate on assessment of development processes and fall

short in evaluating the most important element of the development process – the

resulting product [Nastro]. Therefore,

• maturity level predictions provide erroneous results

� actual product maturity does not correspond to predicted product maturity

� more defects in product than expected

� to many defects are only discovered in late development phases

• resulting project risks

� start of serial production with reduced functionality comp. to specification

� absolute number of critical and severe defects are found too late during

development

� extra costs due to additional test effort for additional releases and

additional quality assurance measures

Objectives for the Product Maturity Model

• better determination of product maturity

• application of the quality model

– for continuous measurement and assessment of quality level in

development projects,

– for a meaningful comparison of quality levels of different states of

development and releases,

– independent of make or buy decisions

• supporting systematic selection of counter measures

• selection of a “doable” set of assessment criteria for a start

9

Elements of the Product Maturity Model

• Continuous tracking and assessing of the internal quality (observation of internal software characteristics and structures)

Hardware Quality

System Quality

Software Quality

Process Quality

• Continuous tracking and assessing of the external quality (testing the application against the specification)

• Continuous assessment of quality assurance (observation of testing intensity, testing scope and testing depth)

• Organizational aspects (consideration of general conditions and resources, consideration of project management, version and configuration management, quality and test management)

• Continuous tracking and assessing of hardware

Criteria System for the Product Maturity Model

Hardware Quality

System Quality(External Quality)

Software Quality(Internal Quality)

Process Quality

Specification Quality

Project Planning and Controlling

Quality of Software Design

Implementation Quality

Test Quality

Hardware Quality

Functional Quality

Approval Quality

10

Criteria System for the Product Maturity Model:

Test Quality Criteria

Systematics of testing, testing methodology

Scope and completeness of tests regarding

markets, vehicle configurations, etc.

Character, scope, and intensity of accomplished quality assurance measures

Requirements coverage measurement

Main Criteria

Code coverage measurement

Scope and completeness of the Defect

Management (OEM and supplier)

Existence of module, integration, and system testing

Existence of coverage reports

Sub-Criteria

Existence of and compliance with a test plan (OEM, supplier)

Structuring of the test in test phases

Testing aims, testing priorities, testing objects

Appropriate testing infrastructure and testing tools

Character and scope of constructive QA measures

Character and scope of static analysis methods

Character and scope of organizational QA measures

Respective requirements management and test management reports

Test scope and intensity

Number of defects not found by the supplier (also in relation to the number of defects found), defects per

testing hour

Missing status changes

Review

Classification Tree Method

Analysis Activities

Requirements Tracing

Coverage Measurements

Defect Management

SPICE Assessment


• Assessment of each state of development with regard to its maturity level based on the following criteria

– Quality of planned features for the state of development(Rollout Plan)

• Specification quality

• Testing quality

• Functional quality

– Internal quality of the state of development from

code quality assessments

– External quality of the state of development from qualification testing

• Each criterion can contribute a previously defined, maximum number of quality points to the overall quality of a state of development

• The weighting of feature quality to internal quality and external quality is 2:1:1

Spezifikationsqualität

Projektplanung und -controlling

HW-Qualität



Qualität des SW-Design

Implementierungsqualität

Prüfqualität

Funktionale Qualität

HW-Qualität



Qualität des SW Design

Implementierungsqualität

Prüfqualität

Funktionale Qualität

HW-Qualität

Projektstart Serienbeginn

…

Freigaben Freigaben

Spec-Freeze E1 EMPB

11


0.1 0.45 0.45

2 2 N N

Functional QualitySpecification Quality Test Quality

Internal Quality

Application Quality

Quality Feature NQuality Feature 1 . . . Approval Quality

Criteria System for the Product Maturity ModelMaximal reachable quality points for the different criteria (example)

0

100

200

300

400

500

600

700

800

900

R1 R2 R3 R4 R5 R6 R7 R8

Features

Feature Quality

Internal Quality

Approval Quality

Maximum reachable

12



Internal Quality

Application Quality




Internal Quality

Application Quality


No. of feature changesGeneral Specification Quality Specification Gap Probability

Motivation: Starting point for development

Definition:Identifiability of individual requirements (Id)classifiability of requirements (class)the absence of redundancies in requirements (rd)clarity of requirements (clarity)testability of the requirements (test)

Analysis activity: These criteria can only be checked and evaluated within a specification review with respective check questions.

10

34 TestClarityRdClassIdGSQ

++++=

100*)( GSQGSQq =

13



Internal Quality

Application Quality



Motivation: Tests with no requirements indicate gaps in the specification

Definition: Number of tests for which no requirement exists in relation to the number of tests to which requirements can be assigned

Analysis activity: Can be easily determined from requirements tracing (tests not linked to requirements)



Internal Quality

Application Quality



Motivation: Frequent modifications disrupt the software and system development and increase the probability of defects

Definition: Number of changed feature requirements in relation to the number of feature requirements

Analysis activity: Can be easily determined from change management

quirementseNoOfFeatur

quirementsdFeatureNoOfChange CF

Re

Re=

), * CF). * ((q(CF) 0100512100max +−=

14



Internal Quality

Application Quality



0,040 – 0,080

0,020 – 0,040

0,000 – 0,020

Metric value (CF)

50 – 0

75 - 50

100 - 75

Quality

0

10

20

30

40

50

60

70

80

90

100

0,0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0

quirementseNoOfFeatur

quirementsdFeatureNoOfChange CF

Re

Re=

), * CF). * ((q(CF) 0100512100max +−=



Internal Quality

Application Quality



0.5 0.2 0.3

15



Internal Quality

Application Quality




Internal Quality

Application Quality


Requirements Coverage Test Coverage

Motivation: Evaluation of test completeness based on the requirements

Definition: Number of tested requirements in relation to the number of testable requirements

Analysis activity: Can be easily determined from requirements tracing (requirements without tests). Testability might be hard to check but is already verified in specification quality

0.55 0.45

16



Internal Quality

Application Quality


Requirements Coverage Test Coverage

Motivation: Evaluation of test intensity of defined test based on the minimum test coverage criteria

Definition: Minimum number of test cases as defined in the classification-tree method in relation to the defined number of test cases

Analysis activity: Automatically calculated by the classification-tree editor CTE XL

0.55 0.45



Internal Quality

Application Quality


17



Internal Quality

Application Quality


Defect frequency and criticalityCompleteness of Impl. Correctness of Impl.

Motivation: Evaluation of completeness of implementation

Definition: Number of implemented requirements in relation to number of demanded requirements

Analysis activity: Can be easily determined from requirements tracing and test management



Internal Quality

Application Quality



Motivation: Evaluation of correctness of implementation

Definition: Number of incorrectly implemented requirements in relation to the number of implemented requirements

Analysis activity: Can be easily determined from requirements tracing and test management

18



Internal Quality

Application Quality



Motivation: Assessment of defect frequency and severity

Definition: Number and severity of the determined defects in relation to the number of requirements

Analysis activity: Can be easily determined from defect management

uncriticalseverecritical

uncritical

severe

critical

DRDRDRDR

quirementsNo

alDefectsNoUncriticDR

quirementsNo

fectsNoSevereDefectsNoSevereDeDR

quirementsNo

DefectsNoCritical*fectsNoCritalDeDRDefectRate

++=

=

+==

+==

Re*35.0

4.0Re

*.50 else ,0 if 0

6.0Re

8.0 else ,0 if 0:

)),(DR*(q(DR) 0100100max −=



Internal Quality

Application Quality



uncriticalseverecritical

uncritical

severe

critical

DRDRDRDR

quirementsNo

alDefectsNoUncriticDR

quirementsNo

fectsNoSevereDefectsNoSevereDeDR

quirementsNo

DefectsNoCritical*fectsNoCritalDeDRDefectRate

++=

=

+==

+==

Re*35.0

4.0Re

*.50 else ,0 if 0

6.0Re

8.0 else ,0 if 0:

)),(DR*(q(DR) 0100100max −=

0.500 – 1.000

0.250 – 0.500

0.000 – 0.250

Metric value (DR)

50 – 0

75 - 50

100 - 75

Quality0

10

20

30

40

50

60

70

80

90

100

0,0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0

Assessment idea – sampling points:

� one critical failure already results in a red

assessment of the quality

� Severe errors are important but are given less

weight. 1 severe error per 5 requirements results in a red assessment

19



Internal Quality

Application Quality


Separate Quality Model for Internal Quality is used

• 95 metrics are drawn upon for the assessment of internal quality

– 18 architecture quality

– 38 design quality

– 39 code quality

• Metric values are determined in Code Quality Assessments

• for the assessment of design and code quality various tools are used

• usually, metrics have to be corrected manually



Unit Test

Integration

Integration Test

System Design

Vehicle Test

Production Test

Vehicle Integration

System Test

OEM

Supplier

Model Test

Review

System testing


SPICE-Assessments

Implementation

Component

Development

Code-Analytik

20



Internal Quality

Application Quality


Architecture Design Code

Documentation

Validation ofStructure

Validation ofInterfaces

Validation ofLayers

Concepts

OO-Usage

DesignPitfalls

Bundling &Complexity

Encapsulation &Information Hiding

OverallImpression

Inheritance

Quality-Attributes

Coding-Guidelines

OverallImpression

OverallImpression



Internal Quality

Application Quality


Failure Removal Trend Absolute Failure Number Failure Growth Failure Frequency and Criticality

Absolute number of failures in relation to the overall number of requirements

Number of new errors per operating hour from qualification tests

Number of expected failures in the final release under the supposition of a continued failure-removal rate

21



Internal Quality

Application Quality






leaseesCurrentNoOfFailur∆

lease)∆esLastNoOfFailurleaseresCurrent(NoOfFailuFET

leaseastTimeSinceL

leasenalTimeLeftFiRe

ReRe

Re

Re+

−

=



Internal Quality

Application Quality






Number and severity of the determined failures in relation to the overall number of requirements

22

First Experiences

• First evaluation in a project with 211 features and 1700 singular

requirements ongoing

• Assessments already performed for five releases showing an increase

from 50 to 169 quality points

• Initial effort: 10 – 20 man days

• Additional effort for assessment of one release: ~ 3 man days (due to

manual calculation of some metrics)

• Overall specification quality only partially applied

• Sometimes test quality assessment only based on requirements

coverage

Summary, Outlook

• Reliable product maturity figures necessary to control projects

• In practice, mainly process maturity evaluations are performed

• Process maturity models address product maturity only indirectly

• Therefore, we propose a product maturity model combining elementsfrom

– Process quality

– Internal quality and

– External quality

• First evaluations are currently performed

• In the future, the product maturity model will help to compare

– different releases of a product as well as

– products from different suppliers, different projects, and different domains

Documents

A Product Maturity Model for Industrial · PDF file1 A Product Maturity Model for Industrial Practice Dr. Joachim Wegener [email protected] jw, 09.02.2009 Overview