Joint Process & Product Assessment: Concept and Benefits · 2008-05-23 · Joint Process & Product Assessment: Concept and Benefits Jean-Louis LETOUZEY ... ISO 9126 Product Ass. Synthesis

Joint Process & Product Assessment:Concept and Benefits

Jean-Louis LETOUZEY

DNV ITGS France

© Det Norske Veritas AS. All rights reserved Slide 223 May 2008

Agenda

1. Concept

2. Methodology

3. Report samples

4. Benefits

5. Conclusion

1. The concept


The need for Joint Process & Product Assessment

In some situation, management needs:

� Detailed visibility on (intermediate or final) product quality

� Precise identification of major product risks

AND

� Precise identification of necessary corrective actions

� Good understanding of impact, relationshionship between- Process quality and Product quality- Process weaknesses into Product weaknesses


Process & Product Assessment overview

Preparation

A four stages process:

CMMi

Process Ass.

ISO 9126

Product Ass.

Synthesis Restitution

Tailoring to theassessment’s objectives

Sponsor interview

Logistics

….

….

Findings

Interactions

Recommandations

Report.ppt

2/3 Weeks

2. The Methodology


Process assessment: 1/3

� The Process assesment is a CMMI “capacity” assessment

� As the engineering level 3 PAs (RD, TS, PI, VER, VAL) are driving intermediate or final products quality, they are included in the scope of the process assessment in addition to all level 2 PAs

� Tayloring rules allow to include additional engineering add-on PAs covering security or safety

� Assessment method is close to CMMI “SCAMPI B class” standard:- Typical 4 members assessment team - Document review + interview- Confidentiality

- Strenghts and weaknesses identification- Findings by PAs…and global findings- ….



Organizational Process Performance (OPP)Quantitative Project Management (QPM)

4 – QuantitativelyManaged

3 – Defined

5 – Optimizing

2 – Managed

Requirements Development (RD)Technical Solution (TS)Product Integration (PI)Verification (VER)Validation (VAL)Organizational Process Focus (OPF)Organizational Process Definition (OPD) +IPPDOrganizational Training (OT)Integrated Project Management (IPM) +IPPDRisk Management (RSKM)Decision Analysis and Resolution (DAR)

Requirements Management (REQM)Project Planning (PP)Project Monitoring and Control (PMC)Supplier Agreement Management (SAM)Measurement and Analysis (MA)Process and Product Quality Assurance (PPQA)Configuration Management (CM)

Organizational Innovation and Deployment (OID)Causal Analysis and Resolution (CAR)

Continuousprocess improvement

Quantitative management

Process standardization

Basic project management



Rating rules:

� Each Practices is rated using a 4 level scale- Fully implemented- Largely implemented- Partially implemented

- Not implemented- Not Applicable

� Consolidation at Practice level

� No Maturity level characterisation


Product Assessment: 1/8

ObjectivesObjectives

ProductProduct

Quality ModelQuality Model

Analysis ModelAnalysis Model

The four components of a product assessment:


Code


� A quality model :- The decomposition of code quality into caracteristics and subcaracteristics up to

detailed metrics or controls.

Detailed decomposition and control points depends of the assessment objectives

RxR4

Readability

R3R2R1 Tx comments.

Quality

Portability Reliability ….ilitiesMaintainability

Understandability Changeability

UnstructurnessEssential Complexity

Line count

# violations of rule X

Coupling



Reuse

Maintainability

Portability

Efficiency

……

ISO 9126 Quality Model

Reliability

Maintain

Deliver

Evolve

Test

Code

?

Testability

Changeability

Efficiency

Reliability

Maintainability

Reusability



6 ordered,layeredquality attributes

Reuse

Testability

Changeability

Efficiency

Reliability

Reusability

Maintain

Evolve

Test

Code

ReusabilityMaintainability

EfficiencyChangeability

ReliabilityTestability

Maintainability

Deliver

© Copyright DNV 2007-2008



Any control needed within two subcharacteristics is atta ched to the one of lower level

Detail of sub-characteristicsCharacteris tic SubCharacteris ticReusability Architecture re la ted ReusabilityMainta inability Unders tandabilityMainta inability ReadabilityEfficiency Processor UseEfficiency Memory UseEfficiency Rom UseChangeability Logic ChangeabilityChangeability Architecture ChangeabilityChangeability Data ChangeabilityReliability Ins truction re la ted ReliabilityReliability Exception HandlingReliability Fault ToleranceReliability Architecture re la ted ReliabilityReliability Logic re la ted ReliabilityReliability Data re la ted ReliabilityTes tability Unit leve l Tes tabilityTes tability Integra tion leve l Tes tability


Product Assessment: 6/8Sample of Controls

Depending on the language and the assessment object ives, the number of control points vary from 35 to 50

Mainta inability Unders tandability There is no function with an essentie l complexity eV(G) > 5Mainta inability Unders tandability The comment percentage is > 25%Mainta inability Unders tandability There is no break ins truction outs ide a switch s tructureMainta inability Readability There is no file with more than 1000 linesMainta inability Readability There is no dead code



� An analysis model: - Set of rules that allow to consolidate low level metrics and indicators

within synthetic high level indicators used for managing the quality.

- Consolidation covers two axes:- The hierarchy of code artifacts- The quality model hierarchy

Sub charact.

Characteristic Component

File

A B C

X

MA MBMC

MX ?

Consolidation by “average” is masking most weakness es



� All measure values are unified within a file related index

� This index « represent », « evaluate » the workload necessary to put the file back in conformity

- Taking account the typical workload associated to each Non Conformity (NC) (range from 0.01 to 20)

- Taking account the number of NC occurrences

� This total index is the basis to characterize each file quality within 5 levels (A to E)

� The index consolidation rule is logic and simple ;Consolidation is done by adding the indexes

These indexes are pure intrinsic measurement. They do not pretend to reflect any business or context related information like criticity of the piece of code, frequency of execution…..

These indexes represent a distance to an objective. The lower the index, the lower the distance

When index decreases, quality improves

3. Result samples


Result samples

� Table of contents of the 3 reports

Process Assessment….

� Global findings

� Findings per PA (S & W)

� Quotation

� Recommandations

Product Assessment….

� Findings per characteristics

� Quotation and synthesis

� Recommandations


Using the product assessment models

Some guidelines

Cop

yrig

ht D

NV

200

5-20

07

Remediation Load

Maintainability

Efficiency

Reliability

Testability

Changeability (

Difficult situation

Maintainability

Efficiency

Reliability

Testability

Changeability (

Ideal situation

Impact

Long term

Short term

Maintainability

Efficiency

Reliability

Testability

Changeability(

Presentation

Syntax

Structure



Interpreting the restitution

Maintainability

Efficiency

Reliability

Testability

Changeability

2 points of view for the modelAn external view that represents the percieved quality evaluatedby consolidation of the hierarchy of characteristics

Σ

Σ

Σ

An analytic view provided by orthogonal characteristics

One understands impact of each Non Conformity and improvement on quality characteristic and life cycle issues.



S/W Product Assessment report sample

� Volumetry- Lines count- Complexity

- Comments- Distribution

� By Characteristic- NC and Indexes- Distribution analysis- Main risks (Top 10)


S/W Product Assessment report sample

� Quotation- By part- Repartition


REQM PP PMC SAM MA PPQA CM RD TS PI VER VAL

ML2 ML3

No (few) non functionnal requirements

- Testability, reliability….

- Coding rules

- Architecture directives

Consolidation of findings « process » and « product »

� Impact of process weaknesses:

The process weaknesses explain some of the product weaknesses : root cause analysis

Detected by

Product Evaluation

QualityDashboard

Peer ReviewFeedback

PoorCode

Quality

Result in


REQM PP PMC SAM MA PPQA CM RD TS PI VER VAL

ML2 ML3

� Compensation of product weaknesses:

Low integration level testability

Expected quality is achieved through an extra (but expensive and time consuming) workload at validation phase: Compensati on mecanism

“ Delivered” quality

Does not supporthigh reliability expectations

Does not supporthigh reliability expectations

Consolidation of findings « process » and « product »

4. Benefits


Benefits related to product assessment

� Low intrusiveness- 2 auditors with a local coordinator- Supplier’s involvement is limited to the delivery of the source code - The audit team brings its own set of audit tools

� Precise and reproductible- Analysis report is only based on factual, quantified findings and

measurements (no interviews) made with automated analysis tools- Check points, metrics used for the assessment are selected for having no

“false positive” situation or context

� Prioritized recommandations- Takes into account dependencies between quality caracteristics- Takes into account their impact on the life cycle

� Useable report- Short- Graphic oriented


Joint assessment benefits as a Conclusion

� Detailed and precise picture of a given situation

� Clear understanding of weaknesses and their impact on product- Process assessment provides a root cause analysis of product weaknesses

� Clear understanding of the recommandations and their impact - Product refactoring on product quality- Process improvement on development cost- Process improvement on TCO

- Process improvement on product quality

Thanks for your attentionThanks for your attention

Questions ?Questions ?

Documents

Joint Process & Product Assessment: Concept and Benefits · 2008-05-23 · Joint Process & Product Assessment: Concept and Benefits Jean-Louis LETOUZEY ... ISO 9126 Product Ass. Synthesis