Software EngineeringSoftware Engineering

Software Process and Project Metrics

ObjectivesObjectives

To introduce the necessity for software measurement, metrics and indicators.

To differentiate between process and project metrics.

To compare and contrast Lines-Of-Code (LOC) and Function Point (FP) metrics.

To describe statistical process control for managing variation between projects.

Measurement & MetricsMeasurement & Metrics

Against:

Collecting metrics is too hard ... it’s too time consuming ... it’s too political ... they can be used against individuals ... it won’t prove anything

For:

In order to characterize, evaluate, predict and improve the process and product a metric baseline is essential.

“Anything that you need to quantify can be measured in some way that is superior to not measuring it at all” Tom Gilb

TerminologyTerminology Measurement:

Measure: Quantitative indication of the extent, amount, dimension, or size of some attribute of a product or process. A single data point.

Metrics: The degree to which a system, component, or process possesses a given attribute. Relates several measures (e.g. average number of errors found per person hour).

Indicators: A combination of metrics that provides insight into the software process, project or product.

Types: Direct: Immediately measurable attributes of an artefact (e.g. line of code,

execution speed, defects reported). Indirect: Aspects of an artefact that are not immediately quantifiable (e.g.

functionality, quantity, reliability).

Faults: Errors: Faults found by the practitioners during software development. Defects: Faults found by the customers after release.

A Good Manager MeasuresA Good Manager Measures

measurementmeasurement

What do weWhat do weuse as ause as abasis?basis?

• • size?size? • • function?function?

project metricsproject metrics

process process metricsmetrics

procesprocesss

productproduct

product product metricsmetrics

“Not everything that can be counted counts, and not everything that counts can be counted.” - Einstein

Process MetricsProcess Metrics

Focus on quality achieved as a consequence of a repeatable or managed process. Strategic and Long Term.

Statistical Software Process Improvement (SSPI). Error Categorization and Analysis: All errors and defects are categorized by origin The cost to correct each error and defect is recorded The number of errors and defects in each category is computed Data is analyzed to find categories that result in the highest cost

to the organization Plans are developed to modify the process

Defect Removal Efficiency (DRE). Relationship between errors (E) and defects (D). The ideal is a DRE of 1:

)/( DEEDRE

Project MetricsProject Metrics

Used by a project manager and software team to adapt project work flow and technical activities. Tactical and Short Term.

Purpose: Minimize the development schedule by making the necessary

adjustments to avoid delays and mitigate problems. Assess product quality on an ongoing basis.

Metrics: Effort or time per SE task Errors uncovered per review hour Scheduled vs. actual milestone dates Number of changes and their characteristics Distribution of effort on SE tasks

Product MetricsProduct Metrics

focus on the quality of deliverables

Product metrics are combined across several projects to produce process metrics.

Metrics for the product: measures of the analysis model

complexity of the design internal algorithmic complexity

architectural complexity

data flow complexity

code metrics

Metrics GuidelinesMetrics Guidelines

Use common sense and organizational sensitivity when interpreting metrics data.

Provide regular feedback to the individuals and teams who have worked to collect measures and metrics.

Don’t use metrics to appraise individuals. Work with practitioners and teams to set clear goals and

metrics that will be used to achieve them. Never use metrics to threaten individuals or teams. Metrics data that indicate a problem area should not be

considered “negative.” These data are merely an indicator for process improvement.

Don’t obsess on a single metric to the exclusion of other important metrics.

Normalization for MetricsNormalization for Metrics

How does an organization combine metrics that come from different individuals or projects?

Depend on the size and complexity of the project.

Normalization: compensate for complexity aspects particular to a product.

Normalization approaches: size oriented (lines of code approach)

function oriented (function point approach)

Typical Normalized MetricsTypical Normalized Metrics

Size-Oriented: errors per KLOC (thousand lines of code), defects per KLOC, R

per LOC, page of documentation per KLOC, errors / person-month, LOC per person-month, R / page of documentation

Function-Oriented: errors per FP, defects per FP, R per FP, pages of documentation

per FP, FP per person-month

Project LOC FP Effort (P/M)

R(000) Pp. doc

Errors Defects People

alpha 12100 189 24 168 365 134 29 3

beta 27200 388 62 440 1224 321 86 5

gamma 20200 631 43 314 1050 256 64 6

Why Opt for FP Measures?Why Opt for FP Measures?

Independent of programming language. Some programming languages are more compact, e.g. C++ vs. Assembler.

Use readily countable characteristics of the “information domain” of the problem.

Does not “penalize” inventive implementations that require fewer LOC than others.

Makes it easier to accommodate reuse and object-oriented approaches.

Original FP approach good for typical Information Systems applications (interaction complexity). Variants (Extended FP and 3D FP) more suitable for real-time and scientific software (algorithm and state transition complexity).

Computing Function PointsComputing Function Points

Establish count for input domain and system interfaces

Analyze information domain of the application and develop counts

Weight each count by assessing complexity

Assign level of complexity (simple, average, complex) or weight to each count

Grade significance of external factors, F_i, such as reuse, concurrency, OS, ...

Assess the influence of global factors that affect the application

Compute function points

FP = SUM(count x weight) x C where complexity multiplier C = (0.65+0.01 x N) degree of influence N = SUM(F_i)

Analyzing the Information DomainAnalyzing the Information Domain

complexity multiplier

function points

number of user inputs number of user outputs number of user inquiries number of files number of ext.interfaces

measurement parameter

3 4 3 7 5

countweighting factor

simple avg. complex

4 5 4 10 7

6 7 6 15 10

= = = = =

count-total

X X X X X

Taking Complexity into AccountTaking Complexity into Account Complexity Adjustment Values (F_i) are rated on a scale of 0 (not

important) to 5 (very important):1. Does the system require reliable backup and recovery?

2. Are data communications required?

3. Are there distributed processing functions?

4. Is performance critical?

5. System to be run in an existing, heavily utilized environment?

6. Does the system require on-line data entry?

7. On-line entry requires input over multiple screens or operations?

8. Are the master files updated on-line?

9. Are the inputs, outputs, files, or inquiries complex?

10. Is the internal processing complex?

11. Is the code designed to be reusable?

12. Are conversion and instillation included in the design?

13. Multiple installations in different organizations?

14. Is the application designed to facilitate change and ease-of-use?

Exercise: Function PointsExercise: Function Points

Compute the function point value for a project with the following information domain characteristics:Number of user inputs: 32

Number of user outputs: 60

Number of user enquiries: 24

Number of files: 8

Number of external interfaces: 2

Assume that weights are average and external complexity adjustment values are not important.

Answer:7.40165.0)72108424560432(

Example: SafeHome FunctionalityExample: SafeHome Functionality

UserUser SafeHome System

SafeHome System

SensorsSensors

UserUser

Monitor and

Response System

Monitor and

Response System

System Config Data

System Config Data

Password

Zone Inquiry

Sensor Inquiry

Panic Button

(De)activate

Messages

Zone Setting

Sensor Status

(De)activate

Alarm AlertPassword, Sensors, etc.

Test Sensor

Example: SafeHome FP CalcExample: SafeHome FP Calc

complexity multiplier

function points

number of user inputs number of user outputs number of user inquiries number of files number of ext.interfaces

measurement parameter

3 4 3 7 5

countweighting factor

simple avg. complex

4 5 4 10 7

6 7 6 15 10

= = = = =

count-total

X X X X X

3

2

2

1

4

9

8

6

7

22

58

52

]46.065.0[]01.065.0[ iF 1.11

Exercise: Function PointsExercise: Function Points

Compute the function point total for your project. Hint: The complexity adjustment values should be low ( )

Some appropriate complexity factors are (each scores 0-5):1. Is performance critical?

2. Does the system require on-line data entry?

3. On-line entry requires input over multiple screens or operations?

4. Are the inputs, outputs, files, or inquiries complex?

5. Is the internal processing complex?

6. Is the code designed to be reusable?

7. Is the application designed to facilitate change and ease-of-use?

10 iF

Measuring QualityMeasuring Quality

Software quality can be difficult to measure and is often highly subjective.

Correctness: the degree to which a program operates according to

specification.

metric = Defects per FP.

Maintainability: the degree to which a program is amenable to change.

metric = Mean Time to Change. Average time taken to analyze, design, implement and distribute a change.

Further Measures of QualityFurther Measures of Quality

Integrity: the degree to which a program is impervious to outside attack.

metric = .

Summed over all types of security attacks, i, where t = threat (probability that an attack of type i will occur within a given time) and s = security (probability that an attack of type i will be repelled).

Usability: the degree to which a program is easy to use.

metric = (1) the skill required to learn the system, (2) the time required to become moderately proficient, (3) the net increase in productivity, (4) assessment of the users attitude to the system.

Covered in HCI course

i

ii st )1(

McCall’s Triangle of QualityMcCall’s Triangle of Quality

PRODUCT TRANSITIONPRODUCT TRANSITIONPRODUCT REVISIONPRODUCT REVISION

PRODUCT OPERATIONPRODUCT OPERATIONCorrectness

Reliability

Usability

IntegrityEfficiency

Maintainability

Flexibility

Testability

Portability

ReusabilityInteroperability

McCall’s Quality FactorsMcCall’s Quality Factors

Graded on a scale of 0 (low) to 10 (high): Auditability (the ease with which conformance to standard can be checked) Accuracy (the precision of computations and control) Communication commonality (the degree to which standard interfaces,

protocols and bandwidth are used) Completeness (the degree to which full implementation of required function

has been achieved) Conciseness (the compactness of the program in LOC) Consistency (the use of uniform design and documentation techniques) Data Commonality (the use of standard data structures and types throughout

the program) Error Tolerance (the damage that occurs when the program encounters an

error) Execution efficiency (the run-time performance of a program) Expandability (the degree to which architectural, data or procedural design can

be extended)

Further Quality FactorsFurther Quality Factors Generality (The breadth of potential application of program components) Hardware independence (The degree to which the software is decoupled from

the hardware on which it operates) Instrumentation (The degree to which the program monitors its own operation

and identifies errors) Modularity (the functional independence of components) Operability (the ease of operation of a program) Security (the availability of mechanisms that control or protect programs and

data) Self-documentation (the degree to which the source code provides meaningful

documentation) Simplicity (The degree to which a program can be understood without

difficulty) Software system independence (the degree of independence from OS and

nonstandard programming language features) Traceability (the ability to trace program components back to analysis) Training (the degree to which the software assists new users)

Deriving Quality MetricsDeriving Quality Metrics

Each of McCall’s Quality Metrics combines different quality factors.

Examples:Correctness = Completeness + Consistency + Traceability

Portability = Generality + Hardware Independence + Modularity + Self-Documentation + System Independence

Maintainability = Conciseness + Consistency + Instrumentation + Modularity + Self-Documentation + Simplicity

This technique depends on good objective evaluators because quality factor scores can be subjective.

“McCall’s quality factors were proposed in the early 1970s. They are as valid today as they were in that time. It’s likely that software built to conform to these factors will exhibit high quality well into the 21st century, even if there are dramatic changes in technology.”

Managing VariationManaging Variation

How can we determine if metrics collected over a series of projects improve (or degrade) as a consequence of improvements in the process rather than noise?

Statistical Process Control: analyzes the dispersion (variability) and location

(moving average).

Determine if metrics are (a) stable (the process exhibits only natural or controlled changes) or (b) unstable (process exhibits out of control changes and metrics cannot be used to predict changes).

Control ChartControl Chart

Compare sequences of metrics values against mean and standard deviation. e.g. metric is unstable if eight consecutive values lie on one side of the mean.

0

1

2

3

4

5

6

1 3 5 7 9 11 13 15 17 19

Projects

Er, Er

rors

foun

d/

revie

who

ur

Mean

- std. dev.

+ std. dev.