Lecture 21 Measurements and Metrics in Design

SWE 316: Software Design and Architecture

Obj

ectiv

esLecture 21

Measurements and Metrics in Design


Understand software measurements Understand the different types of

metrics Understand how to measure the

internal quality attributes of the software.

Handout

Ch 10b


Introduction Measurement: the process by

which numbers and symbols are assigned to attributes.

“You can neither predict nor controlwhat you cannot measure”.

-DeMarco

“Count what is countable, measure what is measurable, and what is not measurable, make measurable“

-Galileo

Introduction Advantages & Disadvantages

Why do we measure?

Types of Metrics

Triangle of Quality 2/36


Measurement and metrics Software metrics are units of measurement that

are used to characterize: Software products, e.g. designs, source code, and

test cases.

Software process, e.g. the activities of analysis, designing, and coding.

Software people, e.g. the efficiency of an individual tester, or the productivity of an individual designer.

Software project.


Why do we measure?

Types of Metrics



Software metrics Software metrics provide a quantitative

way to assess the quality of internal product attributes, thereby enabling the software engineer to assess quality before the product is built.

Metrics provide the insight necessary to create effective analysis, design, code and tests.

Product metrics can be used for general predictions or to identify anomalous components.


Why do we measure?

Types of Metrics



Software metrics Although some companies have introduced

measurement programs, most organisations still don’t make systematic use of software measurement.


Why do we measure?

Types of Metrics



Advantages of software metrics Metrics quantitatively define success and failure

and or degree of success or failure for the product, the process or the people.

Metrics identify and quantify improvement, lack of improvement or degradation in the products, the product or people.

Metrics make meaningful and useful managerial and technical decisions.

Metrics identify trends. Metrics make meaningful estimates.


Why do we measure?

Types of Metrics



Disadvantages of software metrics Metrics can be harmful if they are misused or

collected in a wrong way.

Faulty analysis (statistical) of metrics can make metricsuseless, or even harmful.

For metrics to be useful, metrics must be gathered systematically and regularly, preferably in an automated tool, which is not an easy task.

Metrics are interrelated. Usually the influence of one metric has impact on other metrics, which makes it hard to control them.


Why do we measure?

Types of Metrics



Disadvantages of software metrics Collecting metrics should follow a formal

procedurein order to get metrics collected correctly. This needsa lot of effort.

Metrics need to be validated in order to be used, which is again not a trivial task.


Why do we measure?

Types of Metrics



Why do we measure? Assess the status of an ongoing project

Track potential risks

Uncover problem areas before they go “critical,”

Adjust work flow or tasks,

Evaluate the project team’s ability to control quality of software work products.

?Introduction Advantages &

DisadvantagesWhy do we measure?

Types of Metrics



Who can benefit from software metrics? Managers can know:

How much does each process cost? How productive is the staff? How well is the code being developed? Will the user be satisfied with the product? How to improve?

Engineers can know: If the requirements are testable If they have found all the faults If the product or and the processes meet the goals What will happen in the future


Why do we measure?

Types of Metrics



Measurement acceptance No wide acceptance of software measurements:

Management personnel rarely know how to analyze, interpret or use the software metrics.

Various ways of analyzing software metrics.

Lack of measurement validation.


Why do we measure?

Types of Metrics



Measurement principles The objectives of measurement should be

established before data collection begins; Each technical metric should be defined in

an unambiguous manner; Metrics should be derived based on a

theory that is valid for the domain of application (e.g., metrics for design should draw upon basic design concepts and principles and attempt to provide an indication of the presence of an attribute that is deemed desirable);

Metrics should be tailored to best accommodate specific products and processes.


Why do we measure?

Types of Metrics



Measurement process1- Formulation. The derivation of software measures and metrics appropriate for the representation of the software that is being considered.2- Collection. The mechanism used to accumulate data required to derive the formulated metrics.3- Analysis. The computation of metrics and the application of mathematical tools.4- Interpretation. The evaluation of metrics results in an effort to gain insight into the quality of the representation.5- Feedback. Recommendations derived from the interpretation of product metrics transmitted to the software team.


Why do we measure?

Types of Metrics



Metrics attributes Simple and computable. It should be relatively easy to

learn how to derive the metric, and its computation should not demand inordinate effort or time

Empirically and intuitively persuasive. The metric should satisfy the engineer’s intuitive notions about the product attribute under consideration

Consistent and objective. The metric should always yield results that are unambiguous.

Consistent in its use of units and dimensions. The mathematical computation of the metric should use measures that do not lead to bizarre combinations of unit.

Programming language independent. Metrics should be based on the analysis model, the design model, or the structure of the program itself.

An effective mechanism for quality feedback. That is, the metric should provide a software engineer with information that can lead to a higher quality end product


Why do we measure?

Types of Metrics



Goal-oriented software measurement The Goal/Question/Metric Paradigm

Establish an explicit measurement goal that is specific to the process activity or product characteristic that is to be assessed

Define a set of questions that must be answered in order to achieve the goal, and

Identify well-formulated metrics that help to answer these questions.

Goal definition template Analyze {the name of activity or attribute to be

measured} for the purpose of {the overall objective of the analysis} with respect to {the aspect of the activity or attribute

that is considered} from the viewpoint of {the people who have an interest

in the measurement} in the context of {the environment in which the

measurement takes place}.


Why do we measure?

Types of Metrics



Metrics assumptions A software property can be measured.

The relationship exists between what we can measure and what we want to know. We can only measure internal attributes but are often more interested in external software attributes. This relationship has been formalised and

validated.

It may be difficult to relate what can be measured to desirable external quality attributes.


Why do we measure?

Types of Metrics



Metrics guidelines Use common sense and organizational sensitivity

when interpreting metrics data. Provide regular feedback to the individuals and

teams who have worked to collect measures and metrics.

Don’t use metrics to appraise individuals. Work with practitioners and teams to set clear

goals and metrics that will be used to achieve them.

Never use metrics to threaten individuals or teams.

Metrics data that indicate a problem area should not be considered “negative.” These data are merely an indicator for improvement.

Don’t obsess on a single metric to the exclusion of other important metrics.


Why do we measure?

Types of Metrics



Internal and external attributes

Reliability

Number of procedure parameters

Cyclomatic Complexity

Program size in Lines of Code

Number of Error Messages

Length of User Manual

Maintainability

Usability

Portability

External attributes Internal attributes


Why do we measure?

Types of Metrics



Data collection A metrics program should be based on a set of

product and process data. Data should be collected immediately and, if

possible, automatically.

Data accuracy Don’t collect unnecessary data

The questions to be answered should be decided in advance and the required data identified.

Tell people why the data is being collected. It should not be part of personnel evaluation.

Don’t rely on memory Collect data when it is generated not after a project

has finished.


Why do we measure?

Types of Metrics



Collection and analysis principles Whenever possible, data collection and

analysis should be automated; Valid statistical techniques should be

applied to establish relationship between internal product attributes and external quality characteristics

Interpretative guidelines and recommendations should be established for each metric

It is not always obvious what data means Analyzing collected data is very difficult.

Professional statisticians should be consulted if available.


Why do we measure?

Types of Metrics



Types of metrics to measure

measurement

What do weuse as abasis? • size? • function?

project metrics

process metricsprocess

product

product metrics


Why do we measure?

Types of Metrics



Process measurement We measure the efficacy of a software process

indirectly. That is, we derive a set of metrics based on the outcomes

that can be derived from the process. Outcomes include

Measures of errors uncovered before release of the software Defects delivered to and reported by end-users Work products delivered (productivity) Human effort expended Calendar time expended Schedule conformance Other measures.

We also derive process metrics by measuring the characteristics of specific software engineering tasks.


Why do we measure?

Types of Metrics



Project measurement Used to minimize the development schedule by

making the adjustments necessary to avoid delays and mitigate potential problems and risks

Used to assess product quality on an ongoing basis and, when necessary, modify the technical approach to improve quality.

Every project should measure: Inputs—measures of the resources (e.g., people, tools)

required to do the work. Outputs—measures of the deliverables or work products

created during the software engineering process. Results—measures that indicate the effectiveness of the

deliverables.


Why do we measure?

Types of Metrics



Product metrics A quality metric should be a predictor of

product quality. Classes of product metric:

Dynamic metrics which are collected by measurements made of a program in execution;

Static metrics which are collected by measurements made of the system representations;

Dynamic metrics help assess efficiency and reliability; static metrics help assess complexity, understandability and maintainability.


Why do we measure?

Types of Metrics



Dynamic and static metrics Dynamic metrics are closely related to software

quality attributes It is relatively easy to measure the response time of a

system (performance attribute) or the number of failures (reliability attribute).

Static metrics have an indirect relationship with quality attributes You need to try and derive a relationship between these

metrics and properties such as complexity, understandability and maintainability.


Why do we measure?

Types of Metrics



Software product metricsSoftware metric Description

Fan in/Fan-out Fan-in is a measure of the number of functions or methods that call some other functionor method (say X). Fan-out is the number of functions that are called by function X. Ahigh value for fan-in means that X i s tightly coupled to the rest of the design andchanges to X will have extensive knock-on effects. A high value for fan-out suggeststhat the overall complexity of X m ay be high because of the complexity of the controllogic needed to coordinate the called components.

Length of code This is a measure of the size of a program. Generally, the larger the size of the code of acomponent, the more complex and error-prone that component is likely to be. Length ofcode has been shown to be one of the most reliable metrics for predicting error-proneness in components.

Cyclomatic complexity This is a m easure of the control complexity of a p rogram. This control complexity maybe related to program understandability.

Length of identifiers This is a measure of the average length of distinct identifiers in a p rogram. The longerthe identifiers, the more likely they are to be m eaningful and hence the moreunderstandable the program.

Depth of conditionalnesting

This is a measure of the depth of nesting of if-statements in a program. Deeply nested ifstatements are hard to understand and are potentially error-prone.

Fog index This is a measure of the average length of words and sentences in documents. The higherthe value for the Fog index, the more difficult the document is to understand.


Why do we measure?

Types of Metrics



Object-oriented metricsObject-oriented metric Description

Depth of inheritance tree This represents the number of discrete levels in the inheritance tree where sub-classes inherit attributes and operations (methods) from super-classes. The deeper the inheritance tree, the more complex the design. Many different object classes may have to be understood to understand the object classes at the leaves of the tree.

Method fan-in/fan-out This is directly related to fan-in and fan-out as described above and means essentially the same thing. However, it may be appropriate to make a distinction between calls from other methods within the object and calls from external methods.

Weighted methods per class

This is the number of methods that are included in a class weighted by the complexity of each method. Therefore, a simple method may have a complexity of 1 and a large and complex method a much higher value. The larger the value for this metric, the more complex the object class. Complex objects are more likely to be more difficult to understand. They may not be logically cohesive so cannot be reused effectively as super-classes in an inheritance tree.

Number of overriding operations

This is the number of operations in a super-class that are over-ridden in a sub-class. A high value for this metric indicates that the super-class used may not be an appropriate parent for the sub-class.


Why do we measure?

Types of Metrics



Class-oriented metrics (by Chidamber and Kemerer) Weighted Methods per Class (WMC): This metric is the

sum of the complexities of all the local methods. Depth of Inheritance Tree (DIT): This metric is the total

number of ancestor classes in the hierarchy that can affect a given class. Classes with high DIT values are usually complex classes.

Number Of Children (NOC): This metric is the total number of immediate subclasses in the class hierarchy that inherit from a given class.

Coupling Between Objects (CBO): This metric is the total number of classes in the hierarchy to which a given class is linked and related.

Response For a Class (RFC): This metric is a measure of the set of methods that can be called by a local method in a given class.

Lack of Cohesion of Methods (LCOM): This metric is the count of pairs of the instance variable set.


Why do we measure?

Types of Metrics



Another class-oriented metrics Number of Ancestor Classes (NAC): This metric captures the

number of classes that have potential influence on a given class because of the inheritance relation, since with multiple inheritance there might be two values for the DIT metric.

Number of Local Methods (NLM): This metric counts the number of local methods, which are accessible outside a given class (public methods).

Class Method Complexity (CMC): This metric is the summation of the internal structure complexity of all public and private local methods.

Number of Descendent Classes (NDC): This metric is proposed as an alternative to the NOC metric, since a class that inherits from more than one class can have different values of the NOC metric.

Coupling Through Abstract data types (CTA): This metric measures the total number of classes that are used as abstract data types in the data attribute declaration of a given class.

Coupling Through Message passing (CTM): This metric measures the number of different messages sent out from a class to other classes excluding the messages sent to objects created as local objects in the local methods of the class.


Why do we measure?

Types of Metrics



Lack of Cohesion of Methods: example Example: Consider a class C with three methods

M1,M2 and M3. Let {I1} = { a , b . c , d , e } and

{I2} = { a , b , e } and {I3} = {x,y,z}.

{I1} ∩ {I2} is nonempty, but {I1} ∩ {I3} and {I2} ∩ {I3} are null sets.

LCOM is the (number of null intersections - number of nonempty intersections), which in this case is 1.


Why do we measure?

Types of Metrics



Lack of Cohesion of Methods example (2)

m1 m2 m3

a b c dNDC={(m1, m2)}NNC ={(m1, m3) (m2, m3)}LOCM = max(2-1,0) =1

m1 m2 m3

a b c dNDC={(m1, m2) (m2, m3)}NNC ={(m1, m3)}LOCM = max(1-2,0) =0


Why do we measure?

Types of Metrics



McCall’s Triangle of Quality

MaintainabilityMaintainabilityFlexibilityFlexibility

TestabilityTestability

PortabilityPortability

ReusabilityReusability

InteroperabilityInteroperability

CorrectnessCorrectness

ReliabilityReliabilityEfficiencyEfficiency

IntegrityIntegrityUsabilityUsability

PRODUCT TRANSITIONPRODUCT TRANSITIONPRODUCT REVISIONPRODUCT REVISION

PRODUCT OPERATIONPRODUCT OPERATION


Why do we measure?

Types of Metrics

Triangle of Quality

32/36


Product operation

usability

Product revision

Product transition

integrity

maintainability

testability

reusability

portability

interoperability

operabilitytraining

I/O volume

Access controlAccess auditStorage efficiency

consistency

instrumentationexpandabilitygeneralitySelf-descriptiveness

modularitymachine independences/w system independence

comms. commonality

efficiency

correctness

reliability

flexibility

communicatativeness

I/O rate

execution efficiencytraceabilitycompleteness

accuracyerror tolerance

simplicityconciseness

data commonality

Quality taxonomyIntroduction Advantages &

DisadvantagesWhy do we measure?

Types of Metrics

Triangle of Quality

33/36


Measuring quality We have to turn our vague ideas about quality into

measurablesThe Quality Concepts

(abstract notions ofquality properties)

Measurable Quantities(define some metrics)

Counts taken fromDesign Representations(realization of the metrics)

usability

minutes taken for

some user task???

time taken to learn how

to use?

complexity

count procedure calls???

information flow between

modules?

reliability

run it and count crashes per hour???

mean time to failure?

Source: Budgen, 1994, pp60-1


Why do we measure?

Types of Metrics

Triangle of Quality

34/36


Establishing a metrics program Identify your business goals. Identify what you want to

know or learn. Identify your subgoals. Identify the entities and

attributes related to your subgoals.

Formalize your measurement goals.

Identify quantifiable questions and the related indicators that you will use to help you achieve your

measurement goals. Identify the data elements

that you will collect to construct the indicators that help answer your questions.

Define the measures to be used, and make these definitions operational.

Identify the actions that you will take to implement the measures.

Prepare a plan for implementing the measures.


Why do we measure?

Types of Metrics

Triangle of Quality

35/36


Summary Measurement: the process by which numbers

and symbols are assigned to attributes.

A metric as “a quantitative measure of the degree to which a system, component, or process possesses a given attribute.”

Measurement help evaluate, assess and control quality of software work products.


Why do we measure?

Types of Metrics


Documents

Lecture 21 Measurements and Metrics in Design