Large Scale Requirements Engineering

Lecture 21

Sjaak Brinkkemper

Garm Lucassen

17 October 2014

1

Outline

This presentation contains two recent publications on Requirements Engineering in large scale software producing organizations.

1. Linguistic engineering for requirements similarity matching

2. Factors affecting decision making

2

Speeding up Requirements Management in a Product Software Company

Linking Customer Wishes to Product Requirements through Linguistic Engineering

Johan Natt och Dag

Lund Institute of Technology, Sweden

Vincenzo Gervasi

Pisa University

Italy

Sjaak Brinkkemper

Utrecht University

The Netherlands

Björn Regnell

Lund University

Sweden

Published as: Johan Natt och Dag, Vincenzo Gervasi, Sjaak Brinkkemper, Björn

Regnell: A Linguistic-Engineering Approach to Large-Scale

Requirements Management. IEEE Software 22(1): 32-39 (2005)

3

Complexity in large scale RE

• Market- and technology-driven software development

• Large, complex software systems

• Product lines

• New releases through projects

• Several thousands of whishes, needs, and requirements stored in a database

• Distributed development

• Uncertainty, frequent change, and time pressure

• Requirements written in plain text

• Multiple levels of abstraction

4

Baan

• Founded in 1978 in the Netherlands – 15,000+ customer sites worldwide – 1000+ employees – 1,000.000+ users

• Markets – Manufacturing industry – Engineering industry

• Headquartered in Barneveld, The Netherlands – Offices and partners worldwide – 400 employees Development – 200 employees Customer Service & Support

• Now part of Infor

5

Requirements Management at Baan (SSA Global)

Market Requirements

Business Requirements

Market Requirements

Business Requirements

Market Requirements

Business Requirements

Develo

pm

ent

Managem

ent

Require

ments

Managem

ent

ongoing, continuous

release based

release based

ReleaseInitiation Version

Definition

ConceptualSolution

DefinitionStudy

FunctionalDesign

TechnicalDesign

SoftwareComponentSoftware

Component

6

Linking requirements

• Customer wish = Market Requirement – A customer wish related to current or future markets, defined

using the terminology and context of the customer

• Product req. = Business Requirement – Generic customer wish to be covered by Baan solutions

described in Baan’s terminology and context

Market

Requirements

Business

Requirements

Market

Requirements

Business

Requirements

Market

Requirements

Business

Requirements

7

Example requirements

8

Linkage in the Baan RDB

Year

# Business

Requirements

# Linked

# Market

Requirements

# Linked

1996 0 0 183 113

1997 5 4 683 262

1998 275 169 1,579 388

1999 709 261 2,028 502

2000 669 167 1,270 397

2001 1,000 153 864 224

2002 1,121 340 1,695 514

Total 3,779 1,094 8,302 2,400

9

Linkage backlog

0

500

1000

1500

2000

1996 1997 1998 1999 2000 2001 2002

Year

#re

qs.

Unlinked

Linked

0

200

400

600

800

1000

1200

#re

qs.

8,302 market

requirements

2,400 linked

3,779 business

requirements

1,094 linked

10

Potential solutions

• Add more resources

– Too costly!

• Reduce the amount of incoming requirements

– Not a good idea or can not be done!

• Rules and guidelines

– Will not or can not be used!

• Provide support through linguistic engineering

– Not really tried in this context.

11

Linguistic Engineering

• Pragmatic approach

– ”to engineer products which deal with NL and which satisfy the constraints in which they have to operate”

• Criteria – Usability

– Cost-benefit

– Resources

– Scale

– Techniques

– Robustness

– Flexibility

– Openness

– Domain dependentness

– Efficiency

– Implementation

– Intermediate results

– Testing and evaluation

12

Suggestion list

MR10013

Pos Requirement Similarity

1 BR10012 0.45

2 BR10156 0.43

3 BR10006 0.42

4 BR10536 0.38

5 BR10987 0.36

6 BR10273 0.36

7 BR10740 0.34

8 BR10419 0.33

9 BR10622 0.24

10 BR10082 0.21

13

Steps in linguistic process

Market Requirements

Business Requirements

ongoing, continuous

Suggestions

Flattening

Tokenization

Stemming

Stop word removal

Similaritycalculation

Flattening

Tokenization

Stemming

Stop word removal

Market Requirements

Business Requirements

Market Requirements

Business Requirements

Market Requirements

Business Requirements

ongoing, continuous

Suggestions

Flattening

Tokenization

Stemming

Stop word removal

Similaritycalculation

Flattening

Tokenization

Stemming

Stop word removal

14

Example and metric

i bi m

bi m

bm

irir

irirrr

)(log1)(log1

)(log1)(log1),(

22

22

Tokenization

storing

multiple

charts

in

one

file

the

user

should

be

able

to

store

several

charts

in

a

single

sov

file

store

multiple

chart

in

one

file

Lemmatisation

the

user

should

be

able

to

store

several

chart

in

a

single

sov

file

Stop word removal

user

store

several

chart

sov

file

store

multiple

chart

file

Storing

multiple

charts in one

file.

The user

should be able

to store

charts in a

single (.SOV)

file

Flattening

market requirement

business

requirement

15

Evaluation: Recall related to top list size

MR10013

Pos Requirement Similarity

1 BR10012 0.45

2 BR10156 0.43

3 BR10006 0.42

4 BR10536 0.38

5 BR10987 0.36

6 BR10273 0.36

7 BR10740 0.34

8 BR10419 0.33

9 BR10622 0.24

10 BR10082 0.21

16

Evaluation: Recall related to top list size

MR10013

Pos Requirement Similarity

1 BR10012 0.45

2 BR10156 0.43

3 BR10006 0.42

4 BR10536 0.38

5 BR10987 0.36

6 BR10273 0.36

7 BR10740 0.34

8 BR10419 0.33

9 BR10622 0.24

10 BR10082 0.21

17

Evaluation: Recall related to top list size Manually linked and presumed correct

MR10013

Pos Requirement Similarity

1 BR10012 0.45

2 BR10156 0.43

3 BR10006 0.42

4 BR10536 0.38

5 BR10987 0.36

6 BR10273 0.36

7 BR10740 0.34

8 BR10419 0.33

9 BR10622 0.24

10 BR10082 0.21

TP FP

FN TN

18

Evaluation: Recall related to top list size Manually linked and presumed correct

MR10013

Pos Requirement Similarity

1 BR10012 0.45

2 BR10156 0.43

3 BR10006 0.42

4 BR10536 0.38

5 BR10987 0.36

6 BR10273 0.36

7 BR10740 0.34

8 BR10419 0.33

9 BR10622 0.24

10 BR10082 0.21

Recall(7) = 2 / 3 = 67%

19

Result 1

20

So what? A comparative cost-benefit evaluation

Subset providing 100% recall using a top-10 list

BRs 690

The MRs linked by product managers 1,249

Savings: ~66% or ~115 hours

Manual search

– Assume 1 search term is enough

~30 hits

Automatically supported 10 hits

21

Potential improvements

• Aggregate similarity measures using other techniques

• Reuse information in already linked requirements

• Incorporate semantics (synonyms, hypernyms, hyponyms)

• Expert validation

22

Questions and Discussion

23

Decision making

Paper:

Factors Affecting Decision Outcome and Lead-time in Large-Scale Requirements Engineering

Joint work with: Krzysztof Wnuk, Jaap Kabbedijk, Bjorn Regnell and David Callele

Submitted for publication

Goal:

Identifying which characteristics of change requests ( e.g. number of products, release number, type of customer) may influence the decision lead time and the decision outcome

24

Decision making

Knowledge intensive activities in decision making

• Defining the scope of a product by selecting the right features.

• Change impact analysis

• Difficulties such as shifting, ill–defined or competing goals and values

• Multi-objective decision problems

• Power and politics

• Subjective judgments

• Risk of inappropriate decisions is high

• Often unstructured or semi–structured and made only once

• Hindering the evaluation of the decision outcomes

25

Case company

Type of company Developer of globally marketed embedded systems

Number of employees More than 4000

The size of the product line More than 2000 features integrated into the platform and millions LOC.

Major project A platform project with a two year lead time

Minor projects Further platform releases usually focused on the platform’s adaptations for different products

Decision entity A feature, defined as a group of requirements that constitute new functionality enhancements to the platform upon which market value and implementation cost can be estimated.

Number of features in a typical

project

A typical major project has from several hundreds, up to one thousand. A typical minor project has less than 100 features.

The scope management process The scope is maintained in a document called a feature list, updated each week after a meeting of the change control board.

26

Change Request process

27

Decision log entry

ID 54

Change Request HD resolution for video

Decision Outcome Accepted

Comments This will enlarge our market share in this sector

Description of proposed change Add HD resolution for recording

Justification Requested by a large provider

Proposition Area Video

Main affected Technical Group Video Group

The products impacted by a change All products with a camera

The feature(s) impacted by a change F14, F18 and F23

The release of the platform project impacted

by a change

The second release of the main platform

The markets impacted by a change US and Canada

Affected key customer Customer X

Affected Functional Group HD Group

Submittal Date 09-02-09

RM tool ID 10F1

Decision Date 18-02-09

28

Analysis of decision log

• 1439 decisions of the 2008 releases

• Lead time: time between submission and decision making

• Outcome: Accepted or Rejected

29

Research Questions

TOWARDS EXPLORING FACTORS AFFECTING DECISION OUTCOME AND LEAD-TIME IN LARGE-SCALE REQUIREMENTS ENGIN

Research question Aim

RQ1: Which characteristics of a

decision affect the decision lead

time?

To understand which decision characteristics (e.g.,

number of products, release number, type of customer)

have a significant impact on the decision lead time

RQ2: Which characteristics of

a decision affect the decision

outcome?

To understand the relationship between the decision

characteristics and the acceptance or rejection of a

decision

RQ3: Is the outcome of a decision

related to the decision’s lead

time?

To understand the relation between the acceptance rate

and the decision lead time

30

Lead time

31

More

• Findings will be communicated when the paper has been published.

32