Introduction to Search Based Software Engineering SSS SEBASE Summer School, Birmingham University, July 2007 Mark Harman King’s College London

Introduction toSearch Based Software Engineering

SSS SEBASE Summer School, Birmingham University, July 2007

Mark Harman

King’s College London


SSSS Second SEBASE Summer School, Birmingham University, July 2008

Mark Harman



How on earth can software engineers do what John says they claim do

Mark Harman


How on earth can software engineers do what John says they claim do

-To engineer is to optimize

Mark Harman



Thanks

John Clark, University of York,Sebastian Elbaum, University of Nebraska Lincoln

Rob Hierons, Brunel UniversityZheng Li, King’s College London

Kiarash Mahdavi, King’s College LondonSpiros Mancoridis, Drexel University

Afshin Mansouri, King’s College LondonJian Ren, King’s College London

Joachim Wegener, DaimlerChryslerShin Yoo, King’s College London

YuanYuan Zhang, King’s College London

Mark Harman Introduction to SBSE

6

Where is King’s College London?

20 minutes’ walk


7

Search Based Software Engineering

SBSE what and whySBSE case studiesSBSE how

Current and future trendsAdvantages Multi Objective SearchInsight through searchCo Evolution


8



Current and future trends – personal viewsAdvantages Multi Objective SearchInsight through searchCo Evolution


9



Current and future trends – IMHOAdvantages Multi Objective SearchInsight through searchCo Evolution


10



Current and future trends – In Mark Harman’s OpinionAdvantages Multi Objective SearchInsight through searchCo Evolution


11

What is SBSE

In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures properties of the acceptable software artefacts we seek.


15

What is SBSE


So it is merely another application of Optimization Techniques?


16

What is SBSE


So it is merely another set of applications of Optimization Techniques?


17

What is SBSE


So it is merely another rich set of applications of Optimization Techniques?


18

What is SBSE


So it is merely another rich set of applications of Optimization Techniques?


19

What is SBSE


Have nails … seek hammer


20

What is SBSE


Have nails … seek hammers


21

What is SBSE



Generalise, compare, develop


22

What is SBSE




in an unbiased manner


23

What is Research?

In research we apply social search techniques to search large search spaces, guided by a natural evolution of memes





24

What is Research?

In research we apply social search techniques to search large search spaces, guided by a natural evolution of memes and all the usual human frailties





25

Why?


26

The Eight Queens Problem


27



28



29


Perfect


30


Perfect Score 0


31



32



33


Two Attacks


34


Two Attacks

Score -2


35



36



37


Three Attacks


38


Three Attacks

Score -3


39

That was easy


40

Generate a solution

Place

8 queens

on the

board

so that

there are

no

attacks


41

Scale up: Generate a solution

Place

44 queens

on the

board

so that

there are

no

attacks


42

Scale up: Generate a solution

Place

400 queens

on the

board

so that

there are

no

attacks


43

Checking vs Generating

Task One:

Write a method to determine which is the better of two placements of N queens

Task Two:

Write a method to construct a board placement with N non attacking queens


44


Task One:

Write a method to determine which is the better of two placements of N queens

Task Two:

Write a method to construct a board placement with N non attacking queens


45



Write a method to determine which is the better of two solutions

Conventional Software Engineering

Write a method to construct a perfect solution


46







47







48



Write a fitness function to determine which is the better of two solutions




49



Write a fitness function to guide a search aaaa




50



Write a fitness function to guide automated search




51

Examples


52

Case Studies


53

Evolutionary Algorithms

Selection

Insertion

Recombination

Mutation

Fitness evaluationTest

execution

End?


54

Evolutionary Testing

Selection

Insertion

Recombination

Mutation

Fitness evaluation

Test cases

Testexecution

End?


55


Selection

Insertion

Recombination

Mutation

Fitness evaluation

Test cases

Testexecution

End?

Execution


56


Selection

Insertion

Recombination

Mutation

Fitness evaluation

Test cases

Testexecution

End?

Execution

Monitoring


57

Temporal Testing

decelerationtmin

tmax

t

optimal point of time

for triggering the airbag

igniter Selection

Insertion

Recombination

Mutation

Fitness evaluation

Individuals

Test data

Monitoring

Fitness values

Testexecution

End?


58

Temporal Testing

decelerationtmin

tmax

t

optimal point of time

for triggering the airbag

igniter Selection

Insertion

Recombination

Mutation

Duration

Individuals

Test data

Monitoring

Fitness values

Testexecution

End?


59

Generation

Ex

ecu

tio

n

Tim

e (i

n c

ycl

es)

Evolutionary Test

Random Test

Evolution vs Random for Temporal Testing


60

Search Based Testing Applications

Temporal testingCoverage based testingFunctional testingRegression testingFinite State Machine testingInteraction testingException testingStress testingRobustness testing


61



Phil will talk

about some of these in

more detail

shortly


62




63


64


65


66


67


68


69

Finding UIOsHierons et al. 2004-2008

s1

s2

s3

a/0

a/0

a/1

b/1

b/1

b/0

a/1s3

b/0s2

b/1,a/1s1

UIOState


70

Target

Level 4

Level 3

Level 2

Level 1

Fitness = Approximation_Level + Local_Distance

Evaluation of predicate in a branching condition

if A = B Local_Distance = | A - B |

Identify relevant branching statements using control dependence

TargetTargetTarget

1. 1. Approximation levelApproximation level1. 1. Approximation levelApproximation level

Structural Evolutionary Testing


71

Evolutionary Testing vs Random

0

200000

400000

600000

800000

1000000

1200000

1400000

Num

ber

of te

st c

ases

ET 16915 42086 23633 35263

RT 199743 215834 470931 1251038

RT / ET 11,8 5,1 19,9 35,5

Triangle_int Triangle_float Complex My_atof0

20

40

60

80

100

120

Ach

ieve

d co

vera

ge

ET coverage 100 100 100 100

RT coverage 90,5 90,5 98,1 66,5

Triangle_int Triangle_float Complex My_atof

Effectiveness Cost


72

Autonomous Parking System


73



74


Stop


75

Autonomous Parking System - Input

psi

gap

dist2space

space length

space width


76


Generation 0

Generation 10

Critical

Generation 20

collision

Generation 20

collision


77

Regression Testing

Test case prioritzation

Test suite reduction

Test case selection


78

Regression Testing

Test case prioritzation Li, Harman, Hierons, TSE 2007


Test case selection


79

Test Case Prioritization

Order test cases

higher priority => earlier execution


80

Greedy

Optimal orderings:

A B C

B C A

C B A


81

Regression Testing



Test case selection


82

Regression Testing

Test case prioritzationTest suite reduction Yoo and Harman, ISSTA 07

Test case selection


83

Regression Test Selection


84

Regression Testing


Test suite reductionTest case selection you and someone else, TSE 2008/9 ?


85

Regression Testing


Test suite reductionTest case selection yoo and me, TSE 2008/9 ?


86

Does this look familiar?


87

Does this look familiar?


88

MDGM1

M2 M3

M4

M5 M6

Good Partition!

M1

M2

M3

M4

M5

M6

Bad Partition!

M1

M2

M4

M3M5

M6

MQ(Good Partition) > MQ(Bad Partition)

The Bunch ApproachMancoridis et al. 1998 - 2005


89

Hill Climbing


90

Hill Climbing


91

Hill Climbing

Maybe this is

no longer

the optimal approach

…


92

Hill Climbing

Kata Praditwong et al.


93

SBSE is so generic

TestingFitness function: execution time

Representation: input vector


94

SBSE is so generic



RestructuringFitness function: cohesion and coupling

Representation: mapping from modules to clusters


95

SBSE is so generic



RequirementsFitness function: customer satisfaction, cost

Representation: bitset of requirements


96

SBSE is so generic



Program ComprehensionFitness function: (in)direct aspect of cognition

Representation: code, architecture, visualization, model


97

SBSE is so generic



Project ManagementFitness function: project duration

Representation: mapping from teams to WPs


98

SBSE is so generic

Time for two more examplesProgram Comprehension Mahdavi et al. ICSM 05

Requirements Analysis Zhang et al. GECCO 07


99

SBSE is so generic

Time for two more examplesProgram Comprehension Mahdavi et al. ICSM 05

Requirements Analysis


100

Concept Assignment

“The process of assigning descriptive terms usually relating to computational intent.”

MOVE ‘EXAMPLE’ TO PRINT-LLMOVE ‘13’ TO PRINT-CC.CALL ‘PRINT’ USING P-PRINTLINE.MOVE POLICY-NUM TO OUT-PNUM.MOVE SCHEME-REF TO OUT-SREF.CALL ‘WRITE’ USING OUT-REC.

Write

Print


101

Allowing Overlapping Concept Binding

Fitness: Signal to noise ratioRepresentation: Space of possible overlapping bindings

MOVE ‘EXAMPLE’ TO PRINT-LLMOVE POLICY-NUM TO OUT-PNUM.MOVE ‘13’ TO PRINT-CC.MOVE SCHEME-REF TO OUT-SREF.CALL ‘PRINT’ USING P-PRINTLINE.CALL ‘WRITE’ USING OUT-REC.

PrintWrite

Call


102

SBSE is so generic

Time for two more examplesProgram ComprehensionRequirements Analysis Zhang et al. GECCO 07


103

The Next Release Problem

,i jvalue r c

1,..., ,...,j mC c c c 1,..., ,...,i nR r r r

1,..., ,...,j mWeight w w w 1,..., nCost cost cost

1

,m

i j i jj

score w value r c


104

Motorola Cell Phone Requirements


105

SBSE ApplicationsTransformation Cooper, Ryan, Schielke, Subramanian, Fatiregun, WilliamsRequirements Bagnall, Mansouri, ZhangEffort prediction Aguilar-Ruiz, Burgess, Dolado, Lefley, Shepperd Management Alba, Antoniol, Chicano, Di Pentam Greer, RuheHeap allocation Cohen, Kooi, Srisa-an Regression test Li, Yoo, Elbaum, Rothermel, Walcott, Soffa, Kampfhamer SOA Canfora, Di Penta, Esposito, Villani Refactoring Antoniol, Briand, Cinneide, O’Keeffe, Merlo, Seng, TrattTest Generation Alba, Binkley, Bottaci, Briand, Chicano, Clark, Cohen, Gutjahr,

Harrold, Holcombe, Jones, Korel, Pargass, Reformat, Roper, McMinn,Michael, Sthamer, Tracy, Tonella,Xanthakis, Xiao, Wegener, Wilkins

Maintenance Antoniol, Lutz, Di Penta, Madhavi, Mancoridis, Mitchell, SwiftModel checking Alba, Chicano, GodefroidProbe dist’ion Cohen, Elbaum UIOs Derderian, Guo, HieronsComprehension Gold, Li, MahdaviProtocols Alba, Clark, Jacob, TroyaComponent sel Baker, Skaliotis, Steinhofel, YooAgent Oriented Haas, Peysakhov, Sinclair, Shami, Mancoridis


106





107





108

The two SBSE ingredients

Representation

Fitness function


109


Representation

Fitness function

Should be easy

We always represent

Software Engineering

problems in data

structures


110


Representation

Fitness function

Often easy

We often define metrics


111

So far

SBSE is useful

SBSE is generic

SBSE is easy


112

Future

Multiple objective search

Co evolution

Sensitivity analysis

Metric validation

Landscape analysis

Interactive evolution


113

Future

Multiple objective search

Co evolution


Metric validation

Landscape analysis

Interactive evolution


114

Co Evolution

Two populations; Two fitness functions

Collaboration, and competition


115

Co Evolution

Two populations; Two fitness functions

Collaboration, and competition

applied to mutation testing Adamopoulos, Harman and Hierons GECCO 2004


116

Mutants Test Cases


117

Mutants Test Cases

Fitness?


118

Mutants Test Cases

Fitness for test cases

How many mutants

do you kill ?


119

Mutants Test Cases

Fitness for mutants

How many test cases

do you survive?


120


Inputs to search are typically estimates

Estimated Software Engineering attributes

Which estimates matter most?


121


Inputs to search are typically estimates

Estimated Software Engineering attributes

Which estimates matter most?

applied to component selectionYoo, Ren, Harman, Not yet published


122

Motorola Cell Phone Requirements


123

Metric Robustness

Metric performs under …

change, noise, fuzziness

Not just good solutions; robust solutions


124

Metric Robustness

Metric performs under …

change, noise, fuzziness

Not just good solutions; robust solutions

Metrics are fitness functions too!Harman and Clark, Metrics Symposium 2004


125

Landscape Analysis

What landscape is denoted by the metric?

Landscape properties

smoothness, plateaux, …


126

Landscape Analysis and Improvement

Improve the fitness landscape by transformation

SBSE allows transformation of program under testA Testability Transformation Harman et al. 2004 – 2008


127

Landscape Analysis and Improvement

Improve the fitness landscape by transformation

SBSE allows transformation of program under testA Testability Transformation Harman et al. 2002 – 2008

An example for Structural Testing McMinn et al. ISSTA 2007


128

Hitchcock Landscape

void alfred(double a, double b){ if (a == b) { // target 1 double c = b + 1;

if (c == 0) { // target 2 . . .


129

Hitchcock Landscape


130

Hitchcock Landscape


131

Interactive evolutionHarman ICPC 2007

Human in the loop

Caters for intangibles

Can flush out implicit assumptions

Co evolution of cognitive models

Lots of fun things to do here …


132

Summary

Software Engineering problems can be optimizedSoftware Engineering problems are often search problems

This should not be a surprise

The SBSE approach gives us a whole new set of ToolsConceptsTechniquesInsights

To optimize


133

Summary

Software Engineering problems can be optimizedSoftware Engineering problems are often search problems

This should not be a surprise

The SBSE approach gives us a whole new set of ToolsConceptsTechniquesInsights

To optimize is to engineer


134

Search Based Software Engineering is the focus of major initiatives

SEBASE $5m

5 year project

Search based software engineering

Led by King’s

Birmingham, York, Brunel

Daimler, Motorola, IBM

EvoTest $4m

3 year project

Evolutionary testing

Led by Valencia

King’s, FIRST, INRIA,

Daimler, Motorola, RILA, Xenon


135

Summary


136

Summary


137

Conclusions

Search is what you seek


138

Conclusions


www.sebase.org


139

Conclusions


www.sebase.org

Questions?

Documents

Introduction to Search Based Software Engineering SSS SEBASE Summer School, Birmingham University, July 2007 Mark Harman King’s College London