Sayyad slides ase13_v4

Scalable Product Line Configura4on: A Straw to Break the Camel’s Back

Abdel Salam Sayyad Joseph Ingram Tim Menzies Hany Ammar

Sound bites

Real SoGware Product Lines… … are big, complex … representa4ve feature models are now available for research … scale up or die !

2

What does it take? Mul4-‐objec4ve problem formula4on An op4mizer that relies heavily on user preferences Respect domain constraints, and...

… A Straw to Break the Camel’s Back

Roadmap

①   State of the Art ②   Experiment & Results ③   Conclusion

Roadmap


Feature Models •  Feature models = a

lightweight method for defining a space of op4ons

•  De facto standard for modeling variability

5

Cross-‐Tree Constraints

Cross-‐Tree Constraints

Size ? 10 Features, 8 Rules

Feature Model Repository

6

Size ? 290 Features, 421 Rules

cardinal = 2.26x1049 [Pohl ‘11]

Feature Models of Real SoGware Projects

7

Size ? 6888 Features; 344,000 Rules cardinal? Forgeddabou4t!

Size isn’t all…

Property SPLOT Linux Size Significantly smaller Significantly larger

Constraints Significantly less Significantly more Feature Groups High ra4o Low ra4o

Leaves Deeper Shallower

8

T. Berger, S. She, R. Lotufo, A. Wasowski, and K. Czarnecki, "A Study of Variability Models and Languages in the Systems SoGware Domain," IEEE Tran So< Eng. 2013

A Mul4-‐Objec4ve Formula4on Mul4-‐objec4ve op4miza4on = naviga4ng compe4ng concerns

–  Success criteria = choose features that achieve the user preferences!

9

Suppose each feature had the following metrics: 1.  Boolean USED_BEFORE? 2.  Integer DEFECTS 3.  Real COST

Show me the space of “best op4ons” according to the objec4ves:

1.  That sa4sfies most domain constraints (0 ≤ #viola4ons ≤ 100%) 2.  That offers most features 3.  That we have used most before 4.  Using features with least known defects 5.  Using features with least cost

No single “op4mum” solu4on

10

Higher-‐level Decision Making

The Pareto Front

The Chosen Solu4on

State of the Art

11

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)

Objec4ves

Mul4-‐obj Single-‐obj

State of the Art

12

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)

Benavides ‘05

Single-‐obj, CSP Up to 25 features Exponen[al [me

Objec4ves


State of the Art

13

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)

Pohl ‘11

Benavides ‘05 BDD, SAT, CSP, 90 models

“All products” not a`empted if cardinal > 3x106 Objec4ves


State of the Art

14

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)

Pohl ‘11 Lopez-‐Herrejon

‘11 Benavides ‘05

Fixing model inconsistencies 27 min. for 94 features Objec4ves


State of the Art

15

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)


‘11

Sayyad ’13a

Benavides ‘05 Mul[-‐obj configura[on, Z3

>15 days for E-‐Shop Objec4ves


State of the Art

16

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)


‘11

Sayyad ’13a

Velazco ‘13

Benavides ‘05

Objec4ves


White ‘07, ‘08, 09a, 09b, Shi ‘10, Guo ‘11 Scale up with randomly-‐

generated feature models

State of the Art

17

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)


‘11

Sayyad ’13a

Velazco ‘13

Johansen ‘11

Benavides ‘05

Test covering arrays, [med out on all ops for Linux 6888 features (and some ops on smaller FMs)

Objec4ves


State of the Art

18

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)


‘11

Henard ‘12

Sayyad ’13a

Velazco ‘13

Johansen ‘11

Benavides ‘05

Test covering arrays 20 hours for Linux 6888 features 6-‐wise coverage

Objec4ves


State of the Art

19

Features

9

290

544

6888

SPLO

T Linu

x (LVA

T)


‘11

Henard ‘12

Sayyad ’13a

Velazco ‘13

Sayyad ’13b Johansen ‘11

Benavides ‘05

Mul[-‐obj configura[on, IBEA 30 configs in 30 minutes for Linux 6888 features

Objec4ves


Roadmap


Two Algorithms

21

1)  NSGA-‐II [Deb et al. ‘02] Non-‐dominated Sor4ng Gene4c Algorithm

2) IBEA [Zitzler and Kunzli ‘04] Indicator-‐Based Evolu4onary Algorithm

Binary Dominance

Con4nuous Dominance

7 Feature Models

22

Feature Fixing

•  Look for mandatory or dead features. •  Fix those feature in the evolu4on. •  Skip rules with only those features.

23

Experiment

24

Low Crossover & Muta[on rates Stopping Criteria

25

HV = Hypervolume of dominated region %Correct = % of solu4ons that are fully correct (out of 300 solu4ons)

Comparing %Correct

26

0%

20%

40%

60%

80%

100%

ToyBox axTLS eCos FreeBSD Fiasco uClinux Linux X86

NSGA-‐II No FF

NSGA-‐II with FF

IBEA No FF

IBEA with FF

i.e. % of solu4ons that are fully correct (out of 300 solu4ons)

How to crack the nut?

•  What if we told IBEA what good solu4ons look like?

27

•  Good solu4on? … A correct solu4on, with: •  Minimal skeleton? •  Maximum features?

•  Then what? •  Seed the ini4al popula4on with a pre-‐calculated “good” solu4on.

28

•  How to generate good (correct) solu4ons? •  Z3 SMT (Sa4sfiability Module Theory) Solver •  Two-‐objec4ve IBEA, min viola4ons, max features

•  Z3: super fast, low on selected features •  2-‐obj IBEA: slow, but feature-‐rich

Correct solu4ons aGer 30 minutes for Linux Kernel feature model

29

Correct solu4ons aGer 30 minutes for Linux Kernel feature model

30

… A Straw to Break the Camel’s Back

More recent result •  added the “PULL” trick (give more objec4ve weight to constraint viola4ons)

31

Why is this interes4ng? •  20 correct solu4ons to choose from within 2 minutes

32

Do I have the user’s aven4on? •  2 seconds… •  to stay focused

33

•  10 seconds… •  to stay on task

•  few minutes? •  To provide

4mely input at a mee4ng

Roadmap


Sound bites Scalability

Method innova[on is key

35

Problem Formula4on Give the best (comprehensive yet concise) picture to the user.

IBEA with Feature Fixing

Con[nuous dominance Respect domain constraints

Enter Z3?

One day we will build that bridge!

Acknowledgment This research work was funded by the Qatar

Na[onal Research Fund (QNRF) under the Na[onal Priori[es Research Program

(NPRP) Grant No.: 09-‐1205-‐2-‐470.

Education

Sayyad slides ase13_v4