Introduction to Experimental Design and Analysis - Department of

Dr. John Mellor-Crummey

Department of Computer ScienceRice University

johnmc@cs.rice.edu

Introduction to ExperimentalDesign and Analysis

COMP 528 Lecture 11 22 February 2005

2

Experimental Design and Analysis

Understand how to• Design a experiments for measurement or simulation• Develop a model that describes the data obtained• Estimate the contribution of each factor to performance• Isolate measurement errors• Estimate confidence intervals for model parameters• Check if alternatives are significantly different• Check if a model is adequate

3

Goals for Today

Understand• What are the benefits of experimental design• Terms• Avoiding mistakes in experimental design• Basic taxonomy of experimental designs

—simple design, factorial design, fractional factorial design

• Understand 22 factorial design and its analysis—sign table method—properties—analysis—allocating variation

4

Why Experimental Design?

• Obtain maximum information from fewest experiments— minimize time spent gathering data

• Quantify effects from different factors using analysis• Determine if a factor’s effect is significant

—differences might be random variations caused by– measurement errors– parameters not controlled

5

Terms - I

• Response variable: outcome of an experiment—generally represents measured performance of the system—e.g. throughput: transactions/second, round-trip latency, etc.

• Factors: variables with alternatives that affect response—also called predictor variables or predictors—e.g. CPU type, memory size, # disk drives, workload used, etc.

• Levels: values a factor can assume—also called “treatment” in experimental design literature—e.g. CPU type levels: Itanium2, Alpha 21264, Opteron—e.g. memory size levels: 512MB, 1GB, 2GB, 4GB

• Primary factors: factors whose effects need to be quantified• Secondary factors: factors that are not being quantified

—they impact performance; we may not be interested in how much

6

Terms - II

• Replication: repetition of some or all experiments—if all experiments repeated 3x, experiment is said to have 3 replications

• Experimental design: plan for experimentation—number of experiments, factor level combinations for each, replications

• Experimental unit: any entity used for experiments—workstations, patients, land in agriculture expts—goal of experimental design: minimize impact of variation among units

• Interaction: if level of A changes effect of level change of B

86B253B1A2A1

Non-interacting

96B253B1A2A1

Interacting

lines parallel in graph of A vs. B

lines not parallel in graph of A vs. B

7

Common Mistakes in Experimentation - I

• Ignore variation due to experimental error—every measured value is a random variable—measured values change even if controllable vars kept constant—must compare variation due to factor vs. experimental error

– don’t make decision about factor’s effect without this comparison!

• Fail to control important parameters—only some parameters selected as factors and varied—must control other variables expected to have a significant effect

– e.g. if measuring impact of memory speed on performance,• use same CPU type/speed for experiments if CPU not a factor

• Fail to isolate effects of different factors—if varying several factors are being varied simultaneously

– design experiments so that effects of factors can be separated

8

Common Mistakes in Experimentation - II

• Use simple one-factor-at-a-time designs—leads to too many experiments—yields too little information per experiment—proper design ⇒narrower confidence intervals with same # expts

• Ignore interactions between factors—cannot estimate interactions with one-factor-at-a-time

experiments

• Conduct too many experiments—# experiments needed depends upon # factors, # factor levels—enormous single-step design vs. several steps

– better to use multiple steps• each with small designs and # levels

– first step: test analysis assumptions and whether transformationsrequired

– remaining steps: more factors and levels

9

Types of Experimental Designs

• Simple designs• Full factorial design• Fractional factorial design

10

• What is a simple design?— start with a typical configuration— vary one factor at a time to see how performance changes

• Example: comparing workstation configurations1. run typical configuration using a benchmark2. run experiments to pick the best CPU by varying CPU only3. using the best CPU, run a set of experiments to find the

minimum memory size yielding good performance4. using the best (CPU,memory) configuration, examine the

impact of disk RPM on the benchmark’s performance• Given: k factors; ith factor has ni levels

• Drawbacks— if factors interact, may yield wrong conclusions— fails to make best use of # experiments: statistically inefficient

Simple Designs (Not recommended)

number of experiments

!

n =1+ (ni"1)

i=1

k

#

11

Full Factorial Designs

• Explores every possible combination at all levels of factors

• Example n = (5 CPU types)(4 memory sizes)(2 disk RPMs)(4 workloads) = 160 experiments

• Advantages—thorough: every possible configuration of workload is examined

– can find effect of every factor, secondary factors, and interactions

• Disadvantages—cost: too many experiments, especially with repetitions

• Ways to reduce cost—reduce number of levels per factor (2 is very popular)—reduce # factors:

– initially only examine a few levels of each factor– prune unimportant factors, then try more factors per level

—use fractional factorial designs

number of experiments

!

n = ni

i=1

k

"

12

Fractional Factorial Designs

• A full factorial design may require many experiments• How can we get by with less: fractional factorial design• Example

—full factorial design (here, a 24 design) n = (2 CPU types)(2 memory sizes)(2 disk RPMs)(2 workloads) = 16 experiments

—fractional factorial design (here a 24-1 design)

WorkloadWorkloadWorkloadWorkloadWorkloadWorkloadWorkloadWorkload

CPUCPUCPUCPUCPUCPUCPUCPU

DiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemory

“half replicate design”

13

In Class Exercise

• A system’s performance depends upon the following factors—CPU type: Pentium4, Opteron, Athlon—OS type: Linux, Windows, Solaris—file compression utility: bzip2, gzip, zip

• How many experiments are necessary if:—there is significant interaction among the factors—the interactions are small compared to the main effects—there is no interaction among the factors

14

Beginning2k Factorial Designs

15

2k Factorial Designs

• What are they?—design to determine effect of k factors, each with 2 levels

• Why consider them?—easy to analyze—helps order factors based on impact—useful to identify

– factors that have significant impact ⇒ study with full factorial design– factors have little impact ⇒ not of interest for quantitative study

• How to select 2 levels—if factor effect is expected to be unidirectional ⇒ select min, max

– performance increases or decreases with factor– e.g. performance improves with more memory

16

22 Factorial Designs

• Special case of 2k factorial designs, k = 2—two factors at two levels

• Utility—22 designs are simple to analyze with regression

17

Example: a 22 Design

Consider impact of memory & cache sizes on performance

Define 2 categorical variables

Model performance using a non-linear equation in xA and xB

Solve using regression

5.2 GFLOPS2.5 GFLOPS6

3.2 GFLOPS1.5 GFLOPS4

Memory Size - 2GBMemory Size - 1GBL3 Cache Size (MB)

xA=-1 if 1GB memory+1 if 2GB memory xB=

-1 if 4MB cache+1 if 6MB cache

!

y = q0

+ qA xA + qB xB + qAB xA xB

18

Example: a 22 Design

Regress performance in GFLOPS on xA and xB

Substitute y, xA and xB

4 equations, 4 unknowns: unique solution

Interpretation—mean performance: 3.1 GFLOPS—effect of memory: 1.1GFLOPS—effect of cache: .75 GFLOPS—cache and memory interaction: .25 GFLOPS

!

y = q0

+ qA xA + qB xB + qAB xA xB

!

1.5 = q0" qA " qB + qAB

3.2 = q0

+ qA " qB " qAB

2.5 = q0" qA + qB " qAB

5.2 = q0

+ qA + qB + qAB

!

y = 3.1+1.1xA + .75xB + .25xA xB

19

!

q0

=1

4(y1

+ y2

+ y3

+ y4)

qA =1

4("y

1+ y

2" y

3+ y

4)

qB =1

4("y

1" y

2+ y

3+ y

4)

qAB =1

4(+y

1" y

2" y

3+ y

4)

Solving the equations for the qi’s

Computing Effects for 22 Design

!

y = q0

+ qA xA + qB xB + qAB xA xB

!

y1

= q0" qA " qB + qAB

y2

= q0

+ qA " qB " qAB

y3

= q0" qA + qB " qAB

y4

= q0

+ qA + qB + qAB

Substituting 4 observations into the model

Notice: expressions for qA, qB, qAB (contrasts)• are linear combinations of responses• sum of coefficients is 0

y4114y31-13y2-112y1-1-11yBAExpt

20

Calculating Effects with a Sign Table

total/4.25.751.13.1total134.412.45.22.53.21.5y

1111-11-11-1-1111-1-11

ABBAI

All possible combinations of -1, 1AB is product of columns A and By is set of observationscompute product of each column and y;write product underneathdivide through totals by 4 to compute regression coefficients

21

1111-11-11-1-1111-1-11

ABBAI

Sign Table Properties

• Sum of entries in columns A, B, AB is 0

• Sum of squares of entries in each column is 4

• Columns are orthogonal since inner product of column pairs is 0!

xAi

= 0i=1

4

"

!

xBi

= 0i=1

4

"

!

xAixBi

= 0i=1

4

"

!

xAi

2= 4

i=1

4

"

!

xAixBi( )

2

= 4i=1

4

"

!

xBi

2= 4

i=1

4

"

!

xAixAixBi( ) = 0

i=1

4

"

!

xAixBi

= 0i=1

4

"

!

xBixAixBi( ) = 0

i=1

4

"

22

Computing Sample Mean with Sign Table

!

y =1

4yi

i=1

4

"

!

y =1

4q0

+ qA xAi + qB xBi + qAB xAixBi( )i=1

4

"

sample mean

!

=1

4q0

i=1

4

" +1

4qA xAi

i=1

4

" +1

4qB xBi

i=1

4

" +1

4qAB xAixBi

i=1

4

"

!

= q0

23

Computing Total Variation with Sign Table

!

= yi " y ( )2

i=1

4

#

!

= qA xAi + qB xBi + qAB xAixBi( )2

i=1

4

"

Total variation

!

= qA xAi( )2

i=1

4

" + qB xBi( )2

i=1

4

" + qAB xAixBi( )2

i=1

4

" + product terms

!

= qA2

xAi( )2

i=1

4

" + qB2

xBi( )2

i=1

4

" + qAB2

xAixBi( )2

i=1

4

"

!

= 4qA2

+ 4qB2

+ 4qAB2

SST = SSA + SSB + SSAB =

SSA = SSB = SSAB =

!

4qB2

!

4qA2

!

4qAB2

!

4qA2

+ 4qB2

+ 4qAB2

24

Allocating Variation

Importance of a factor = how much variation it explains

SST = SSA + SSB + SSAB = 4(1.1)2 + 4(.75)2 + 4(.25)2 = 7.34

SSA 4(1.1)2

SST 7.34 Variation due to A = = = .66

SSB 4(.75)2

SST 7.34Variation due to B = = = .31

SSAB 4(.25)2

SST 7.34Variation due to interaction A & B = = = .03