An Introduction to Design of Experiments...One‐Day DOX Course 8 Four Eras in the History of DOX •The agricultural origins, 1908 – 1940s •W.S. Gossett and the t-test (1908)

One‐Day DOX Course 1

An Introduction to Design of Experiments

Bradley JonesJMP Division of SASCary, North Carolina

Douglas C. MontgomeryRegents’ Professor of Industrial Engineering and Statistics

ASU Foundation Professor of EngineeringArizona State University

Reference

• Design and Analysis of Experiments, 9thedition (2017), D.C. Montgomery, Wiley, Hoboken NJ

• Website:• www.wiley.com/college/montgomery• Resources for students

• Data (Excel, JMP, Minitab Design‐Expert)• Supplemental material for each chapter

• Resources for instructors (pwrd required)• Student resources plus• Power Point slides• Solutions to end‐of‐chapter problems


Reference

• Goos, P. and Jones, B. (2011), “Optimal Design of Experiments: A Case Study Approach”, Wiley, UK


And this Journal of Quality Technology paper (2011)


Design of Engineering ExperimentsPart 1 – IntroductionChapter 1, Text

• Why is this trip necessary? Goals of the course

• An abbreviated history of DOX• Some basic principles and terminology• The strategy of experimentation• Guidelines for planning, conducting and

analyzing experiments


Introduction to DOX

• An experiment is a test or a series of tests• Experiments are used widely in the engineering

world • Process characterization & optimization• Evaluation of material properties• Product design & development• Component & system tolerance determination

• “All experiments are designed experiments, some are poorly designed, some are well-designed”


Engineering Experiments

• Reduce time to design/develop new products & processes

• Improve performance of existing processes

• Improve reliability and performance of products

• Achieve product & process robustness

• Evaluation of materials, design alternatives, setting component & system tolerances, etc.


Four Eras in the History of DOX• The agricultural origins, 1908 – 1940s

• W.S. Gossett and the t-test (1908)• R. A. Fisher & his co-workers• Profound impact on agricultural science• Factorial designs, ANOVA

• The first industrial era, 1951 – late 1970s• Box & Wilson, response surfaces• Applications in the chemical & process industries

• The second industrial era, late 1970s – 1990• Quality improvement initiatives in many companies• Taguchi and robust parameter design, process

robustness• The modern era, beginning circa 1990


William Sealy Gosset (1876-1937)

Gosset's interest in barley cultivation led him to speculate that design of experiments should aim, not only at improving the average yield, but also at breeding varieties whose yield was insensitive (robust) to variation in soil and climate.

Gosset was a friend of both Karl Pearson and R.A. Fisher, an achievement, for each had a monumental ego and a loathing for the other.

Gosset was a modest man who cut short an admirer with the comment that “Fisher would have discovered it all anyway.”


R. A. Fisher (1890 – 1962) George E. P. Box (1919 – 2013)


The Basic Principles of DOX

• Randomization• Running the trials in an experiment in random order• Notion of balancing out effects of “lurking” variables

• Replication• Sample size (improving precision of effect estimation,

estimation of error or background noise)• Replication versus repeat measurements? (see pages 12, 13)

• Blocking• Dealing with nuisance factors


Strategy of Experimentation

• “Best-guess” experiments• Used a lot• More successful than you might suspect, but there are

disadvantages…

• One-factor-at-a-time (OFAT) experiments• Sometimes associated with the “scientific” or “engineering”

method• Devastated by interaction, also very inefficient

• Statistically designed experiments• Based on Fisher’s factorial concept


Factorial Designs

• In a factorial experiment, allpossible combinations of factor levels are tested

• The golf experiment:• Type of driver• Type of ball• Walking vs. riding• Type of beverage• Time of round• Weather • Type of golf spike• Etc, etc, etc…


Factorial Design (a 22 factorial)


These are least squares estimates – you’ll do them by computer


Factorial Designs with Several Factors



Factorial Designs with Several FactorsA Fractional Factorial


Planning, Conducting & Analyzing an Experiment1. Recognition of & statement of problem2. Choice of factors, levels, and ranges3. Selection of the response variable(s)4. Choice of design5. Conducting the experiment6. Statistical analysis7. Drawing conclusions, recommendations


Planning, Conducting & Analyzing an Experiment

• Get statistical thinking involved early• Your non-statistical knowledge is crucial to success• Pre-experimental planning (steps 1-3) vital• Think and experiment sequentially (use the KISS

principle)• See Coleman & Montgomery (1993) Technometrics paper

+ supplemental text material


Design of Engineering Experiments –The 2k Factorial Design

• Text reference, Chapter 6• Special case of the general factorial design; k

factors, all at two levels• The two levels are usually called low and high

(they could be either quantitative or qualitative)• Very widely used in industrial experimentation• Form a basic “building block” for other very

useful experimental designs (DNA)• Special (short-cut) methods for analysis• We will make use of software



The Simplest Case: The 22

“‐” and “+” denote the low and high levels of a factor, respectively

• Low and high are arbitrary terms

• Geometrically, the four runs form the corners of a square

• Factors can be quantitative or qualitative, although their treatment in the final model will be different


Chemical Process Example

A = reactant concentration, B = catalyst amount, y = recovery


Analysis Procedure for a Factorial Design• Estimate factor effects• Formulatemodel

• With replication, use full model• With an unreplicated design, use normal probability plots

• Statistical testing (ANOVA)• Refine the model• Analyze residuals (graphical)• Interpret results


Estimation of Factor Effects

12

12

12

(1)2 2[ (1)]

(1)2 2[ (1)]

(1)2 2[ (1) ]

A A

n

B B

n

n

A y y

ab a bn nab a b

B y y

ab b an nab b a

ab a bABn n

ab a b

See textbook, pg. 235‐236 for manual calculations

The effect estimates are: A = 8.33, B = ‐5.00, AB = 1.67

Practical interpretation?


Statistical Testing ‐ ANOVA

The F-test for the “model” source is testing the significance of the overall model; that is, is either A, B, or AB or some combination of these effects important?


JMP output, full model


JMP output, reduced model


Residuals and Diagnostic Checking


The Response Surface




Software can perform these calculations. Some JMP output is on the next slide.

Also see: Jones, B. and Montgomery, D.C. (2017), “Partial Replication of Small Two-Level Factorial Designs”, Quality Engineering, Vol. 29, No. 3, pp. 190-195.



The 23 Factorial Design


Effects in The 23 Factorial Design

etc, etc, ...

A A

B B

C C

A y y

B y y

C y y

These are least squares estimates

Analysis done via computer


An Example of a 23 Factorial Design

A = gap, B = Flow, C = Power, y = Etch Rate


Table of – and + Signs for the 23 Factorial Design (pg. 218)


Properties of the Table

• Except for column I, every column has an equal number of + and – signs

• The sum of the product of signs in any two columns is zero

• Multiplying any column by I leaves that column unchanged (identity element)

• The product of any two columns yields a column in the table:

• Orthogonal design

• Orthogonality is an important property shared by all factorial designs

2

A B ABAB BC AB C AC


Estimation of Factor Effects


ANOVA Summary – Full Model


JMP Output for the full model


Refine Model – Remove Nonsignificant Factors


Model Interpretation

Cube plots are often useful visual displays of experimental results


How Much Replication?

Chapter 6 Design & Analysis of Experiments 9E 2017 Montgomery 47

Full factorial model, α = 0.05, and an effect size of two standard deviations

Chapter 6 Design & Analysis of Experiments 9E 2017 Montgomery 48


The General 2k Factorial Design

• Section 6‐4, pg. 253, Table 6‐9, pg. 25• There will be kmain effects, and

two-factor interactions2

three-factor interactions3

1 factor interaction

k

k

k


6.5 Unreplicated 2k Factorial Designs

• These are 2k factorial designs with oneobservation at each corner of the “cube”

• An unreplicated 2k factorial design is also sometimes called a “single replicate” of the 2k

• These designs are very widely used

• Risks…if there is only one observation at each corner, is there a chance of unusual response observations spoiling the results?

• Modeling “noise”?


Spacing of Factor Levels in the Unreplicated 2k Factorial Designs

If the factors are spaced too closely, it increases the chances that the noise will overwhelm the signal in the data

More aggressive spacing is usually best


Unreplicated 2k Factorial Designs

• Lack of replication causes potential problems in statistical testing

• Replication admits an estimate of “pure error” (a better phrase is an internal estimate of error)

• With no replication, fitting the full model results in zero degrees of freedom for error

• Potential solutions to this problem• Pooling high-order interactions to estimate error• Normal probability plotting of effects (Daniels, 1959)• Other methods…Lenth’s method (also see text)


Example of an Unreplicated 2k Design

• A 24 factorial was used to investigate the effects of four factors on the filtration rate of a resin

• The factors are A = temperature, B = pressure, C = mole ratio, D= stirring rate

• Experiment was performed in a pilot plant


The Resin Plant Experiment


The Resin Plant Experiment

The image part with relationship ID rId2 was not found in the file.


Estimates of the Effects

Estimates of the Effects




The Half‐Normal Probability Plot of Effects


Design Projection: ANOVA Summary for the Model as a 23 in Factors A, C, and D



The Regression Model



Model Residuals are Satisfactory


Model Interpretation – Main Effects and 2FI Interactions


Model Interpretation – Response Surface Plots

With concentration at either the low or high level, high temperature and high stirring rate results in high filtration rates



The 2k design and design optimality

The model parameter estimates in a 2k design (and the effect estimates) are least squares estimates. For example, for a 22 design the model is

0 1 1 2 2 12 1 2

0 1 2 12 1

0 1 2 12 2

0 1 2 12 3

0 1 2 12 4

(1) ( 1) ( 1) ( 1)( 1)(1) ( 1) (1)( 1)( 1) (1) ( 1)(1)(1) (1) (1)(1)

(1) 1 1 1 11 1 1 1

, ,1 1 1

y x x x x

ab

ab

abab

y = Xβ + ε y X

0 1

1 2

2 3

12 4

, ,1

1 1 1 1

β ε

The four observations from a 22 design


The least squares estimate of β is

1

0

14

2

12

ˆ

4 0 0 0 (1)0 4 0 0 (1)0 0 4 0 (1)0 0 0 4 (1)

(1)4ˆ (1) (

ˆ (1)1ˆ (1)4

(1)ˆ

a b aba ab bb ab a

a b ab

a b ab

a b ab a ab ba ab bb ab a

a b ab

-1β = (X X) X y

I

1)4

(1)4

(1)4

b ab a

a b ab

The matrix is diagonal –consequences of an orthogonal design

X X

The regression coefficient estimates are exactly half of the ‘usual” effect estimates

The “usual” contrasts


The matrix has interesting and useful properties:X X

2 1

2

ˆ( ) (diagonal element of ( ) )

4

V

X XMinimum possible value for a four-run

design

|( ) | 256 X X Maximum possible value for a four-run

design

Notice that these results depend on both the design that you have chosen and the model

What about predicting the response?


21 2

1 2 1 22

2 2 2 21 2 1 2 1 2

1 22

1 2

1 22

1 2

ˆ[ ( , )][1, , , ]

ˆ[ ( , )] (1 )4

The maximum prediction variance occurs when 1, 1ˆ[ ( , )]

The prediction variance when 0 is

ˆ[ ( , )]

V y x xx x x x

V y x x x x x x

x x

V y x xx x

V y x x

-1x (X X) xx

4What about prediction variance over the design space?average


Average prediction variance1 1

21 2 1 2

1 11 1

2 2 2 2 21 2 1 2 1 2

1 12

1 ˆ[ ( , ) = area of design space = 2 4

1 1 (1 ) 4 4

49

I V y x x dx dx AA

x x x x dx dx




For the 22 and in general the 2k

• The design produces regression model coefficients that have the smallest variances (D‐optimal design)

• The design results in minimizing the maximum variance of the predicted response over the design space (G‐optimal design)

• The design results in minimizing the average variance of the predicted response over the design space (I‐optimaldesign)


Optimal Designs

• These results give us some assurance that these designs are “good” designs in some general ways

• Factorial designs typically share some (most) of these properties

• There are excellent computer routines for finding optimal designs (JMP is outstanding)

DOX Short Course 74

Design of Engineering Experiments The 2k‐p Fractional Factorial Design• Text reference, Chapter 8• Motivation for fractional factorials is obvious; as the number of factors becomes large enough to be “interesting”, the size of the designs grows very quickly

• Emphasis is on factor screening; efficiently identify the factors with large effects

• There may be many variables (often because we don’t know much about the system)

• Almost always run as unreplicated factorials, but often with center points

DOX Short Course 75

DOX Short Course 76

Why do Fractional Factorial Designs Work?

• The sparsity of effects principle• There may be lots of factors, but few are important• System is dominated by main effects, low‐order interactions

• The projection property• Every fractional factorial contains full factorials in fewer factors

• Sequential experimentation• Can add runs to a fractional factorial to resolve difficulties (or ambiguities) in interpretation

DOX Short Course 77

The One‐Half Fraction of the 2k

• Section 8.2, page 321

• Notation: because the design has 2k/2 runs, it’s referred to as a 2k‐1

• Consider a really simple case, the 23‐1

• Note that I =ABC

DOX Short Course 78

The One‐Half Fraction of the 23

For the principal fraction, notice that the contrast for estimating the main effect A is exactly the same as the contrast used for estimating the BCinteraction.

This phenomena is called aliasing and it occurs in all fractional designs

Aliases can be found directly from the columns in the table of + and ‐ signs

DOX Short Course 79

Aliasing in the One-Half Fraction of the 23

A = BC, B = AC, C = AB (or me = 2fi)

Aliases can be found from the defining relation I = ABC by multiplication:

AI = A(ABC) = A2BC = BC

BI =B(ABC) = AC

CI = C(ABC) = AB

Textbook notation for aliased effects:

[ ] , [ ] , [ ]A A BC B B AC C C AB

DOX Short Course 80

The Alternate Fraction of the 23‐1

• I = ‐ABC is the defining relation• Implies slightly different aliases: A = ‐BC, B= ‐AC, and C = ‐AB

• Both designs belong to the same family, defined by

• Suppose that after running the principal fraction, the alternate fraction was also run

• The two groups of runs can be combined to form a full factorial – an example of sequential experimentation

I ABC

DOX Short Course 81

Design Resolution

• Resolution III Designs:• me = 2fi• example

• Resolution IV Designs:• 2fi = 2fi• example

• Resolution V Designs:• 2fi = 3fi• example

3 12III

4 12IV

5 12V

DOX Short Course 82

Construction of a One-half Fraction

The basic design; the design generator

DOX Short Course 83

Projection of Fractional Factorials

Every fractional factorial contains full factorials in fewer factors

The “flashlight” analogy

A one-half fraction will project into a full factorial in any k – 1 of the original factors

DOX Short Course 84

Example 8.1

DOX Short Course 85

Example 8.1Interpretation of results often relies on making some assumptions

Ockham’s razor

Confirmation experiments can be important

Adding the alternate fraction – see page 322

DOX Short Course 86

The AC and AD interactions can be verified by inspection of the cube plot

DOX Short Course 87

Confirmation experiment for this example: see page 332

Use the model to predict the response at a test combination of interest in the design space – not one of the points in the current design.

Run this test combination – then compare predicted and observed.

For Example 8.1, consider the point +, +, ‐, +. The predicted response is

Actual response is 104.

DOX Short Course 88

Possible Strategies for

Follow‐Up Experimentation

Following a Fractional

Factorial Design

DOX Short Course 89

The One‐Quarter Fraction of the 2k

DOX Short Course 90

The One‐Quarter Fraction of the 26‐2

Complete defining relation: I = ABCE = BCDF = ADEF

DOX Short Course 91

The One‐Quarter Fraction of the 26‐2

• Uses of the alternate fractions

• Projection of the design into subsets of the original six variables

• Any subset of the original six variables that is not a word in the complete defining relation will result in a full factorial design

• Consider ABCD (full factorial)• Consider ABCE (replicated half fraction)• Consider ABCF (full factorial)

, E ABC F BCD

DOX Short Course 92

The General 2k‐p Fractional Factorial Design

• Section 8.4, page 340• 2k‐1 = one‐half fraction, 2k‐2 = one‐quarter fraction, 2k‐3= one‐eighth fraction, …, 2k‐p = 1/ 2p fraction

• Add p columns to the basic design; select pindependent generators

• Important to select generators so as to maximizeresolution, see Table 8.14

• Projection – a design of resolution R contains full factorials in any R – 1 of the factors

• Blocking

DOX Short Course 93

DOX Short Course 94

Plackett‐Burman Designs

• These are members of a class of fractional factorials designs called non‐regular designs

• The number of runs, N, need only be a multiple of four and the designs are resolution III

• N = 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, …• The designs where N = 12, 20, 24, etc. are called nongeometric PB designs

DOX Short Course 95


DOX Short Course 96

This is a nonregular design because there is partial aliasing of main effects and two-factor interactions

DOX Short Course 97

Projection of the 12-run design into 3 and 4 factors

All PB designs have projectivity 3 (contrast with other resolution III fractions)

The partial aliasing may allow the estimation of main effects and a few two-factor interactions


Documents

An Introduction to Design of Experiments...One‐Day DOX Course 8 Four Eras in the History of DOX •The agricultural origins, 1908 – 1940s •W.S. Gossett and the t-test (1908)