Simplified Design of Experiments Quality Day Orange Empire 0701 October 28, 2010

Simplified Design ofExperiments

Simplified Design ofExperiments

Quality DayOrange Empire 0701

October 28, 2010

Goals of Today’s Session

• Understand why Design of Experiments is preferable.

• Learn how to perform basic experiments to optimize results and reduce variation.

Approaches to Experimentation

• Build-test-fix (What experimentation?)

• One-factor-at-a-time (OFAT)

• Designed experiments (DOE)

Build-Test=Fix• The tinkerer’s approach• Impossible to know if true

optimum achieved– Quit when it works!

• Consistently slow– Requires intuition, luck, rework– Continual fire-fighting

One Factor at a Time (OFAT)

• Approach:– Run all factors at one condition– Repeat, changing condition of one factor– Continuing to hold that factor at that

condition, rerun with another factor at its second condition

– repeat until all factors at their optimum conditions

• Slow and requires many tests• Can miss interactions!

“One Factor at a Time” is Like

What we conclude may be determined by where we are looking!

The Blind Man and the Elephant

Design of Experiments• A statistics-based approach to

designed experiments.

• A methodology to achieve a predictive knowledge of a complex, multi-variable process with the fewest trials possible.

• An optimization of the experimental process itself.

Key Concepts• DOE is about better understanding of our processes

PROCESS:

A Blending of Inputs which

Generates Corresponding

Outputs

INPUTS(Factors)

X variables

OUTPUTS(Responses)Y variables

People

Materials

Equipment

Policies

Procedures

Methods

Environment

responses related to performing a

service

responses related to producing a

produce

responses related to completing a task

Illustration of a Process

PROCESS:

Manufacturing Injection

Molded Parts

INPUTS(Factors)

X variables


Type of Raw Material

Mold Temperature

Holding Pressure

Holding Time

Gate Size

Screw Speed

Moisture Content

thickness of molded part

% shrinkage from mold size

number of defective parts

Manufacturing Injection Molded Parts

Injection Molding Process

PROCESS:

Discovering Optimal

Concrete Mixture

INPUTS(Factors)

X variables


Type of cement

Percent water

Type of Additives

Percent Additives

Mixing Time

Curing Conditions

% Plasticizer

compressive strength

modulus of elasticity

modulus of rupture

Optimum Concrete Mixture

Poisson's ratio

Concrete Mixing Process

Microwave Popcorn Making Process

PROCESS:

Making the Best

Microwave popcorn

INPUTS(Factors)

X variables


Brand:Cheap vs Costly

Time:4 min vs 6 min

Power:75% or 100%

Height:On bottom or raised

Taste:Scale of 1 to 10

Bullets:Grams of unpopped

corns

Making microwave popcorn

Three Key Principles• Replication [DOE’s version of Sample Size]

– Replication of an experiment– Allows an estimate of experimental error– Allows for a more precise estimate of the sample

mean value• Randomization [Run experiments in random order]

– Cornerstone of all statistical methods– “Average out” effects of extraneous factors– Reduce bias and systematic errors

• Blocking [What can influence my experiment?]– Increases precision of experiment– “Factor out” variables not studied

Steps in a Designed Experiments Study

1. Brainstorm the problem / causes2. Design the Experiment3. Perform the experiment4. Analyze the data5. Validate your results

FACTORIAL EXPERIMENTIMPROVEMENT OF THE MEAN

Determining Input Factors to Optimize Output

Case Study• You have been assigned to look

into a problem with your company’s lamination process. Customers have been complaining about separation in your wood laminate, and you are leading a team to determine how to reduce the separation.

Step 1. Brainstorm• You met with your team, and

through your investigation, you determine that the best approach would be to minimize the amount of CURL

Curl

Laminate

Step 1. Brainstorm• After brainstorming and list

reduction, your team decided to look at 3 factors that they felt influenced the amount of curl:A Top Roll Tension, currently set at 22B Bottom Roll Tension, currently set at 22C Rewind Tension, currently set at 9

Step 2. Design the Experiment

The team decides to study 2 levels for each of the factors, and determine the effect on the response variable, curl:

Factor Low (-) Setting

High (+) Setting

Top Roll Tension 16 28

Bottom Roll Tension 16 28

Rewind tension 6 12

Note: Team will make sure that their selected values are FEASIBLE.


• This experiment will have 2 levels (high and low settings) and 3 factors (top roll tension, bottom roll tension, and rewind tension)

• Number of Experiments = 23 = 8


Run Top Roll Tension

Bottom Roll Tension

Rewind Tension

1 - - -2 + - -3 - + -4 + + -5 - - +6 + - +7 - + +8 + + +

Alternate -,+ Alternate - -,+ +

Alternate - - - - + + + +

Note: If another variable is added, there would be 24=16 runsThe next design generation would be “- - - - - - - -, + + + + + + +”

Step 3. Perform the ExperimentRun

Top Roll Tension

Bottom Roll

Tension

Rewind Tension

Replication 1

Replication 2 Average Std.

Deviation

1 16 16 6 87 88 87.5 0.707

2 28 16 6 76 78 77.0 1.414

3 16 28 6 90 92 91.0 1.414

4 28 28 6 83 80 81.5 2.121

5 16 16 12 101 96 98.5 3.535

6 28 16 12 92 91 91.5 0.707

7 16 28 12 100 104 102.0 2.828

8 28 28 12 92 91 91.5 0.707

Perform the runs in random order to ensure statistical validity

Step 4. Analyze the Data

• We are going to calculate the MAIN EFFECTS for the 3 factors.

• We are also going to calculate the INTERACTION EFFECTS between each of the 2 factor combinations and the three factor combination.

• Let’s start with the main effects…

Step 4. Analyze the Data Visualization of Main Effects

Top Roll Tension

Bottom Roll Tension

Rewind Tension

- +-

-

+

+

91.5

87.5

77.0

91.0

81.5

98.5

91.5

102.0

Note: lowest curl achieved when Top Roll Tension HIGH and Bottom Roll and Rewind Tensions set LOW

Step 4 Analyze the DataMain Effect Calculations

• Average the “High” Settings and subtract the average of the “Low” Settings:

77.0 81.5 91.5 91.5 87.5 91.0 98.5 102.0Main Effect of Top Roll Tension= 9.38

4 4

91.0 81.5 102.0 91.5 87.5 77.0 98.5 91.5

Main Effect of Bottom Roll Tension= 2.884 4

98.5 91.5 102.0 91.5 87.5 77.0 91.0 81.5Main Effect of Rewind Tension= 11.63

4 4

Step 4. Analyze the DataInteraction Effects

Run

Top Roll Tension

(A)

Bottom Roll

Tension (B)

Rewind

Tension (C)

AB AC BC ABC Avg Curl

1 - - - + + + - 87.5

2 + - - - - + + 77.0

3 - + - - + - + 91.0

4 + + - + - - - 81.5

5 - - + + - - + 98.5

6 + - + - + - - 91.5

7 - + + - - + - 102.0

8 + + + + + + + 91.5

Simply multiply the signs of the columns, i.e. “+ times + equals +” “- times - equals +” “+ times - equals -” and “- times + equals -”

Step 4 Analyze the DataInteraction Effect Calculations

• Average the “High” Settings and subtract the average of the “Low” Settings:

87.5 81.5 98.5 91.5 77.0 91.0 91.5 102.0Interaction AB= 0.63

4 4

87.5 91.0 91.5 91.5 77.0 81.5 98.5 102.0Interaction AC= 0.63

4 4

87.5 77.0 102.0 91.5 91.0 81.5 98.5 91.5Interaction BC= 1.13

4 4

77.0 91.0 98.5 91.5 87.5 81.5 91.5 102.0Interaction ABC= 1.13

4 4

Step 4 Analyze the DataPareto of Effect Size

Rewind

Tens

ion

(C)

Top

Roll T

ensio

n (A

)

Botto

m R

oll T

ensio

n (B

)

BC Inte

ract

ion

3-W

ay In

tera

ction

AB Inte

ract

ion

Inte

ract

ion

0

4

8

12

Effect Size

Effect Size

Pareto shows the absolute value of the effects for comparability of significance.

Step 4. Analyze the Data Visualization of Interaction Effects

Top Roll Tension Low

Top Roll Tension High

75

80

85

90

95

100

Interaction – Top and Bottom Roll Tension

Bottom Roll LowBottom Roll High

Cu

rl Slope= -10.0

Slope= -8.75

Note: parallel lines indicate LACK OF INTERACTION


Top Roll Tension Low

Top Roll Tension High

020406080

100120

Interaction – Top Roll and Rewind Tension

Rewind LowRewind High

Cu

rl

Slope= -8.75

Slope=-10.0



Bottom Roll Tension Low

Bottom Roll Tension High

75

80

85

90

95

100

Interaction –Bottom Roll and Rewind Tension

Rewind LowRewind High

Cu

rl Slope = 1.75

Slope = 4.0


Step 4. Analyze the DataTest of Significance

• We can test for the significance of the effects using the t-distribution and confidence intervals.

• The first step is to calculate the POOLED STANDARD DEVIATION for all of the observations….

Step 4 Analyze the DataPooled Standard

Deviation

We are studying 7 factors/ interactions (A, B, C, AB, AC, BC, ABC) in 8 runs

We did 2 replications of each run.2 2 2

1 1 2 2

1 2

... for v runs

...v v

pv

n S n S n SS

n n n

Note: if the number of replicates is the same for all runs, we can simply calculate the pooled standard deviation as the square root of the average of the variances

Step 4 Analyze the DataPooled Standard

Deviation

Run Replication 1

Replication 2

Average Std.Deviatio

nVarianc

e

1 87 88 87.5 0.707 0.4998

2 76 78 77.0 1.414 1.9994

3 90 92 91.0 1.414 1.9994

4 83 80 81.5 2.121 4.4986

5 101 96 98.5 3.535 12.4962

6 92 91 91.5 0.707 0.4998

7 100 104 102.0 2.828 7.9976

8 92 91 91.5 0.707 0.4998

Avg Variance=3.8113Std Dev = 1.952

Simplified Calculation:

Step 4. Analyze the Datat Statistic for Significance

• Key Information for calculation:=risk (Confidence = 1- ) [5%, 95%]p=number of “+” per effect column [4]r=number of replicates [2]Sp=Pooled Standard Deviation [1.952]

f=degree of fractionalization (in our case 0, since we are doing a full factorial)

k=number of factors [3]Degrees of Freedom=(r-1)2k-f [8]


The t statistic for 95% confidence (5% risk), 2 tailed test, 8 degrees of freedom is 2.306

The Error is calculated as:

The confidence interval for each of the effects is:

– Effect +/- Error

t table for 2 tailed test

.025,8

2 22.306 1.952 2.25

(4)(2)pError t Spr


• Since the Error = 2.25, any effect that is contained in the limits of: 0 +/- 2.25

• Is considered NOT STATISTICALLY SIGNIFICANT

Factor Effect

Rewind Tension m(C) 11.63

Top Roll Tension (A) -9.38

Bottom Roll Tension (B) 2.88

BC Interaction -1.13

ABC Interaction -1.13

AB Interaction -0.63

AC Interaction 0.63

StatisticallySignificant

NotStatisticallySignificant

Step 4. Analyze the DataConclusion

• Based on this outcome, I will conclude that Top Roll tension, Bottom Roll Tension, and Rewind Tension are significant at the 95% significance level

• I will conclude that I should set:– Top Roll Tension HIGH (28)– Bottom Roll Tension LOW (16)– Rewind Tension LOW (6)

Step 4. Analyze the DataConclusion

Run

Top Roll Tension

Bottom Roll

Tension

Rewind Tension

Replication 1

Replication 2 Average Std.

Deviation

1 16 16 6 87 88 87.5 0.707

2 28 16 6 76 78 77.0 1.414

3 16 28 6 90 92 91.0 1.414

4 28 28 6 83 80 81.5 2.121

5 16 16 12 101 96 98.5 3.535

6 28 16 12 92 91 91.5 0.707

7 16 28 12 100 104 102.0 2.828

8 28 28 12 92 91 91.5 0.707

Grand Average=90.0625

Analysis of Data - Response Model

• The factor effects can be used to establish a model to predict responses.

EffectA EffectB EffectCExpected Response=Grand Average+

2 2 2

EffectAB EffectAC EffectBC EffectABC

2 2 2 2

where A=setting for factor A(-1)

where B=settin

A B C

AB AC BC ABC

g for factor B(+1)

where C=setting for factor C(+1)

Calculating Expected Response

So, if we set A at low (-1), B high (+1) and C High (+1), we would predict: 90.0625 - 4.69 - 1.44 - 5.815 + 0.315 - 0.315 - 0.565 - 0.565 = 76.99

Factor Effect Effect/2 Setting Value

Grand Average 90.0625

Top Roll Tension (A) -9.38 -4.69 1 -4.69

Bottom Roll Tension (B) 2.88 1.44 -1 -1.44

Rewind Tension m(C) 11.63 5.815 -1 -5.815

AB Interaction -0.63 -0.315 (1)(-1) 0.315

AC Interaction 0.63 0.315 (1)(-1) -0.315

BC Interaction -1.13 -0.565 (-1)(-1) -0.565

ABC Interaction -1.13 -0.565 (1)(-1)(-1) -0.565

Step 5. Validation of Model

• Run the process with the new settings for a trial period to collect data on the curl response.

• Compare the new data to the historical data to confirm improvement.

• A simple Test of Hypothesis of before and after data with a t test can be used.


• We are trying to prove that the curl with the new process settings is significantly less than the curl with the old process settings at a 95% level of significance. Our historical standard deviation has been 3.5.

–Ho: µold µnew

–Ha: µold > µnew

43

Commonly Used Z-Values

1-α (or 1-β) α (or β) Z Value

.995 .005 2.575

.990 .010 2.387

.975 .025 1.960

.950 .050 1.645

.900 .100 1.282

.800 .200 0.842


• Sample Size Required:

• = 5% = 10%• = 3.4 Change to detect =

2

2 2 2 2

2 2

(1.645 1.282) 3.424.75 25

2

Z Zn

Step 5 Validation of Model

• Suppose our historical data (25 data points) is as follows for curl:– 100, 95, 98, 102, 97, 90, 91, 98, 101,

95– 94, 105, 96, 104, 100, 96, 98, 96, 91,

99– 100, 102, 98, 95, 96

• Average = 97.48• Sample Standard Deviation =

3.8419

Step 5 Validation of Model

• Now, we run our new settings and collect 25 additional sets of data:– 86, 77, 85, 81, 81, 83, 78, 79, 81, 80– 81, 78, 76, 75, 79, 83, 85, 81, 78, 78– 85, 81, 79, 83, 72

• Average = 80.20• Sample Standard Deviation =

3.391

t test – Assuming Equal Variances in ExcelOld New t-Test: Two-Sample Assuming Equal Variances100 86

95 77 Old New98 85 Mean 97.48 80.2

102 81 Variance 14.76 11.597 81 Observations 25 2590 83 Pooled Variance 13.13

91 78 Hypothesized Mean Difference 098 79 df 48

101 81 t Stat 16.8603420795 80 P(T<=t) one-tail 4.21133E-2294 81 t Critical one-tail 1.677224197

105 78 P(T<=t) two-tail 8.42266E-2296 76 t Critical two-tail 2.010634722

104 75100 79

96 8398 8596 8191 7899 78

100 85102 81

98 7995 8396 72

t critical (24 degrees of freedom) = 1.677Calculated t value = 16.86

We can conclude with more than 95% confidence that the new parameter settings have significantly reduced the amount of curl in our process.

FACTORIAL EXPERIMENTREDUCTION OF VARIATION

Determining Input Factors to Reduce Variation

Variation Reduction• Standard deviations are not normally

distributed, and therefore cannot be used directly as a response variable.

• Options include:– Use the natural or base 10 log to obtain

normality– Use –log10(s) for normality– Use the F Statistic on the average

variances for the high and low settings

Data – Curl ReductionRun Top

Roll Tension (A)

Bottom Roll Tensio

n (B)

Rewind

Tension (C)

AB AC BC ABC Avg Curl Std Dev Variance

1 - - - + + + - 87.5 0.707 .004998

2 + - - - - + + 77.0 1.414 1.9994

3 - + - - + - + 91.0 1.414 1.9994

4 + + - + - - - 81.5 2.121 4.4986

5 - - + + - - + 98.5 3.535 12.4962

6 + - + - + - - 91.5 0.707 004998

7 - + + - - + - 102.0 2.828 7.9976

8 + + + + + + + 91.5 0.707 004998

Calculate an F Value for each Effect

2

2

1.994 4.4986 .004998 .0049981.6256

4.004998 1.9994 12.4962 7.9976

5.62454

S

S

For Top Roll Tension

2arg

2

3 0

.025,4,4

5.62453.460

1.6256

( 1)2 1 2Degrees of Freedom= 4

2 2Confidence 95%, Risk 5%

F 9.60

L er

Smaller

k f

SF

S

r x

r=# replicates (2)k=# factors (3)f=fractionalization (0)

Any effects with a F value greater than 9.60 are significant at the 95% level.

Significant for reduced variation???

F Table for = .025


2

2

1.9994 4.4986 7.9976 .0049983.625

4.004998 1.9994 12.4962 .004998

3.6264

S

S

For Bottom Roll Tension

2arg

2

3 0

.025,4,4

3.6261.000

3.625



F 9.60

L er

Smaller

k f

SF

S

r x





2

2

12.4962 .004998 7.9976 .0049985.126

4.004998 1.9994 1.9994 4.4986

2.12564

S

S

For Rewind Tension

2arg

2

3 0

.025,4,4

5.1262.412

2.1256



F 9.60

L er

Smaller

k f

SF

S

r x





2

2

.004998 4.4986 12.4962 .0049984.2512

41.9994 1.9994 .004998 7.9976

3.00034

S

S

For Interaction of Top Roll and Bottom Roll Tension

2arg

2

3 0

.025,4,4

4.25121.4169

3.0003



F 9.60

L er

Smaller

k f

SF

S

r x





2

2

.004998 1.9994 .004998 .0049980.5036

41.9994 4.4986 12.4962 7.9976

6.747954

S

S

For Interaction of Top Roll and Rewind Tension

2arg

2

3 0

.025,4,4

6.7479513.3994

0.5036



F 9.60

L er

Smaller

k f

SF

S

r x





2

2

.004998 1.9994 7.9976 .0049982.5017

41.9994 4.4986 12.4962 .004998

4.74984

S

S

For Interaction of Bottom Roll and Rewind Tension

2arg

2

3 0

.025,4,4

4.74981.8986

2.5017



F 9.60

L er

Smaller

k f

SF

S

r x





2

2

1.9994 1.9994 12.4962 .0049984.1250

4.004998 4.4986 .004998 7.9976

3.12654

S

S

For Interaction of All Factors

2arg

2

3 0

.025,4,4

4.12501.31937

3.1265



F 9.60

L er

Smaller

k f

SF

S

r x




Lets Summarize…

Factor F Value

Top Roll Tension (A) 3.460

Bottom Roll Tension (B) 1.0000

Rewind Tension (C) 2.412

AB Interaction 1.4169

AC Interaction 13.3994

BC Interaction 1.8986

ABC Interaction 1.31937

Note, the AC (Top Roll-Rewind Tension) is the only factor or interaction that exceeds the F Critical Value of 9.60.

Conclusion• To minimize variation, we find that variation is

minimized when Top Roll Tension and Rewind Tension are set to the same levels (i.e. High-High or Low-Low).

• Our optimal settings to minimize curl were Top Roll High, Bottom Roll Low, and Rewind Low. Top Roll and Rewind influenced the curl in OPPOSITE DIRECTIONS!

How can we decide?

Case StudiesInstructions:Work in teams to read the case study and then develop an experimental design. You should include:

• What factors to study• How many and what levels to study• How many replicates to take• How many runs?• Design your experiment.

Case Study 1• You have been working late recently, and

subsisting on microwave popcorn. As a result, you have decided to find the formula for the best popcorn. You are down to two brands, A and B. You also find that the time varies between 4 and 6 minutes and power between 75% and 100%.

• You judge the quality of the popcorn by taste and the number of unpopped kernels.

Case 1 Design• What factors to study• How many and what levels to study• How many replicates to take• How many runs?• Design your experiment


Case Study 2• Your company has a high-pressure chemical reactor which

filters impurities from your product. You have been asked to determine the proper settings to maximize the filtration rate. After a brainstorming session, your team decides that temperature, pressure, chemical concentration % and stir rate all influence the filtration rate (gallons per hour).

• Looking over the historical records, you see temperature has varied from 24oC to 35oC. Pressure can range from 10 PSIG to 15 PSIG, and chemical concentration from 2% to 4%. The lowest stir rate has been 15 RPM and the highest 30 RPM.

• Design an experiment to determine what the optimal levels would be to maximize the filtration rate.



Case Study 3• A manufacturer of ice crease has hired you to assist them

with achieving their target fill rate of 2.50 pounds. After discussing with them the relevant factors, it is concluded that there are two main variables impacting the final weight, fill temperature and the overfill %. Overfill is the percentage of air that is incorporated into the ice crease. The fill temperature is sensitive, and they have machines that vary from 20oF to 25oF. Overfill percentage is also tightly controlled. Studies have shown that overfill percentages greater than 120% are perceived as “lower quality” by customers and percentages less than 90% cause the ice crease to be difficult to scoop.



Points to Remember1. Get a clear understanding of the problem you

intend to solve.2. Conduct an exhaustive and detailed

brainstorming session.3. Teamwork – involve people involved in all

aspects of the process being studied.4. Randomize the experiment trial order5. Replicate to understand and estimate

variation.6. Perform confirmatory runs and experiments to

test your model’s validity.

Your Assignment• Find a problem and run a Designed

Experiment using the methods we learned today within 1-2 weeks of completing this session.

• No problem – Optimize microwave popcorn or chocolate chip cookies!

Questions?

Documents

Simplified Design of Experiments Quality Day Orange Empire 0701 October 28, 2010