Attack the Variance.hurst-1

7/29/2019 Attack the Variance.hurst-1

1/126

Slide-Attack the Variance, Course 1 (Pittcon 2013) 1

Attack the Variance, Course 1Tools to Understand Variance in Analytical

Methods

Olivier Guise

Ph.D. Chemist

Roger Hurst

Ph.D. Chemist

Pittcon Short CourseMarch 19, 2013


2/126


Section 1 Understanding your data

Evaluating data quality are there potential problems? Concepts: Mean, standard deviation, precision, accuracy, Z-score Tools: Hypothesis tests, p-values, Analysis Of Variance (ANOVA), t-

tests, outlier tests, regression analysis

Section 2 How good is the ruler?

Where is the variation sample or measurement or both? Concepts: measured vs. observed variation, effect of interactions,how to correct variation

Tools: Gage Repeatability and Reproducibility (GRR), fishbonediagram (cause and effect diagram)

Section 3 Identify sources of variation . . . Then optimize! Which factors are really important to optimize? Concepts: Knowing which knobs to turn, which way to turn them,

and how much in order to reach the optimal method. Tools: Screening Design of Experiment(DOE) & optimization DOE

Pittcon March 2013

Course Overview


3/126


Your Names & Backgrounds


4/126


Students Course Expectations


5/126


Be involved!

Think constantly about howyou are going to use thiswhen you get back

Ask lots of questions!

Participate in the exercises they are designed to makesure you go home with morethan a notebook!

Cell phone courtesy

Bring up your casestudies that you would likehelp with, from us or fromthe group

If you get ahead, pleasehelp your neighbor. Weneed to stay together toget through as muchmaterial as possible, asquickly as possible.

Instructors Course Expectations


6/126


Pittcon March 2013

Section 1

Understanding Your Data

SABIC Innovative Plastics

Analytical Technology

Evaluating data quality are there potential problems? Concepts: Mean, standard deviation, precision, accuracy, Z-score Tools: Hypothesis tests, p-values, Analysis of Variance (ANOVA),

t-tests, outlier tests, regression analysis


7/126Slide-Attack the Variance, Course 1 (Pittcon 2013) 7

21

1

2

1

)(

n

xx

s

n

i

iMean

1 2 3-1-2-3 1 68.20 % of all data 2 95.54 % of all data 3 99.7% of all data

Process Variability -multiple results fromthe same process

NormalDistribution

(Gaussian)

-4-5-6 4 5 6

6 99.99999980% of all data

Standard Deviation



Accuratenot precise

Addressvariation

issues

Target AnalogyCorrective Action:

Precise,Not accurate

Calibration tocorrect bias

True value

Precise andaccurate

Ideal

Precision vs. Accuracy

True value

True value



PoorProcess

Capability

MeasurementProcess

LSL USL

Target

LSL USL

Target

ExcellentProcess

Capability

LSL USL

Target

So HOW good is good enough??

MeasurementProcess

MeasurementProcess



MeanProcessVariability -

multiple resultsfrom the same

process

When you have aspecified tolerance:

If the limits fall on thevertical lines, the Z

score is . . .

1 2 3-1-2-3 -4-5-6 4 5 6

Z = 1

Z = 2

Z = 3

Z = 4

Higher Z isbetter!

Z Score: A Measure of Variance Relative to Tolerance



Z =USL - x

sobserved

Z =12.0 11.0

0.4

= 2.5

This methodis 2.5 Sigma.

Z = 12.0 7.02.0

= 2.5


LSL USL

x =7.0sobs = 0.4

Z =12.0 7.0

0.4

= 12.5


(Use spec limit nearer to themean . . . may be LSL)

Process

LSL USL

Target

x =11.0sobs = 0.4

12.02.0

7.0

LSL USL

x =7.0sobs = 2.0

Is it Good Enough? . . . Estimate Z Score



Minitab Graphical Summary:A good way to get an overall feel for your data



Minitab Descriptive Statistics:A good way to get an overall feel for your data

54.052.551.049.548.046.545.0

Median

Mean

50.049.549.048.548.047.547.0

1st Quartile 46.958

Median 47.930

3rd Quartile 49.852

Maximum 53.570

46.878 50.238

46.905 49.972

1.616 4.288

A -Squared 0.31

P-V alue 0.496

Mean 48.558

StDev 2.349

V ariance 5.517

S kewness 1.00719

Kurtosis 1.26643

N 10Minimum 45.390

A nderson-Darling Normality Test

95% C onfidence Interval for Mean

95% C onfidence Interv al for Median

95% C onfidence Interval for StDev

95% Confidence Intervals

Summary for Masses



Ho = the null hypothesis (no difference)

Ha = the alternative hypothesis (difference)

True State Test Conclusion

Not Guilty Guilty

Not Guilty Acquit Send to Jail

Guilty Acquit Send to Jail

True State Test Conclusion

In Spec Out of Spec

In Spec Pass Fail

Out of Spec Pass Fail

Which Error is worse?

consider the liability

Compare samplemean and

population of data,two sample means,two sample std.deviations, etc.

Hypothesis Testing



Hypothesis Testing (t-test used to compare means):

Ho: Xbar-A= Xbar-B Ho: Xbar-B = Xbar-C Ho: Xbar-A = Xbar-C

Ha: Xbar-A Xbar-B Ha: Xbar-B Xbar-C Ha: Xbar-A Xbar-C

Stech #1= 3 Stech#2= 1

How do we determine if these are different?

Technique #2Technique #1

A B CA B C



t-distribution (Student t)

Degrees of

Freedom (df)

0.10 0.05 0.005

1 3.078 6.314 63.657

5 1.476 2.015 4.032

10 1.372 1.812 3.169

tcritical

= 0.05

Step 1. Find tcritical

df = n-1

Where nis # ofsamples

T-critical: Determining if two means are different



t-distribution (Student t)

tcriticaln

s

xtcalc

error

If tcalc > tcritical reject Ho, accept Ha

There is a difference between means

Ho: Xbar-A= Xbar-B Ha: Xbar-A Xbar-B

Step 2. Determine tcalc

tcalc

BUTwe cannot run t-tests every time we doan analysis (too many factors!!!), SO.

T-Calculated: Determining if two means are different



Hypothesis Testing

Breakout #1A

Data File: T-test F-test.xls



Instructions:

1. Open the file, T-test F-test.xls

2. Copy and Paste into Minitab.

3. Stack Columns to make one column.

4. Determine if there is a difference in mean value of the groups

First, test for normality

Use Homogeneity of Variance to compare the variance

Use t-test to compare the means

Chemicoolest, Inc. has developed a new mass flow-meter for

industrial process control. Two operators have run experimentsunder the same settings to test the system and operability. The mass

data are stored in the data file, T-test F-test.xls. Use Minitab to

asses the normality, variance, and mean response for this

experiment.

Breakout #1A



Stack columns

Breakout #1A Screen Shots


21/126


Check Normality for Each Operator

(or Subset) Separately



22/126


(Normal if p > 0.05)

Accept Null Hypothesis: Normal



23/126


Test for Homogeneity of Variance



24/126


Reject Null Hypothesis: Unequal Variances

F-test is fornormal data

Levenes test is

for non-normaldata



25/126


2-sample t-test compares themeans of 2 distributions

Do not assume equal variance.F-test showed unequal variance.



26/126


2-sample t-test compares themeans of 2 distributions


Accept Null Hypothesis: Means Appear Equal


27/126


Hypothesis Testing

Breakout #1B

Data File: Hypothesis Testing Class Example.xls


28/126


Instructions:

1. Open the file, Breakout #1B Class Example.xls

2. Copy and Paste into Minitab.

3. Stack Columns to make one column.

4. Determine if there is a difference in mean value of the groups First, test for normality

Use Homogeneity of Variance to compare the variance

Use t-test to compare the means

You just developed a new FTIR method for measuring co-polymer

composition. You and another colleague decide to gauge the methodby running some replicates and comparing the results. The mass

data are stored in the data file, Breakout #1B Class Example.xls.

Use Minitab to asses the normality, variance, and mean response for

this experiment.

Breakout #1B


29/126


Breakout #1B Screen Shots


30/126


Need to investigate more . . .

p > 0.05, normal p < 0.05, Uh-oh



31/126


All data are normal when

analyzed together.

What can cause this?

p > 0.05, normal

All data, bothoperators

Descriptive statistics shows

a possible outlier.

What would you do?


54.052.551.049.548.046.545.0

Median

Mean

50.049.549.048.548.047.547.0

1st Quart il e 46.958

Median 47.930

3rd Quart ile 49.852

M aximum 53.570

46.878 50.238

46.905 49.972

1.616 4.288

A -Squared 0.31

P-V alue 0.496

Mean 48.558

StDev 2.349

V ariance 5.517

S ke wne ss 1. 00719

K urtosis 1.26643

N 10

Minimum 45.390

A nderson-Darling Norm ality Test

95% C onfidence Interval for Mean

95% C onfidence Interval for Median

95% C onfidence Interval for StDev

95% Confidence Intervals

Summary for Masses


32/126


Looking at Levenes Test (since

we had one non-normal dataset), we see that the variances

are not statistically different.


Operator B

Operator A

108642

Operators

95% Bonferroni Confidence Intervals for StDevs

Operator B

Operator A

54.052.551.049.548.046.545.0

Operators

Masses

Test S tatistic 0.71

P-Value 0.746

Test S tatistic 0.06

P-Value 0.818

F-Test

Levene's Test

Test for Equal Variances for Masses

Given the data, we could

accept the null

hypothesis if the data

were normal. . . . Howcan we be more

certain??


33/126


Multiple measurements (n>3) were taken on a given sample. One appearsto deviate wildly, hence could significantly influence the mean, stdev, and

distribution. Can I remove this point with 90% confidence?

30.38

30.23

30.34

29.98

30.29 30.31

29.7

29.8

29.9

30

30.1

30.2

30.330.4

30.5

1 2 3 4 5 6

Measurement No.

Content(%)

Apply a Grubbs Test:

t.v. =|29.98 30.26|

0.1438= 1.947

Compare t.v. to test Grubbs table

t.v. =(Grubbs

test value)

|xi x|

stdev

Outlier Tests When can I throw out a bad result?


34/126


In this case, n = 6 for a one-

tailed distribution.

G(6, 90) = 1.729

Since 1.947 > 1.729

the measurement is anoutlier.

P(one-tailed)

P(two-tailed)

Grubbs Table


35/126


Words of caution with outlier tests:

Apply to multiple measurements of the same

sample, not single measurements of a set of

samples.

First, try to identify and resolve the cause of the

outlier. If you have an outlier and need to report a

single value to represent the result, use the

median value, rather than the mean.

Qobs =Suspect value nearest value

Range

If Qobs > Qcrit, then you

can reject it.

Q-Test Another type of outlier test


36/126


Example:

A set of samples were made

spanning a range of different

compositions.

Each sample was measuredonce.

The measured values were

plotted vs. the theoretical

values (formulation)

Is this due to measurement variation or a bad sample?

Can it be thrown out? . . .

Use a standardized (studentized) residual test.

Validation Set: Measured vs.

Theoretical

y = 0.9942x + 0.1922

R2

= 0.9468

18.0

18.5

19.0

19.520.0

20.5

21.0

21.5

22.0

18.0 19.0 20.0 21.0 22.0

Theoretical Content (%)

MeasuredC

ontent(%)

|yactual yfit|

= residual

Standardized Residuals How to treat outliers in aset of regression data


37/126


Strategy:

Use Minitab to plot the

standardized residuals.

Any residual > 3 is outside the

99.7% confidence interval and

can be considered an outlier.

Investigate and resolve the

reason for the outlier if

necessary.

Class Exercise:

standardized residuals.mpj

Standardized Residuals contd


38/126


2624222018161412108642

4

3

2

1

0

-1

-2

Observation Order

StandardizedResidual

Versus Order(response is Measured)

well over 3 . . .outlier

Tip - Its still recommended to resolve why the sample wasbad than to simply reject it and forget about it.

Standardized Residuals contd


39/126


This tool will determine if the response means associated with the groups arefrom the same population. (Used when data sets vary only in one factor).

Ho: m1 = m2 = m3 = m4

Ha: At least one m different from the others

If p > 0.05 then accept Ho

If p < 0.05 then

One of the means is different than the others.

One-Way ANOVA: Used for Single-Factor Studies


40/126


Grand mean

diameter

Measuremultiple

diameters for

each circle.(Distributionresults from

measurementerror may be

instrument and/or

technique.)

The differenceWITHIN each circleis small relative to

the differenceBETWEEN circles.

. . . So one circle isa differentdiameter.

This doesnt require

ANOVA . . . But whatif we add variation to

the groups?

Example: A single measurement (diameter) and a single factor (circle #).



41/126


Measuremultiple

diameters foreach shape.

(Distribution ofresults comes

frommeasurement

error and actual

changes indiameter.)

Grand mean

diameter

Now it

helps tohaveANOVA



42/126


One-way Analysis of Variance

Source DF SS MS F P

Variety 5 2.43507 0.48701 56.22 0.000

Error 18 0.15593 0.00866

Total 23 2.59100

Tabular Evaluation of 1-way ANOVA OutputAccept Ha: Factor Is Significant

TSS = SSW + SSBTotal Sum of Squared Differences

Sum of Squares Within Group

Sum of Squares Between Group

One-Way ANOVA: Interpretation of Minitab Results

ANOVA: a method ofT W ANOVA G hi ll


43/126


Factor 2

Sample 2

SSB1 = Sabc (for all levels of factor 1)SSB2 = the same for all levels of factor 2

SSE is the total sum of squares error

Fact

or1

Oper-ator 1

2,1

1,21,1

2,2

3,1 3,2

TSS=SSB1+SSB2+SSE

Ybar overall mean

ANOVA: a method ofdetermining the differences inmeans based on the variationin measuring those means.Used when 2 factors are varied:

Such as operator and sample

Oper-ator 2

Sample 1

Sample 2

Sample 3

Sample 3

Avg of 3,1 & 3,2

c

Sample 1

Avg of 1,1 & 1,2a

Avg of 2,1 & 2,2

b

Two-Way ANOVA: Graphically


44/126


ANOVA

Breakout #2

Data Files:

One-Way ANOVA.xls

Two-Way ANOVA.xls


45/126


Breakout 2A: One-Way ANOVA

Open data tab 2A in Breakouts_1.xls

Copy/paste from Excel to Minitab

Stack Columns into one column

Analyze One-Way ANOVA

What can you conclude about the means?

Breakout 2B: Two-Way ANOVA

Open data tab 2B in Breakouts_1.xls

Copy/paste from Excel to Minitab

Stack Columns into one column

Analyze Two-Way ANOVA

Are the means the same in terms of Operator? In terms ofSample?

Breakout #2: One-Way ANOVA and Two-WAY ANOVA


46/126


1. Copy/paste from Excel2. Analyze ANOVA

3. Choose Plots

Breakout #2A One-Way ANOVA Screen Shots

Breakout #2A: One Way ANOVA Screen Shots


47/126


Mean Square (MS)

(Sum of Squares/df)

F value

(MSFactor/MSError)

p is probability of an

alpha error; NOTsignificant if p>0.05

Reject NullHypothesis:

Unequal Means

Breakout #2A: One-Way ANOVA Screen Shots

Operator 3Operator 2Operator 1

34.850

34.825

34.800

34.775

34.750

Data

Boxplot of Operator 1, Operator 2, Operator 3

Breakout #2B: Two Way ANOVA Screen Shots


48/126


1. Copy/paste from Excel

2. Stack Columns into one column

3. Analyze ANOVA

Breakout #2B: Two-Way ANOVA Screen Shots

Breakout #2B: Two Way ANOVA Screen Shots


49/126


Samples are differentOperators are the same

Breakout #2B: Two-Way ANOVA Screen Shots

Operators

sample

Percent Acid Operator 2Percent Acid Operator 1

43214321

2.00

1.75

1.50

1.25

1.00

0.75

0.50

Data

Individual Value Plot of Data vs Operators, sample


50/126


Pittcon March 2013

Section 2

How good is the ruler?SABIC Innovative Plastics


Where is the variation sample or measurement or both? Concepts: measured vs. observed variation, effect of

interactions, how to correct variation Tools: Gage Repeatability and Reproducibility (GRR),

fishbone diagram (cause and effect diagram)


51/126


Inputs

ObservationsMeasurementsDataProcess Outputs Measure-ment

ProcessInputs Outputs

2

tMeasuremen

2

Process

2

Total

Possible Sources of Variation

Long-term

ProcessVariation

Actual Process Variation Measurement Variation

Observed Process Variation

Short-term

ProcessVariation

Variation dueto instrument

Variationdue to

operator

Accuracy(Bias)

Precision(Measurement

Error)


52/126


ProcessVariability

Does Sample A = B = C??

A B C

Technique #2Technique #1

sprocess= 5

stech #1= 3 stech#2= 1

What about the measurement system?

A B C A B C



53/126


How do I know where the variability is originating?

2

ilityReproducib

2

ityRepeatabil

2

tMeasuremen

2

tMeasuremen

2

Process

2

Observed


Long-term

ProcessVariation

Actual Process Variation Measurement Variation

Observed Process Variation

Short-term

ProcessVariation

Variation dueto instrument

Variationdue to

operator

Accuracy(Bias)

Precision(Measurement

Error)


54/126


GR&RMeasurement System

Interpretation

GR&R 10% Very capable

10% GR&R 20% Capable

20% GR&R 30% Marginally capable

GR&R 30% Not capable

%100)(15.5

% xT

sGRR

meas

%100)(

)(57.2% xSLx

sGRRmeas

Two-sided spec (T=USL-LSL) One-sided spec:

%GRR describes what % of your working range is taken up byvariability in your measurement system. The lower the %GRR, the

better you can distinguish actual process movement (often what yourcustomers are after) from measurement variability.

Meas.StdDev

GRR indicates method precision . . .

not accuracy (see note)

Gauge Repeatability & Reproducibility (%GRR)


55/126


PolyStat, a polymer formulator, is analyzing a polymer sample for an

additive using NIR. They believe that they have optimized the methodand now want to perform a gauge R&R.

The two operators who will be running the test are asked to run thegauge samples.

Manufacturing has said that the upper spec is 0.8% and the lowerspec. is 0.3%. They have also indicated that the standard deviation ofthe process is 0.3%.

The Black Belt has picked samples she feels are representative of theprocess and has told the operators how the samples are to be run.

The results are in the Excel file GRR Example.xls.

Using Minitab, calculate the GRR based on the process capability andthe tolerance. Is this a good method?

If so, why? If not, why not and what can you do to fix it?

Breakout #3: GRR


56/126


Open the file GRR Example.xls

Copy the .xls data into Minitab

Stack the data using the stack function in Minitab

Perform a GR&R using the Stat feature

Use the ANOVA approach, enter the process variance

(6 x 0.3 = 1.8)How good is the method?

Breakout #3: Instructor will demonstrate.

Breakout #4: Students will practice.

Instructions for Breakout Problem #3


57/126


Have to stack columns before GRR

analysis in Minitab

Breakout #3: GRR Screen Shots

Operator 1 Operator 2 Sample Replicate

0.49 0.48 1 1

0.49 0.45 1 2

0.45 0.49 1 3

0.55 0.55 2 1

0.57 0.59 2 2

0.62 0.63 2 3

0.23 0.21 3 1

0.23 0.21 3 20.25 0.24 3 3

0.5 0.55 4 1

0.49 0.53 4 2

0.48 0.48 4 3


58/126


1. Duplicate Samples

2. Set up GRR

analysis



59/126


Enter Process variation

Whats this?

Well get to that later..



60/126


Part-to-PartReprodRepeatGage R&R

100

50

0

Percent

% Contribution

% Study Var% Process

0.10

0.05

0.00SampleRange

_R=0.0462

UCL=0.1191

LCL=0

Operator 1 Operator 2

0.6

0.4

0.2

SampleMean

__X=0.4483UCL=0.4957

LCL=0.4010


4321

0.6

0.4

0.2

Sample

Operator 2Operator 1

0.6

0.4

0.2

Operator

4321

0.6

0.4

0.2

Sample

Average

Operator 1

Operator 2

Operator

Gage name:

Date of study :

Reported by :

Tolerance:

Misc:

Components of Variation

R Chart by Operator

Xbar Chart by Operator

Data by Sample

Data by Operator

Operator * Sample Interaction

Gage R&R (ANOVA) for Data

1. Good gaugerelative to

part variation

2. Repeatabilityis most ofgauge error

3. No interactionb/w operator

and part

4. Part 3 is low

Interpretation

Breakout #3: GRR Graphical Results


61/126


Now enter process tolerance (USL-LSL)

Breakout #3: Screen Shots



62/126


Part-to-PartReprodRepeatGage R&R

160

80

0

Percent

% Contribution

% Study Var

% Process

% Tolerance

0.10

0.05

0.00Sam

pleRange

_R=0.0462

UCL=0.1191

LCL=0


0.6

0.4

0.2

SampleMean

__X=0.4483UCL=0.4957

LCL=0.4010


4321

0.6

0.4

0.2

Sample

Operator 2Operator 1

0.6

0.4

0.2

Operator

4321

0.6

0.4

0.2

Sample

Average

Operator 1

Operator 2

Operator

Gage name:

Date of study :

Reported by :

Tolerance:

Misc:

Components of Variation

R Chart by Operator

Xbar Chart by Operator

Data by Sample

Data by Operator

Operator * Sample Interaction

Gage R&R (ANOVA) for Data

Measurement gauge R&Ris low relative to thetolerance (i.e. specs)

Repeatabilityerror takes upmore of thetolerance than

reproducibility.

The parts chosen werewell outside thetolerance and themethod was capable of

differentiating them.


GRR Breakout #3: Results in Minitab Session Window


63/126


What doesthis tell us?

Focus on the p-value:p


64/126


Part-to-part variabilityaccounts for 158% of

the tolerance

Std. dev. parts/ Std. dev. gauge *1.41

Measurement gauge variabilityaccounts for 27.13% of the tolerance

GRR Breakout #3: Results in Minitab Session Window


65/126


PolyStat is trialing some manufacturing changes. The preliminary dataindicate that they are now making better product, based on weatheringand tensile data.

The upper spec is 0.8% and the lower spec. is still 0.3%. They haveindicated that the standard deviation of the process is now 0.25%.

The same two operators are asked to run the GRR experiment with anew set of samples. Their results are in the Excel file GRR Example2.xls.

Using Minitab, calculate the GRR based on the process capability andthe tolerance. Is this a good method?

If so, why? If not, why not and what can you do to fix it?

Breakout #4: GRR


66/126


Fishbone Diagram

FTIR & NIRIR Parameters

PeopleMaterialsMeasurements

Environment Methods Machines

Mole% X

Temperature

Film PressSample Placement

Standard

Quality

ContaminationSpectral Regions

Humidity

Sample Prep

Sample Loading

Multitasking

Spectral Quality

Statistical Model

Homogeneity

A great way to brainstorm potential factors that may affect the result

A cross-functional effort, allow enough time

Include all input, whether known or only suspected to affect the result

Better to have too many factors than too few . . . DOE will help screen

Example Fishbone diagramfor FTIR compositional test

method


67/126


Pittcon March 2013

Section 3Identify sources of variation . . .

Then optimize!




68/126


Introduction to the Design of Experiment (DOE)

What is a DOE? When is it appropriate?

Visualizing the Experimental Space

- Factors and Levels, Coding

Importance of Replication and Randomization

Importance of Orthogonality and Blocking

Types of DOEs

Screening and Optimization

- Full and Fractional Factorial DOEs

Appendix

Second Order Designs

Additional Break-out Examples

Section 3: Outline

One at a Time Approach


69/126


Box, Hunter, and Hunter. Statistics for Experimenters. John Wiley and Sons, Inc., 1978.

Challenge: Optimize yield of chemical reactionReaction time (t)

Reaction temperature (T)

One-at-a-time approach:maximum yield of 75 grams at 130 min and 225C

One-at-a-Time Approach


70/126


Box, Hunter, and Hunter. Statistics for Experimenters. John Wiley and Sons, Inc., 1978.

- Time consuming- Expensive testing (time and supplies)

- No consideration of variable interactions

When variables are changedsimultaneously.

Global Maximum:

Yield of 91 grams65 min and 255C

Local Maximum:yield of 75 grams

130 min and 225C

One-at-a-Time Approach


71/126


Structured process, formal plan Investigates the relationship between input and output

factors. One-at-a-time approach of controlling independent variables

and observing dependent variables

By contrast, what is a DOE?

Experimental methodology Multiple independent variables are controlled simultaneously

Structured experiments are conducted in a prescribed way Efficient and accurate analysis which determines the

significance and/or the mathematical relationships of factorsto a measured output.

What is Experimental Design?


72/126


If preliminary work or expert knowledge reveals anobvious cause and a solution exists.

If: Root cause(s) cannot be found Further improvement is desired, after removing

root causes

Many potential factors affect the response You wish to quantify the relationship between the

factors and the response

Is a Designed Experiment Appropriate?

NO

YES



73/126


Challenge:How can we maximize

reaction yield?X + Y Z

Experimental variables from

team brainstorming:Factor A = TemperatureFactor B = TimeFactor C = pH

125C

50C

15min5min

Factor B

Factor A

4

125C

50C

15min

8

5minFactor B

Factor A

Factor C




74/126


2 Factors, 2 Levels = SQUARE

3 Factors, 2 Levels = CUBE

Full Factorial:2^ 2 = 4 runs

Full Factorial:2^ 3 = 8 runs

# Levels^ #Factors = # Runs


+1

-1+1-1

Factor B

Factor A

-1

+1

-1

+1

+1

-1Factor B

Factor A

Factor C

What kinds of Experimental Designs are there?


75/126


Fractional FactorialFull Factorial

What information can we obtain using factorial designs?

Main effects of factors (Xi) and Effects of interactions (Xi * Xj)Variability associated with each factor and interactionEffects of experimental errorPredictive transfer functions, Example: y = b1(Xi) + b2(Xj) + b3(Xi*Xj)

2) Optimization DOE To validate screening DoE, determineoptimum input variables, characterize the input/output response surface(check for curvature) by adding levels or using different designs.

What kinds of Experimental Designs are there?

1) Screening DOE Identify statistically significant variables

I t ti E i Fi hb (C & Eff t)


76/126


Interactive Exercise: Fishbone (Cause & Effect)Diagram

Means of organizing brainstorming ideas on the possible causes of

some stated outcome into major categories

Put down EVERYTHINGyoull filter the list later

Human Materials Machine

Method Nature Measurement

Outcome

R i t f DOE D i


77/126


Several aspects of DOE design ensure the DOE analysis results

in statistically valid andpractically valid conclusions:

Replication

Randomization

Blocking

Balance & Orthogonality

Requirements for DOE Design

Replication and Experimental Error


78/126


In order to determinestatistical significance of main effectsand interaction factors, the magnitude of these effects arecompared to the experimental error present in the system.

In order to estimate the error, experimental runs are replicated.

Where to add replicates? Where you want the mostinformation:

Factor A

Factor B +

+

+Factor C

Corner-pointreplication Provides error data across design

space Keeps experiment balanced

Factor A

Factor B +

+

+Factor C

Center-pointreplication More information about that areaof the design space Higher number of replicatesreduces confidence interval around

the estimate of error at that point

p p

Randomization is a MUST!


79/126


123456789101112

-1+1-1+1-1+1

-1+1-1+1-1+1

-1-1+1+1-1-1

+1+1-1-1+1+1

-1-1-1-1+1+1

+1+1-1+1+1-1

StdOrder

FactorA

FactorB

FactorC

(Replicate Runs)

If the Run Order = Std Order,then any uncontrolled factor thatvaries with time (i.e. noise) willeffect response.

Example:

In an LC method, the temperature in thelaboratory can effect the chromatographicpeak resolution.

If the runs are not randomized, then thetemperature effect, which is not acontrolled factor, can effect the DOE.

Also, if all the replicates are run at the

end, then the experimental noise is notrepresentative of the true noise level.

The runs should be randomized. Replicates should be spread randomly

throughout the experiment.

RandomizedRun Order

83

121015

72946

11

Factor A

Factor B +

+

+

Factor C

Blocking


80/126


What is Blocking?

Systematic way of dividing the runs into subsets. Maximum number

of runs that can be made under homogenous conditions.Why Block?

If there are factors which are believed to affect the response but areof no interest to the experimenter, these factors can confound theresults and produce erroneous conclusions.

By assigning a block to each level of this nuisance factor, the effect ofthis factor can be isolated from those of interest. (Time- shift, days,etc.)

When to Block?

When there is a significant risk of a nuisance factor confounding theexperimental results.. Without replication of these blocks, however,there is no ability to test for the significance of the blocking factor.

The cost of blocking is the ability to determine higher-order effects.Only block in screening DOEs- disregarding a source of variation willnot result in a robust design.

g

Full Factorial DOE: Balance and Orthogonality


81/126


Run Order A B AB

1 (1) -1 -1 1

2 a 1 -1 -1

3 b -1 1 -1

4 ab 1 1 1

Balanced0 for each factor sum

This feature helps to simplify the analysis

X i =

Xi Balance and Orthogonality

Ensure uniform informationthroughout the design space

Ensure independent estimates

of factor effects

These conditions are satisfied

for full factorial designs

(Adapted from Mikel J. Harry. The Vision of Six Sigma. 1994. Page 18.9)

Unbalancedreplication

Not orthogonalEither missed levelor operational spacewas constrained.

g y

OrthogonalThis feature ensures the effects are independent

0 for all dot product pairs =X Xji

Full Factorial DOE: Cost vs. Benefit


82/126


Numberof Factors

Numberof Runs

123456.

.

.10

248163264.

.

.1,024

Cost

$$$$$$

$$$$$$$$$

$$$$$$.

.

.$$$$$$$$$

Highest orderinteraction

123456.

.

.10

Number of 3-way interactions

0014916.

.

.119

Number of 2-wayinteractions

01361015.

.

.45

Question to consider:Do we have the time, money and resources

to run all these experiments,plusreplicates?

Usually-NO!!

2 ^ Factors = # Runs

Full Factorial DOE: Cost vs. Benefit

Fractional Factorial Designs for Screening DOEs

Screening DOE: Fractional Factorial


83/126


123456

78

-1+1-1+1-1+1

-1+1

-1-1+1+1-1-1

+1+1

-1-1-1-1+1+1

+1+1

StdOrder

FactorA

FactorB

FactorC

Full Factorial 23 = 8 runs

Which runs do you choose to run? (or NOT?)

Remember what informationyou get from a full factorial

Grand mean: Y-bar

Main effects: A, B, C

2-Way interactions: AB, AC, BC

3-Way interaction: ABC

ABC

-1+1+1-1+1-1-1+1

Run only the runs for which ABC is -1

3-Way interaction terms are probably

NOT statistically significant

Half Fraction Factorial Design:2k-1 = 23-1 = 22 = 4 runs

Half Fraction design utilize half thenumber of runs of Full Factorial

Designs

Factor A

Factor B +

+

+

Factor C


84/126


1) Identify number of factors

2) Identify levels of each factor

3) Calculate # runs for a full factorial# Full Factorial Runs = Levels ^ Factors

4) Select appropriate resolution

Which interactions can you tolerate?

5) Plan DOE using Minitab or other software

6) Finally... RUN EXPERIMENTS !!

Steps to designing a fractional factorial DOE:


85/126


Resolution of Fractional Factorial Designs - Minitab


86/126


Green = Res V+ Yellow = Res IV Red = Res III

Designs23-1

24-1

25-125-2

26-1

26-2

26-3

Runs4

8

168

3216

8

Alias StructureC = AB

D = ABC

E = ABCDD = AB, E = AC

F = ABCDEE = ABC, F = ACD

D = AB, E = AC, F = BC

ResolutionIII

IV

VIII

VIIV

III

Examples of fractional factorial designs

# Runs = Levels^ (Factors p)

Factorial then p = 1




87/126


Fractional Factorial DOEConstruction

Breakout #5

Breakout #5: Ion Chromatography Example


88/126


Water, extracted anions

Methylene chloride,dissolved polymer

Current Method: Volatile Chloride in PolymersHeat 5g sample to 300C

Collect vapor in H2

OAnalyze by Ion Chromatography

Proposed Method: Water Extraction of AnionsShake 2.5g sample in 20mL of CH2Cl2until dissolvedAdd 15mL H

2O &shake to extract

Allow emulsion time to settleAnalyze concentration in final

solution by IonChromatography

Heated Polymer Water,

volatile chloride

N2

g p y p

72% GRR!!

Breakout #5: Fractional Factorial DOE Construction


89/126


Instructor will demonstrate how to set up DOE.

Then students will create the DOEClass will solve DOE together

1. Open Minitab

2. Create Factional Factorial DOE (2-Levels, 5-Factors)

with 1 center point, 1 block and 2 replicates at the corners

3. Label Xs: mass, volume, shake1, shake2, settle

4. Open file Breakout ppm Cl factorial doe.MPJ in Minitab

5. Analyze factorial design

Graph: regular residuals with a = 0.05

6. Identify any significant interactions (check p values)

7. Investigate ANOVA Main Effects and Interaction plots

Breakout #5: Fractional Factorial DOE Construction

Breakout #5: Create Factorial DOE (Screen Shots)


90/126


Breakout #5: Create Factorial DOE (Screen Shots)


91/126


Replicates let you estimate the error across designspace to help show whether factors are significant

Breakout #5: Enter Results (Screen Shots)


92/126


Add factor labels

After running experiments,add the responses

Breakout #5: Analyze Factorial DOE (Screen Shots)


93/126


Breakout #5: Analyze Factorial DOE (Screen Shots)


94/126


TermsGraphs

This problem hasbeen worked out

before, and its knownnot to have any 3-wayinteractions. To savetime in class, choose

order of 2.

Analyze Factorial DOE - Residuals Plots


95/126


302520151051

3

2

1

0

-1

-2

-3

Observation Order

StandardizedResid

ual

Versus Order(response is ppm Cl)

3210-1-2-3

99

95

90

80

70

60

50

40

30

20

10

5

1

Standardized Residual

Percent

Normal Probability Plot(response is ppm Cl)

0.90.80.70.60.50.4

3

2

1

0

-1

-2

-3

Fitted Value

StandardizedResid

ual

Versus Fits(response is ppm Cl)

2.41.20.0-1.2-2.4

9

8

7

6

5

4

3

2

1

0

Standardized Residual

Frequency

Histogram(response is ppm Cl)

No systematic variationduring DOE

Data are normallydistributed

Residuals are

indicative of noiseand therefore shouldbe normallydistributed andshould not exhibitsystematic variationwith time.

Patterns in theresiduals plots mayindicate a criticalfactor that is notbeing controlled!

Or that data

transformationmay be needed (i.e.Log(x))

Points falling on the line -

normally distributed

Look for trumpet

heteroscedastic data

Pareto Chart of Main Effects


96/126


CE

BE

AC

E

BC

D

C

AD

BD

AB

DE

AE

CD

B

A

121086420

Term

Standardized Effect

2.11

A mass

B v olume

C shake1

D shake2

E settle

Factor Name

Pareto Chart of the Standardized Effects(response is ppm Cl, Alpha = 0.05)

The dotted line isthe boundarybetween significant(above the line) andinsignificant (belowthe line) accordingto alpha.

Quick Visual Confirmation of Significant Factors

Main Effects Plots


97/126


Go to Stat >> DOE >> Factorial >> Factorial Plots . . .Click Setup to select plot type and terms.

1-1

0.75

0.70

0.65

0.60

0.55

1-1 1-1

1-1

0.75

0.70

0.65

0.60

0.55

1-1

mass

Mean

volume shake1

shake2 settle

Main Effects Plot for ppm ClData Means

Interaction Plots


98/126


1-1 1-1 1-1 1-1

0.80

0.65

0.50

0.80

0.65

0.50

0.80

0.65

0.50

0.80

0.65

0.50

mass

volume

shake1

shake2

settle

-1

1

mass

-1

1

volume

-1

1

shake1

-11

shake2

Interaction Plot for ppm ClData Means

Go to Stat >> DOE >> Factorial >>Factorial Plots . . .

Click Setup to select plot typeand terms.

DOE Session Window Evaluation of Fit


99/126


P < 0.05 indicates asignificant factor

Hmmmdoes this make

sense with what we knowfrom experience?

Shake1 * Settle is leastlikely to affect the ppm Cl

Evaluation of Fit

S H d d ill d

DOE Analysis Elimination of Insignificant Terms


100/126


So . . . How do we drill downto the important factors?

Still

significantfactors

Least significant term . . .Eliminate it next!

p-values MAY change each time a

term is eliminated.

Af li i i i i ifi

Elimination of Insignificant Terms contd


101/126


D

C

CD

B

A

121086420

Term

Standardized Effect

2.06

A mass

B v olume

C shake1

D shake2

Factor Name

Pareto Chart of the Standardized Effects

(response is ppm Cl, Alpha = 0.05)

. . . After eliminating insignificant terms

significantfactors

Good! Nosignificantlack of fit.

Transfer Function (in coded units):ppm Cl = 0.6530 (0.1022 x mass) + (0.0728 xvolume) (0.0197 x shake1 x shake2) + (0.0078x shake1) (0.0078 x shake2)

Why do weneed these?

Solve the Transfer Function in Uncoded Units


102/126


Factor

Low

Setting

High

Setting

mass 2.5 7.5

volume 10 20

shake1 10 30

shake2 10 30

settle 5 15

If the actual low, high settings are . . .

Put into newworksheet



103/126


Factor levels now in actual units with the same run order as before

Copy responsesinto last column

Analyze Factorial Design andselect the significant factors



104/126


Transfer function in actual units:

PPM Cl = 0.5602 (0.04087 x mass) +(0.01456 x volume) (0.00020 xshake1*shake2) + (0.00472 x shake1) +(0.000316 x shake2)

Effect of Factors on Response Now Known . . .As long as the effect is linear for each factor

New details in Session window

D

C

CD

B

A

121086420

Term

Standardized Effect

2.05

A mass

B volume

C shake1

D shake2

Factor Name

Pareto Chart of the Standardized Effects(response is ppm Cl, Alpha = 0.05)

Screening DOE: Summary and Important Points toC id


105/126


Helps you focus on the critical factors that influence the response

Points out important interactions between factors or directs your attention toother factors you hadnt considered

Transfer function should be verified experimentally by running selectedexperiments at the low and high settings to check predictive capability.

The screening model assumes that each factor has a linear effect on theresponse because only 2 factor levels were tested.

The screening model does not specifically address variance because eachcorner of the design space is tested only once.

You may need to perform a second screening DOE to include other factors orre-center the range.

The next sections will demonstrate how to strategically model the variance todevelop the most robust method.

An Optimization DOE on significant factors will help show any curvature inthe design space. . . . Next

Consider

Optimization DOE


106/126


Why do an Optimization DOE? Add center-points for curvature

Re-center the design space based on screening results Focus only on significant factors Improve precision of the model by adding replicatesDetermine a transfer function between your xs and yourresponse

How is the experiment different? More than 2 levels to each factor Usually fewer factors Often additional replicates

Youre more knowledgeable now, and set a better design space

Does it take more time than a screening DOE?Usually doesnt take any more time, and sometimes takes less!

Optimization: Second Order Experimental Designs


107/126


Corner Points-Assessment oflinear and 2-way

interactions

Center Points-Needed to determine ifcurvature is present. Replicating centerpoint is a common way to get at pure error.

Star (Axial) Points-For the assessment ofquadratic terms

Central Composite Designs Box-Behnken Designs

Exist only for

number of factors= 3 to 7

No Corner Points-Used in situationswhen design space isphysically constrainedand corner points arenot possible.

65

75

85

95

Rising

Ridge

-4.00-2.00

0.00

2.00

4.00

-4.00

-2.00

0.00

2.00

4.00

AB

Center points and star points areneeded to adequately model thebehavior of a response when thesystem includes curvature.

Example ofcurved

responsesurface

Breakout #6 RS Design with Optimization


108/126


Method development specialists at Polystuff Products analyze the extent of

polymerization in lab-scale reactions by measuring the amount of residualmonomer following reaction in a stirred vessel. The scientists observe that, in

general, the rate of polymerization increases with increased stirrer speed.

However, when the stirrer speed gets too high, shear instability causes the

solids to coagulate out of suspension, reducing the polymerization rate. The

amount of suspended polymer can roughly be measured as turbidity by UV-Vis

absorption.

The scientists designed a central composite design DOE with two factors

(stirrer speed and mass of initial monomer) measuring two results (residual

monomer and turbidity) find the optimal settings for their process. The goal is

to achieve a residual monomer level between 410 ppm and 427 ppm, and

maximum relative turbidity between 0.7 and 0.88.

Experimental data can be found in the breakout file named:

Breakout-6 CCD Optimization-DOE.MPJ

Breakout #6 RS Design with Optimization

Optimization: Create RS (Response Surface) Design


109/126




110/126


After runningexperiments,add the responses

Factorsettings


Optimization: Analyze RS Design


111/126


Include all terms(main effects,

interactions,quadratic)

Residuals plots can begenerated as shown

before


Analyze oneresponse at a time



112/126


Now eliminate least significant terms one-by-one,leaving in the main effects

Mass * Mass is least significant

Lack of Fit is not significant.

Good

Stirrer * Mass is significant

Response = Residual Monomer

The regression is significant.



113/126


Residual Monomer = 406.774 + (9.780 x stirrer) +(46.279 x Mass) (9.439 x stirrer x mass)

Final Transfer Function

Stirrer x Mass remains

significant. Main effects

must be kept.

Response = Residual Monomer



114/126


Response = Turbidity

Turbidity = -0.01829 + (0.12011 x stirrer) +(0.18035 x Mass) (0.00649 x stirrer2)

Final Transfer Function

Stir, Mass, and Stir2

are all significant, but what

about the constant?

After drilling down to thecritical factors . . .


Optimization: Plot the Response Surface


115/126


RS Plots help visualize the interacting

effect of each factor on the response

Contour Plot

flat graph with coloredtopographical lines Wireframe Plot3-D projection of curve

Each plot has Setup

options (examples below)

Specify which

response andfactors to plot

Specify how to

handle otherfactors if more

than 2 factors

exist.

C t Pl t f R id l M Sti d M Wi f Pl t f R id l M Sti d M

Optimization: Plot the Response Surface


116/126


. . . But SO WHAT?

I want to know how to set all the knobs on my method tooptimize both responses simultaneously!

2.152.051.951.851.752.5

0.51.65

3.5Mass

0.6

4.5

0.7

1.555.5

0.8

0.9

6.51.45

7.5 8.5

Turbidity

1.359.510.5

1.2511.5Stirrer

Wireframe Plot of Turbidity vs. Stirrer and Mass

0.5292680.5648590.6004510.6360420.6716330.7072240.7428150.7784070.8139980.8495890.8851800.920772

3 4 5 6 7 8 9 10 11

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2.0

2.1

Stirrer

Mass

Contour Plot of Turbidity vs. Stirrer and Mass

396.719403.990411.260418.531425.802433.072

440.343447.614454.885462.155469.426476.697

3 4 5 6 7 8 9 10 11

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2.0

2.1

Stirrer

Mas

s

Contour Plot of Residual Monomer vs. Stirrer and Mass

2.152.051.951.851.752.5

3901.65

3.5

400

Mass

410

420

4.5

430

440

1.555.5

450

460

470

6.5

480

1.457.5 8.5

1.35

Residual Monomer

9.510.51.25

11.5Stirrer

Wireframe Plot of Residual Monomer vs. Stirrer and Mass

ResidualMonomer

Turbidity

Minitab 15 has snazzier plots than Minitab 12

Optimization: Optimize all Responses


117/126


Optimization: Optimize all Responses


118/126


CurHigh

Low1.0000D

Optimal

d = 1.0000

Targ: 427.0

Residual

y = 427.0000

d = 1.0000

Maximum

Turbidit

y = 0.9031

1.0000Desirability

Composite

1.2757

2.1243

2.7574

11.2426MassStirrer

[7.6026] [2.1243]

Optimal FactorSettings

Predicted

Responses

Drag the vertical red lines

left and right to changethe factors and see theeffect on each response

Now you have it all!

To reset:Right click,Reset tooptimalsettings

Course 1 Summary


119/126


Course 1 Summary

There are several tools to help understand your data

(hypothesis tests, ANOVA, GRR, etc.) These tools can indicate sources of variation or

equivalence of data sets

Several DOE strategies are available to identify the

factors that affect the response Screening DOE simplest design for quickly probing

linear effects for multiple factors

Optimization DOE more complex design forunderstanding higher order interactions.

Proper use of these concepts can help you trulyoptimize your methods.

Revisiting Students Course Expectations


120/126


Revisiting Students Course Expectations

Did you get what you came for?

Additional references

1. Basic Statistics, Tools for Continuous Improvement. Mark J. Kiemele, Stephen


121/126


1. Basic Statistics, Tools for Continuous Improvement. Mark J. Kiemele, StephenR. Schmidt, Ronald J. Berdine. Air Academy Press, 1999. ISBN 1-880156-06-7

2. Chemometrics: Data Analysis for the Laboratory & Chemical Plant. Richard G.Brereton. Wiley, 2003. ISBN 0-471-48978-6

3. Experimental Design: A chemometric approach. Stanley N. Deming, StephenL. Morgan. Elsevier, 1987. ISBN 0-444-42734-1

4. Statistics: A Guide to the Use of Statistical Methods in the Physical Sciences.J.R. Barlow. Wiley, 1989. ISBN 0-471-92295-1

5. Statistics and Chemometrics for Analytical Chemistry. Jane Miller. PrenticeHall, 2005. ISBN 0-131-29192-0

6. Statistics for Analytical Chemistry. Jane C. Miller, James N. Miller. PrenticeHall, 1993. ISBN 0-130-30990-7

7. Statistics for Experimenters, An Introduction to Design, Data Analysis, andModel Building. George E. P. Box, William G. Hunter, and J. Stuart Hunter.Wiley, 1978. ISBN 0-471-09315-7

8. Understanding Industrial Designed Experiments. Stephen R. Schmidt, Robert

G. Launsby. Air Academy Press, 1994. ISBN 978-18801560329. http://science.widener.edu/svb/stats/stats.html

10.www.statease.com

11.www.minitab.com/resources

12.www.itl.nist.gov

Acknowledgements


122/126


Acknowledgements

Pittcon Short Course Committee

SABIC Innovative Plastics Analytical Technology

Statistics Group at GE Global Research Center (Angie Neff,

Martha Gardner)

Appendix


123/126


Pittcon March 2013

Course 1 Appendix



ppe d

Anther DOE Design: Mixture (available with Design Expert software)Appendix


124/126


Simply another class of DoEsRequirements:

Components dependent oneach other

Response is based on ratioof components

If not, investigate response

surface designs

Types of mixture DoEs: Simplex Identical factor ranges for

each component Non-constrained design D-Optimal Highly constrained design

Breakout: full factorial DOE practice

Appendix


125/126


Example: Extraction of Pb from a soil matrix.

The primary factors considered are Time, Temp, and Acid Conc.

Create a full factorial DOE:23 = 8 runs

with replicates at each point = 16 total runs

Create the DOE in Minitab

Input the experimental results from the file:DOE ppm Pb.xls

Label columns and analyze the data.

p

(coded and uncoded data are available on

separate worksheets within the Excel file)

STD

FACTORS

Consider a Full 24 factorial design (16 Runs)

Challenge: If you can only afford to run 8

Breakout: fractional DOE practice Appendix


126/126

STD

Order A B C D ABCD

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

1. Create DOE by hand, filling in 1 & 1 levels

2. Compute the dot product for ABCD3. Choose runs for which ABCD = 1 or1

STD

Order A B C D=ABC ABCD

1

2

3

45

6

7

8

FACTORS

This is the same as using the Design

GeneratorD = ABC:

1.Construct a 23 design with A, B, and C

2 Set D=ABC and compute dot product

C a e ge If you can only afford to run 8runs, which 8 do you choose?

For a 24 full factorial:

Grand mean: X-bar

Main effects: A, B, C, D

2-Way interactions: AB, AC, AD, BC, BD,CD

3-Way interactions: ABC, ABD, ACD, BCD

4-Way interaction: ABCD

16 items estimated from the 16 runs

Documents

Attack the Variance.hurst-1