Diploma in Statistics Design and Analysis of Experiments Lecture 4.21 Design and Analysis of Experiments Lecture 4.2 Part 1: Components of Variation –identifying

Diploma in StatisticsDesign and Analysis of Experiments

Lecture 4.2 1

Design and Analysis of ExperimentsLecture 4.2

Part 1: Components of Variation

– identifying sources of variation– hierarchical design for variance component

estimation– hierarchical ANOVA

Part 2: Measurement System Analysis

– Accuracy and Precision– Repeatability and Reproducibility– Components of measurement variation– Analysis of Variance– Case study: the MicroMeter


Lecture 4.2 2

An invalid comparison

Comparing standard process, A,

with modified process, B

A B

58.3 63.2

57.1 64.1

59.7 62.4

59.0 62.7

58.6 63.6

Means: 58.54 63.20

St. Devs: 0.96 0.68


Lecture 4.2 3

An invalid comparison

Process: batch manufacture of pigment paste

Key variable: moisture content

Sampling plan: single sample from single batch

Measurements: 5 repetitions

s measures measurement error

no measure of variation within batch

no measure of variation between batches


Lecture 4.2 4

Sources of variation in moisture content

• Batch, subject to Process variation

• Sample from batch, subject to within batch variation

• Measurement, subject to Test variation

Model for variation in moisture content:

Y = + eP + eS + eT


Lecture 4.2 5

Values

Values

eP

P


Processvariation

•


Lecture 4.2 6

Values

Values

eP

eS

S

P


Processvariation

Samplingvariation

•

•


Lecture 4.2 7

Values

Values

•

S

T

eP

eS

P

eT

e = eP + eP + eP


Processvariation

Samplingvariation

Testingvariation

y

•

•


Lecture 4.2 8

Components of Variance

Recall basic model:

Y = + eP + eS + eT

Components of variance:2T

2S

2P

2Y

2T

2S

2PY


Lecture 4.2 9

Conclusions for process testing

• Process measurements are subject to a hierarchy of variation sources.

• Several measurements on a single sample from a single batch do not reflect overall variation.

• Several batches and several samples from each batch are necessary to capture overall variation.

• Comparison of process methods must be referred to the relevant level of variation


Lecture 4.2 10

Hierarchical Design forVariance Component Estimation

• A batch of pigment paste consists of 80 drums of material.

• 15 batches were available for testing

• 2 drums were selected at random from each batch and a sample was taken from each drum.

• 2 tests for moisture content were run on each sample.

• The results follow


Lecture 4.2 11

Hierarchical Design forVariance Component Estimation

Batch 1 2 3 4 5 Sample 1 2 3 4 5 6 7 8 9 10 Test 40 39 30 30 26 28 25 26 29 28 14 15 30 31 24 24 19 20 17 17 Batch 6 7 8 9 10 Sample 11 12 13 14 15 16 17 18 19 20 Test 33 32 26 24 23 24 32 33 34 34 29 29 27 27 31 31 13 16 27 24 Batch 11 12 13 14 15 Sample 21 22 23 24 25 26 27 28 29 30 Test 25 23 25 27 29 29 31 32 19 20 29 30 23 23 25 25 39 37 26 28


Lecture 4.2 12

Nested ANOVA: Test versus Batch, Sample

Analysis of Variance for Test

Source DF SS MS F PBatch 14 1216.2333 86.8738 1.495 0.224Sample 15 871.5000 58.1000 64.556 0.000Error 30 27.0000 0.9000Total 59 2114.7333

Variance Components % ofSource Var Comp. Total StDevBatch 7.193 19.60 2.682Sample 28.600 77.94 5.348Error 0.900 2.45 0.949Total 36.693 6.058


Lecture 4.2 13

Nested ANOVA: Test versus Batch, Sample

Expected Mean Squares

1 Batch 1.00(3) + 2.00(2) + 4.00(1)2 Sample 1.00(3) + 2.00(2)3 Error 1.00(3)

Translation:

EMS(Batch) =

EMS(Sample) =

EMS(Error) =

2B

2S

2T 42

2S

2T 2

2T


Lecture 4.2 14

Calculation

= EMS(Error)

= ½[EMS(Sample) – EMS(Error)]

= ¼[EMS(Batch) – EMS(Sample)]2B

2S

2T


Lecture 4.2 15

Theory

Model:

Yijk = + i + i(j) + ijk

Yij. = + i + i(j) + ij.

Yi.. = + i + i(.) + i..

Decomposition:

(Yijk – Y... ) = (Yi.. – Y... ) + (Yij. – Yi.. ) + (Yijk – Yij. )

(Yij. – Yi.. ) = (i(j) – i(.) ) + (ij. – i.. )

EMS involves and

2S 2

T


Lecture 4.2 16

Conclusions fromVariance Components Analysis

Variance Components % ofSource Var Comp. Total StDevBatch 7.193 19.60 2.682Sample 28.600 77.94 5.348Error 0.900 2.45 0.949Total 36.693 6.058

Sampling variation dominates, testing variation is relatively small.

Investigate sampling procedure.


Lecture 4.2 17

Sampling procedure

Standard:

• select 5 drums from batch at random,

• sample all levels of each drum using a specially constructed sampling tube

• thoroughly mix all samples

• take a sample from the mixture for analysis

Actual

• take a sample from a drum for analysis


Lecture 4.2 18

Another Example

Testing drug treatments for pregnant women

22 women, 10 treatment A, 7 treatment B, 5 "control".

Placentas examined for "irregularities":

5 locations,

10 slices, on microscope slides,

5 measurements (counts of "irregularities") per slide,

5,500 measurements in all.

No significant treatment effect (10 vs 7)


Lecture 4.2 19

Yet Another Example

Comparing schools on student performance

Schools

Classes within schools

Students within classes


Lecture 4.2 20

Design and Analysis of ExperimentsLecture 4.2

Part 2: Measurement System Analysis

– Accuracy and Precision– Repeatability and Reproducibility– Components of measurement variation– Analysis of Variance– Case study: the MicroMeter


Lecture 4.2 21

The MicroMeter optical comparator


Lecture 4.2 22

The MicroMeter optical comparator

• Place object on stage of travel table

• Align cross-hair with one edge

• Move and re-align cross-hair with other edge

• Read the change in alignment

• Sources of variation:

– instrument error

– operator error

– parts (manufacturing process) variation


Lecture 4.2 23

Precise

Biased

Accurate

Characterising measurement variation;Accuracy and Precision

Imprecise


Lecture 4.2 24

Characterising measurement variation;Accuracy and Precision

Centre and Spread

• Accurate means centre of spread is on target;

• Precise means extent of spread is small;

• Averaging repeated measurements improves precision, SE = /√n

– but not accuracy; seek assignable cause.


Lecture 4.2 25

Accuracy and Precision: Example

Each of four technicians made six measurements of a standard (the 'true' measurement was 20.1), resulting in the following data:

Technician Data

1 20.2 19.9 20.1 20.4 20.2 20.4

2 19.9 20.2 19.5 20.4 20.6 19.4

3 20.6 20.5 20.7 20.6 20.8 21.0

4 20.1 19.9 20.2 19.9 21.1 20.0

Exercise: Make dotplots of the data. Assess the technicians for accuracy and precision


Lecture 4.2 26

Accuracy and Precision: Example


Lecture 4.2 27

Repeatability and Reproducability

Factors affecting measurement accuracy and precision may include:

– instrument

– material

– operator

– environment

– laboratory

– parts (manufacturing)


Lecture 4.2 28

Repeatability and Reproducibility

Repeatability:

precision achievable under constant conditions:

– same instrument

– same material

– same operator

– same environment

– same laboratory

Reproducibility:

precision achievable under varying conditions:

– different instruments

– different material

– different operators

– changing environment

– different laboratories


Lecture 4.2 29

Measurement Capability of the MicroMeter

4 operators measured each of 8 parts twice, with random ordering of parts, separately for each operator.

Three sources of variation:

– instrument error

– operator variation

– parts(manufacturing process) variation.

Data follow


Lecture 4.2 30

Measurement Capability of the MicroMeter


Lecture 4.2 31

Quantifying the variation

Each measurement incorporates components of variation from

– Operator error

– Parts variation

– Instrument error

and also

– Operator by Parts Interaction


Lecture 4.2 32

Measurement Differences

Part Operator Repeats Diffs Part Operator Repeats Diffs

1 1 96.3 95.4 0.9 5 1 99.4 99.9 -0.5 2 97.0 96.9 0.1 2 100.1 99.8 0.3 3 98.2 97.4 0.8 3 100.9 99.4 1.5 4 97.4 99.6 -2.2 4 100.0 99.4 0.6

2 1 95.5 95.8 -0.3 6 1 93.8 94.9 -1.1 2 96.1 96.8 -0.7 2 95.9 95.8 0.1 3 97.9 99.4 -1.5 3 96.3 98.5 -2.2 4 97.3 100.0 -2.7 4 94.5 94.5 0

3 1 102.8 100.3 2.5 7 1 86.4 85.4 1 2 101.5 101.4 0.1 2 86.8 86.7 0.1 3 102.6 104.3 -1.7 3 88.2 89.6 -1.4 4 101.9 101.9 0 4 88.6 89.0 -0.4

4 1 94.6 96.2 -1.6 8 1 90.5 90.5 0 2 97.8 95.5 2.3 2 89.1 90.2 -1.1 3 96.0 94.3 1.7 3 92.9 92.1 0.8 4 95.3 94.4 0.9 4 92.1 92.4 -0.3


Lecture 4.2 33

Graphical Analysis of Measurement Differences


Lecture 4.2 34

Average measurementsby Operators and Parts


Lecture 4.2 35

Graphical Analysis of Operators & Parts

Part

Measu

rem

ent

876543210

105

100

95

90

85

Operator

34

12

Interaction Plot


Lecture 4.2 36

Graphical Analysis of Operators & Ordered Parts

PartOrder

Meas

876543210

105

100

95

90

85

Operator

34

12

Interaction Plot, ordered by Parts


Lecture 4.2 37

Quantifying the variation

Notation:

E: SD of instrument error variation

P: SD of parts (manufacturing process) variation

O: SD of operator variation

OP: SD of operator by parts interaction variation

T: SD of total measurement variation

N.B.:

so

2E

2OP

2P

2O

2T

2E

2OP

2P

2OT


Lecture 4.2 38

Calculating sE

Part Operator Repeats ½(diff)2 Part Operator Repeats ½(diff)2

1 1 96.3 95.4 0.40 5 1 99.4 99.9 0.13 2 97.0 96.9 0.00 2 100.1 99.8 0.04 3 98.2 97.4 0.32 3 100.9 99.4 1.13 4 97.4 99.6 2.42 4 100.0 99.4 0.18

2 1 95.5 95.8 0.04 6 1 93.8 94.9 0.61 2 96.1 96.8 0.25 2 95.9 95.8 0.01 3 97.9 99.4 1.13 3 96.3 98.5 2.42 4 97.3 100.0 3.65 4 94.5 94.5 0.00

3 1 102.8 100.3 3.13 7 1 86.4 85.4 0.50 2 101.5 101.4 0.00 2 86.8 86.7 0.00 3 102.6 104.3 1.45 3 88.2 89.6 0.98 4 101.9 101.9 0.00 4 88.6 89.0 0.08

4 1 94.6 96.2 1.28 8 1 90.5 90.5 0.00 2 97.8 95.5 2.64 2 89.1 90.2 0.61 3 96.0 94.3 1.45 3 92.9 92.1 0.32 4 95.3 94.4 0.40 4 92.1 92.4 0.05

sum = 18.6 sum = 7.0

s2 = (18.6 + 7.0)/32 = 0.8

sE = 0.89


Lecture 4.2 39

Analysis of Variance

Analysis of Variance for Diameter

Source DF SS MS F P

Operator 3 32.403 10.801 6.34 0.003Part 7 1193.189 170.456 100.02 0.000Operator*Part 21 35.787 1.704 2.13 0.026

Error 32 25.600 0.800

Total 63 1286.979

S = 0.894427


Lecture 4.2 40

Basis for Random Effects ANOVA

F-ratios in ANOVA are ratios of Mean Squares

Check: F(O) = MS(O) / MS(O*P)

F(P) = MS(P) / MS(O*P)

F(OP) = MS(OP) / MS(E)

Why?

MS(O) estimates E2 + 2OP

2 + 16O2

MS(P) estimates E2 + 2OP

2 + 8P2

MS(OP) estimates E2 + 2OP

2

MS(E) estimates E2


Lecture 4.2 41

Variance Components

Estimated StandardSource Value Deviation

Operator 0.5686 0.75

Part 21.0939 4.59

Operator*Part 0.4521 0.67

Error 0.8000 0.89


Lecture 4.2 42

Diagnostic Analysis


Lecture 4.2 43

Diagnostic Analysis


Lecture 4.2 44

Measurement system capability

E P means measurement system cannot distinguish between different parts.

Need E << P .

Define TP = sqrt(E2 + P

2).

Capability ratio = TP / E should exceed 5


Lecture 4.2 45

Repeatability and Reproducibility

Repeatabilty SD = E

Reproducibility SD = sqrt(O2 + OP

2)

Total R&R = sqrt(O2 + OP

2 + E2)


Lecture 4.2 46

Laboratory 1, Part 2A four factor process improvement study

Low (–) High (+)

A: catalyst concentration (%), 5 7,

B: concentration of NaOH (%), 40 45,

C: agitation speed (rpm), 10 20,

D: temperature (°F), 150 180.

The current levels are 5%, 40%, 10rpm and 180°F, respectively.


Lecture 4.2 47

Design Point

Run Order

Catalyst Concentration

NaOH Concentration

Agitation Speed

Temperature Impurity

1 2 5 40 10 150 38 2 6 7 40 10 150 40 3 12 5 45 10 150 27 4 4 7 45 10 150 30 5 1 5 40 20 150 58 6 7 7 40 20 150 56 7 14 5 45 20 150 30 8 3 7 45 20 150 32 9 8 5 40 10 180 59 10 10 7 40 10 180 62 11 15 5 45 10 180 53 12 11 7 45 10 180 50 13 16 5 40 20 180 79 14 9 7 40 20 180 75 15 5 5 45 20 180 53 16 13 7 45 20 180 54

Design and Results


Lecture 4.2 48

Pros and Consof omitting "insignificant" terms

pro:

• the model is simplified

• the error term has more degrees of freedom so that s is more precisely estimated

• in small samples, comparisons are more precisely made


Lecture 4.2 49

Pros and Consof omitting "insignificant" terms

con:

• statistical insignificance does not imply substantive insignificance, so that

– when the excluded term has some effect below statistically significant level, the residual standard deviation is likely to increase, giving less precise comparisons,

– (although this may be a pro if conservative conclusions are valued)

• predictions may be slightly biased.


Lecture 4.2 50

Reading

EM §1.5.3, §7.5, §8.2.1

MS Introduction to Measurement Systems Analysis

BHH, §9.3

Documents

Diploma in Statistics Design and Analysis of Experiments Lecture 4.21 Design and Analysis of Experiments Lecture 4.2 Part 1: Components of Variation –identifying