MER301: Engineering Reliability

L Berkley DavisCopyright 2009

MER301: Engineering ReliabilityLecture 16

1

MER301: Engineering Reliability

LECTURE 16:

Measurement System Analysis and Uncertainty Analysis-Part 1



We must submit the output from our design process to a second (measurement) process

MeasurementProcess

Outputs• Measurements

ProcessInputs Outputs

Inp

uts

Parts(Example)

Measurement as a Process

2



3

Measurement System Concerns..

How big is the measurement error? What are the sources of measurement error? Are the measurements being made with units

which are small enough to properly reflect the variation present?

Is the measurement system stable over time? How much uncertainty should be attached to a

measurement system when interpreting data from it?

How do we improve the measurement system?



4

Measurement System Analysis Total Error in a measurement is defined as the difference

between the Actual Value and Observed Value of Y Two general categories of error – Accuracy or Bias and

Precision Accuracy or Bias of Measurement System is defined as the difference

between a Standard Reference and the Average Observed Measurement Precision of a Measurement System is defined as the standard deviation of

Observed Measurements of a Standard Reference Total Error = Bias Error + Precision Error for independent random variables

Measurement System Error is described by Average Bias Error (Mean Shift)and a statistical estimate of the Precision Error (Variance)

Measurement System Analysis is a Fundamental Part of Every Experiment



5

Not Accurate, Not Precise Accurate, Not Precise

Not Accurate, Precise Accurate, Precise

Precision and Accuracy



6

Measurement System Analysis

Bias or Accuracy error is a constant value and is dealt with by calibrating the measurement system

Variation or Precision error is a random variable which depends on the measurement equipment(the instruments used) and on the measurement system repeatability and reproducibility. Instrument Capability Analysis, Test/retest (repeatability)and Gage R&R studies are used to quantify the size of these errors.

222 0

0

tmeasuremenactualobserved

biasactualobserved

tmeasuremenbiasactualobserved YYY



Actual Defects

LSL USL

= Product Std. Dev.

= Product Mean

LSL USL

Observed Defects

Measurement system variance

Product variance

Impact of Measurement System Variation on Variation in Experimental Data

22mactualobs

actual

actual

actual

obs observedobs

tmeasuremenm

actualact

7



8

Example 16.1-Effect of Measurement System Variation

Calculate the effect of measurement system variation on the acceptance rate for a part with USL and LSL at Z= +/-1.96 respectively. If then what is the percentage of acceptable parts that will be rejected? If on the other hand what is the percentage of acceptable parts that will be rejected?

2/221 actm

20/222 actm

LSL USL

T

Process

Gauge

Failed Goodunits

Gauge

Passed BadUnits



9

Impact of Measurement System Variation on Variation in Experimental Data…

LSL USL

T

Process

Gauge

Failed Goodunits

Gauge

Passed BadUnits



10

Example 16.1(con’t: ) For the product, the spec limits of +/-1.96 mean that the

2.5% of parts in each tail are out of spec. Thus

For the observed standard deviation is

and

Then the acceptable parts now rejected are

act

actuslusl

XZ

96.1

2/221 actm

2/32/22221 actactactmactobs

60.12/3

96.1

2/3

)96.1(,

act

actactactObserveduslZ

060.0945.0975.0260.196.12

actactuslX 96.1

2/221 actm



11

Example 16.1(con’t: ) For the product, the spec limits of +/-1.96 mean that the



and


act

actuslusl

XZ

96.1

20/222 actm

actactactactobs 0247.120/1120/22

913.10247.1

96.1

0247.1

)96.1(,

act


006.0972.0975.02913.196.12

actactuslX 96.1

20/222 actm



12

Example 16.1(con’t)

Process2

2 2Measure

Process2

2 20Measure

Measure

Measure

Process

Process

Observed

Observed

Set Measurement System Requirements Based on the Process Variation

Unacceptable

Acceptable



Gage Repeatability & Reproducibility ---GRR or GR&R---

Gage Repeatability & Reproducibility compares measurement system variation and product variation

The term is the size of an interval containing 99% of the measured values made on a specific item

The Tolerance- often equal to - is the size of the interval where a product has acceptable dimensions, performance, or other characteristics

%100% 15.5 Tolerance

measurmentGRR

tmeasuremen15.5

actual6

13



Gage Performance relative to required Tolerance Band

R&R less than 10% - Measurement system is acceptable. R&R 10% to 30% - Maybe acceptable - make decision

based on classification of characteristic, hardware application, customer input, etc.

R&R over 30% - Not typically acceptable. Find the problem using root cause analysis(fishbone), remove root causes

GRR is a measure of “noise” in the data GRR is a measure of “noise” in the data


measurmentGRR

14



15

Effect of Gage R&R on Variation

GRR <10% means < 0.7% of the variation in the experimental data is from the measurement system

GRR> 30% means that > 5.9% of the variation in the experimental data is from the measurement system

%1006

15.5%

actual

tmeasuremenGRR

2

222

)15.5/6(1/ GRRactualobserved




16

GRR Example 16.2

607.06

2/15.5

6

2/15.5

6

15.51

1

act

act

act

mGRR

192.06

20/15.5

6

20/15.5

6

15.52

2

act

act

act

mGRR

The GRR values for the previous Example 16.1 are

The capabilities of two (or more) measurement systems can be compared by comparing the GRR’s for each. Since GRR2<GRR1 , the second measurement system is more capable than the first. The observed standard deviations quantify how much better….

20/222 actm

2/221 actm

195.1/21obsobs 0247.1/

225.1/

2

1

actobs

actobs

8365.0/12obsobs



17

GRR Example 16.2

607.06

2/15.5

6

2/15.5

6

15.51

1

act

act

act

mGRR

192.06

20/15.5

6

20/15.5

6

15.52

2

act

act

act

mGRR

The GRR values for the previous Example 16.1 are at best marginally acceptable(GRR2 ) or not acceptable(GRR1 )

For a GRR value equal to 10% (0.10) there results

20/222 actm

2/221 actm

74/167.73/1)15.5/610.0(/6

15.510.0 222

actmact

mGRR



18

Example 16.2 ( so ) For the product, the spec limits of +/-1.96 mean that the



and


act

actuslusl

XZ

96.1

74/22actm

actactactactobs 0067.174/1174/22

947.10067.1

96.1

0067.1

)96.1(,

act


0016.09742.09750.02947.1960.12

actactuslX 96.1

74/22actm 10.0GRR



19

Summarizing how it all fits together…..

When a set of measurements are made, the results are always observed values,

If the actual mean and standard deviation are known then the measurement system bias and variance can be calculated

If the item being measured is a standard reference

If the measurement system bias and variance are known then the actual mean and actual variance can be calculated

mbiasactobs YYY 222 0 mactobs

actobsbias 222actobsm

222mobsact

022 obsm

0 biasactobs

biasobsact


measurmentGRR



Total Variation made up of Actual Process Variation and Measurement

System Variation

Total Variation made up of Actual Process Variation and Measurement

System Variation

Sources of Measurement System Error

ProcessInputs Outputs Inputs MeasurementProcess

Outputs

• Observations• Measurements

Long-term Process Variation

Actual Process Variation

Accuracy (Bias)

Accuracy (Bias)

Measurement VariationMeasurement Variation

Observed Process Variation

Short-term Process Variation

Variation due to gauge

Variation due to gauge

Variation due to operator

Variation due to operator

Precision (Pure Error)

Precision (Pure Error)

Stability (time dependent)

Stability (time dependent)

Linearity (value dependent)

Linearity (value dependent)

RepeatabilityRepeatability

ReproducibilityReproducibility

within sample variation

within sample variation

MeasurementSystem

Repeatability

Resolution

20


Engineering ReliabilityLecture 16

21

Measurement System Errors

Repeatability (precision)

Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)

Observed Average

(Low End)

Observed Average

(High End)

Linearity



22

Elements that contribute to Accuracy and Precision Errors

Instrument Capability Resolution Gage Repeatability Linearity

Measurement System - Short Term (ST) Instrument Capability Equipment Calibration(Bias) Test/Re-Test Study(Repeatability)

Measurement System - Long Term (LT) Use Measurement System - Short Term Use Reproducibility Stability

First Two are Entitlement….Third is Reality

Union CollegeMechanical Engineering


21

Gage Performance Characteristics


Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)

Observed Average(Low End)

Observed Average

(High End)

Linearity



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



23

Elements that contribute to Precision or Variation Errors


Measurement System- Short Term (ST) Use Instrument Capability Equipment Calibration(Bias) Test/Re-Test Study(Repeatability)

Measurement System - Long Term (LT) Use Measurement System - Short Term Use (ST) Reproducibility(Gage R&R) Stability(Gage R&R)


2instrument

222, ityrepeatibilinstrumentSTtmeasuremen

22222ilityreproducibityrepeatabilinstrumentLTm



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



24


2222

222

0

0

0

ilityreproducibityrepeatabilinstrumenttmeasuremen

Y



biasactualobserved

tmeasuremenbiasactualobserved

tmeasuremen

YYYY

YYY

From pages119-120…



25

Updating how variances all fit together

When a set of measurements are made, the results are always observed values,

If the actual mean and standard deviation are known then the measurement system bias and variance can be calculated

If the item being measured is a standard reference

If the measurement system bias and variance are known then the actual mean and actual variance can be calculated

mbiasactobs YYY 2222222 0 ilityreproducibityrepeatabilinstrumentactmactobs

222mobsact

22222 0 ilityreproducibityrepeatabilinstrumentobsm

222222ilityreproducibityrepeatabilinstrumentactobsm

)( 22222ilityreproducibityrepeatabilinstrumentobsact

biasactualobserved

biasobsact



26

Emissions Sampling

NOxInstrument Yactual-

NOx fromGas turbine

Cal/ZeroGases

Yobs- NOx Reading

Heated Sampling Line

Calibration Gas

Sample Conditioning


222tmeasuremenactobs

biasactobs



27


Instrument CapabilityResolutionGage RepeatabilityLinearityMeasurement System- Short Term(ST) UseInstrument CapabilityEquipment CalibrationTest/Re-Test StudyMeasurement System- Long term (LT) UseMeasurement System -Short Term(ST) Use ReproducibilityStability 2222




29

Emissions Sampling

NOxInstrument Yactual-

NOx fromGas turbine

Cal/ZeroGases

Yobs- NOx Reading

Heated Sampling L ine

Calibration Gas

Sample Conditioning


222tmeasuremenactobs

biasactobs



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



28

Establish magnitude and sources of

measurement system error due to bias and precision errors

Tools Instrument Capability Analysis Test/Re-test – system precision/repeatability Calibration - bias “Continuous Variable” Gage R&R (Gage

Reproducibility and Repeatability) Attribute Variable Gage R&R Destructive Gage R&R

How Can we Address Accuracy and Precision Errors?



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



29


Instrument Capability Analysis….. Resolution-smallest increment that the gage can resolve in the

measurement process. Gage should be able to resolve tolerance band into ten or more parts. Resolution Uncertainty =

Instrument Accuracy- measure of instrument repeatability or instrument “noise”.. Found by repeated measurements of the same test item. Uncertainty =

Linearity- consistency of the measurement system across the entire range of the measurement system. Linearity Uncertainty =

The variations are combined as follows

00 4 u

rru 4

llu 4

2222

2222

lroinstrument

lroinstrument uuuu



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



30

Instrument Capability Analysis….. Variation for any one instrument equals the sum of the

resolution, repeatability and linearity terms

The Variation for “n” instruments equals the sum of the variations for each individual instrument

Each of the “n” instruments has resolution, repeatability, and linearity terms that must be taken into account

2222linearityityrepeatabilresolutioninstrument

linearityityrepeatabilresolutioninstrument YYYY

222221

21

n

n

instrumentinstrumentinstrumentinstrument

instrumentinstrumentinstrumentinstrument YYYY



31

Instrument Capability Analysis - Resolution of Instruments/Sensors

The measurement uncertainty due to resolution is generally taken as a specified fraction of the smallest increment an instrument can resolve, ie as a fraction of the smallest scale division

General Rule: assign a numerical value for the mean value of equal to one half of the instrument resolution. This means

that half of the smallest scale division is assumed to equal a 95% Confidence Interval ( a wide band) for variation due to resolution

resolution 4 21

0 ou

o4

0u

0u



32

Instrument Capability Analysis Repeatability and Linearity

The manufacturer of an instrument will provide information on the capability of the instrument in the specification sheets provided with the instrument

The numerical values given for Instrument or Sensor Accuracy and Linearity are almost always uncertainties Let = uncertainty due to the equipment

accuracy/repeatability error where Let = uncertainty due to linearity error

where The inherent capability/uncertainty of the instrument/sensor

is then estimated as:

222

lroinstrument uuuu

rurru 4

lullu 4



33

Example 16.3-Instrument Capability The Capability of a force measuring instrument is

described by catalogue data. Calculate an estimate of the variation attributable to this instrument. Express the result both in dimensional terms (N) and in dimensionless terms for a reading R=50N

Resolution 0.25NRange 0 to 100NLinearity within 0.20N over rangeRepeatability within 0.30N over range



34

Example 16.3(con’t) An estimate of the instrument uncertainty depends

on the combined uncertainties due to resolution, repeatability and linearity

The instrument uncertainty is then

0.0076 50

38.0

0.38N)125.0()3.0()2.0( 222

N

N

R

uu

u

dinstrument

instrument

Nu rr 30.04

N 125.02/25.04 oou

Nu ll 20.04



35


Instrument Capability Analysis Summary….. Resolution-smallest increment that the gage can resolve in the

measurement process. Gage should be able to resolve tolerance band into ten or more parts. Resolution Uncertainty =

Instrument Accuracy- measure of gage repeatability or gage “noise”.. Found by repeated measurements of the same test item. Uncertainty =

Linearity- consistency of the measurement system across the entire range of the measurement system. Linearity Uncertainty =

The variations are combined as follows

00 4 u

rru 4

llu 4

2222

2222

lroinstrument

lroinstrument

uuuu



36


Measurement System Short Term Use Includes Instrument Capability Repeatability - variation when one operator repeatedly

makes the same measurement with the same measuring equipment Test/Re-test Study

Calibration/Bias

Measurement System-Long Term Use Includes Measurement System –Short Term Use Reproducibility- variation when two or more operators make

same measurement with the same measuring equipment Stability-variation when the same operator makes the same

measurement with the same equipment over an extended period of time



37

Test/Retest Example 16.4 Test/Retest (Repeatability) Study on a

Measurement System. Thirty repeat measurements were taken on a Standard Reference Item with a thickness of 50mils The tolerance band for the application is 20mils(+/-10).

Data, in mils 53,45,52,47,54,52,52,55,52,48,48,53,55,51,47,52,47,35,

45,54,48,51,53,44,52,52,55,59,53,53



38

Example 16.4(con’t) Objective is to establish the precision and accuracy

of the measurement system Precision-Repeatability

In a good Measurement System, 99% of the measurements of a given item should fall within a band less than

1/10 of tolerance band

Accuracy/Bias Bias = sample mean- true value

LSL USL

T

Process

Gauge

Failed Goodunits

Gauge

Passed BadUnits

10/115.5

tolerance

GRR tmeasuremen

actualobserved YYbias



39

Example 16.4 Run Chart and Histogram

10 20 30

40

50

60

Observation

C1

Number of runs about median:Expected number of runs:Longest run about median:Approx P-Value for Clustering:Approx P-Value for Mixtures:

Number of runs up or dow n:Expected number of runs:Longest run up or dow n:Approx P-Value for Trends:Approx P-Value for Oscillation:

13.000014.9333 6.0000 0.2190 0.7810

19.000019.6667 3.0000 0.3829 0.6171

Run Chart for C1

35.0 37.5 40.0 42.5 45.0 47.5 50.0 52.5 55.0 57.5 60.0

0

5

10

C1

Fre

que

ncy

These results look bad to the eye…there are outliers and mean is high



40

Test/Retest Study Example 16.4 Summary

Descriptive Statistics

Variable N Mean Median StDev SE Mean

C1 30 50.567 52.000 4.561 0.833

Variable Minimum Maximum Q1 Q3

C1 35.000 59.000 47.750 53.000

Conclusions Given the tolerance band of 20 mils,there is an

unacceptable level of device precision

Given the Reference Test item had a known thickness of 50mils, the bias(inaccuracy) is:

bias = 50.57 – 50.0 = 0.57mils

)10/1(174.120/48.2320/15.5 tmeasuremenGRR

bias

bias



45

Not Accurate, Not Precise Accurate, Not Precise

Not Accurate, Precise Accurate, Precise

Example 16.3- Precision versus Accuracy



41


Measurement System-Short Term Use Repeatability-variation when one operator repeatedly

makes the same measurement with the same measuring equipment Test/Re-test Study

Measurement System - Long Term Use Reproducibility- variation when two or more operators

make same measurement with the same measuring equipment

Stability-variation when the same operator makes the same measurement with the same equipment over an extended period of time



42



Measurement System - Short Term (ST) Instrument Capability Equipment Calibration(Bias) Test/Re-Test Study(Repeatability)

Measurement System - Long Term (LT) Use Measurement System - Short Term Use Reproducibility Stability




21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



21



Reproducibility

Operator B

Operator A

Stability

Time 1

Time 2

Observed Average

Accuracy (Bias)

True

True Average

True Average

Accuracy(Low End)

Accuracy(High End)


Observed Average

(High End)

Linearity



43

Measurement System Analysis: A Summary of the Basic Equations

22222

2222

222

2222

222 0

0

ilityreproducibityrepeatabilinstrumentactualobserved

ilityreproducibSTtmeasuremenLT

ityrepeatabilinstrumentST



biasactualobserved


Documents

MER301: Engineering Reliability