Simplifying Measurement Uncertainties

Bill Hirt, Ph.D / February 2016

Start the Process

• Light the fuse (candle)

• Shed light on the way

2

Back to the beginning

• Accredited CALIBRATION labs report measurements on tools and devices needing regular service … typically including uncertainties for those measurements.

• Accredited TESTING labs, when requested or when needed to interpret Statements of Compliance, report uncertainties alongside their test measurements.

3

Framework for uncertainties

• Unless calculating MDL’s or LOD’s or LOQ’s …

• It is presumed that ISO 17025 accredited labs demonstrate the competence to calculate and report GUM uncertainties … estimated at roughly a 95 % confidence.

4

When do accredited labsreport MU’s ?

• Every accredited calibration certificate, unless requested otherwise

• Test reports :

• When a customer requests it

• When the uncertainty affects compliance with a specification limit

• When it is relevant to the validity or application of the result

5

What do we mean by MU?

• Measurement value ± uncertainty (MU)

• MU usually at 95 % confidence – Why?

• MU usually reported at k = 2

6

7

Where do we find most MUs?

• On calibration certificates

• On test reports

• In footnotes

8

Did you know that …

• A high percentage of ISO 17025 accredited calibration laboratories issue incorrect uncertainties on their certificates ?

9

ISO 17025 includes –Calibration and Testing

• A good percentage of calibration labs do not report MU’s on their certificates

• Most testing labs do not report MU’s on their test reports

• And WHY ???

10

First Group Discussion

• Scenario used in 3-day MU course outlined first

• Have groups discuss other real-life scenarios where uncertainty can be critical or life-saving

• Put examples on index cards on the tables

11

Uncertainty and Traceability• Metrological traceability (not sample

traceability)

• Two platforms of confidence

• Comparison with hi-quality stds thru chain to SI through an NMI or DI

• Confidence that GUM unc’s used … and actual measurement error clearly known

12

Uncertainty and Proficiency Testing (PT)

• Calibration PT programs typically request that one or more devices have measurements taken … and Both the measurements and their uncertainties be reported.

• The Cal PT study issues a report with standardized results including an En

13

En value Determination (Calibration Labs)

Interpretation of En values

Accredited Testing Labsand PT

• Very few ISO 17025 accredited PT programs for testing include a requirement for MU’s

• Some now request them

• Many in the future will require them

• In addition to z-scores, many future reports will include En values too

16

17

“The Big Three” in 17025

• Measurement Uncertainty

• Metrological Traceability

• Proficiency Testing

18

Uncertainty and Statements of Compliance• High percentage of measurements are made

to ensure manufactured or natural materials meet a narrow range of specifications

• Manufacturers and regulators define specs

• No measurement is perfect

• MU may or may not be critical to have CONFIDENCE that specs are being met

19

Group discussion - 2

• At your tables, discuss specific areas of compliance specs that you are aware of

• Make a list of at least 6 different groups or types of specs at your table to report later

• The group will share after your discussion

20

Uncertainty and Guard Banding

• Management of manufacturing or other monitoring to assure material is safely within specifications, including consideration of uncertainty

• Organization may adjust their spec to capture either more potentially defective samples or allow more acceptable product out the door

21

22

23

Definitions • Uncertainty – a property of a measurement result

that defines the range of probable values of the measurand

• Uncertainty budget – the systematic description of uncertainty determinations relevant to specific measurements including ranges plus all factors, assumptions and calculations included (must include both type A and type B factors)

24

Basic Steps in Uncertainty Budgets

1. List ALL potential factors affecting variability in measurements - make table

2. Determine the standard uncertainty for each factor (includes distribution)

3. Perform RSS for all factors to create the combined (standard) uncertainty

4. Multiply by distribution factor (k=2 … or ?)

25

Root Sum Squaring

26

Sample Standard Deviation(s)

Calibration MU example

Budget for 6 inch CALIPERS

i Component of UncertaintyUncertainty,

U(xi) Distribution Divisor Std Unc, u(xi)

1 Standard uncertainty 6 Normal, 2s 2.00 3 uin2 Resolution 500 Rectangular 1.73 289 uin3 Repeatability 297.98 Normal, 1s 1.00 298 uin4 Uncompensated error 25 Rectangular 1.73 14.4 uin5 Temperature difference between

instrument and gage blocks20.1 U-Shaped 1.41 14.2 uin

6 Temperature variance from 68º F 16.08 U-Shaped 1.41 11.4 uin7

combined standard uncertainty, uc 416 uincoverage factor, k 2

expanded uncertainty, Uc 832 uinExpanded uncertainty rounded UP to 2 significant figures 840 uin

27

Second Cal MU example

Uncertainty Budget for Bench Scale - 150 lb capacity

i Component of UncertaintyUncertainty,

U(xi) Distribution Divisor Std Unc, u(xi)

1 Standard uncertainty 0.000106 Normal, 2s 2.00 0.000 lb2 Resolution 0.05 Rectangular 1.73 0.029 lb3 Uncompensated error (Three 50 #

weights)0.0105 Rectangular 1.73 0.006 lb

4 Repeatability 0.0217 Normal, 1s 1.00 0.02 lb

combined standard uncertainty, uc 0.0366 lb

coverage factor, k 2

expanded uncertainty, Uc 0.0732 lb

Expanded uncertainty rounded UP to 2 significant figures 0.074 lb

28

Testing MU

• Very often much more complicated than calibration -- why ??

• Many stages in test processes

• Many error types with different units of measure

• Many errors not defined and require guess

29

30

31

Table A6.2: Summary of results from collaborative trial of the method and in-house repeatability check

32

Table A6.4: Combined standard uncertainties

33

Key current and future factor

• SAMPLING and sample error factors

34

Nitrogen in Forage Budget

35

Nitrogen in Dry Feed Budget

36

New proposed TABLE

• Handout shows old / traditional version of the Student’s T Table

• Back side shows our proposed new version

• Let’s review its features

37

Open quiz for the room

• Using the new Student’s T table, what is the k-factor … for an MU … at 95% confidence … when the number of repeatability measurements is :

• 3 ?

• 5 ?

• 10 ?

• 30 ?

38

Open quiz for the room - 2

• … and the test measurement mean is 100 mg and the combined uncertainty is 5 mg, what is each 95% MU :

• With 3 repeats

• With 5 repeats

• With 10 repeats

• With 30 repeats

39

• For an MU … at 95% confidence … when the number of repeatability measurements is :

• 3

• 5

• 10

• 30

Open quiz for the room - 3

# of RepeatMeasurements

k-factorMultiplier

Uexp (95%)To report

3

5

10

30

( test measurement mean is 100 mg and the combined uncertainty is 5 mg )

40

Open quiz for the room – 3+

# of RepeatMeasurements

k-factorMultiplier

Uexp (95%)To report

3 4.3 21.5

5 2.78 13.9

10 2.26 11.3

30 2.0 10

41

Example of questionable MU on

commercial calibration certificate

42

Standard Deviation of the Mean

• The equation on the left for repeatability / SD becomes

43

• Caution – it does NOT replace repeatability SD

44

Control Charts

• Plots of long-term measurements of a single parameter to note trends or variability

• Often the main basis for testing uncertainties

• Main testing lab equivalent of repeatability and/or reproducibility

45

46

47

AAFCO example of Control Chart of Matrix Spike

48

49

Basic Steps in Uncertainty Budgets

1. List ALL potential factors affecting variability in measurements - make table

2. Determine the standard uncertainty for each factor (includes distribution)

3. Perform RSS for all factors to create the combined (standard) uncertainty

4. Multiply by distribution factor (k=2 … or ?)

50

Group discussion - 3

• We want to simplify the process for MU determination

• We want the fundamentals of uncertainties to be understood

• We know that may tests may be confusing even as to whether an MU is needed

• Discuss at your tables and list confusing examples … for our larger discussion

51

52

Fundamental MU Nuggets

• Key definitions to be able to distinguish:

• Type A and Type B factors

• Repeatability vs reproducibility

• Combined uncertainty vs expanded uncertainty (crucial k-factor to use)

• In unc budgets, any factor contributing less than 10% to the total can be eliminated

53

Fundamental MU Nuggets - 2• Use at least 7 data points for any standard

deviation

• Try to use at least 30 data points (equivalent to infinite number) in Type A factors

• Metrological traceability and the traceability chain needs GUM uncertainties

• Calibration PT studies involve uncertainties and testing PT studies may do so soon (Enand z-scores)

54

Key Nuggets - 3• Repeatability/reproducibility studies may not

adequately cover the range of test tolerances. You may need a high and low range repeatability study.

• Be careful not to replace repeatability standard deviation factor with standard error of the mean in a MU budget. Add not replace.

Key Nuggets - 4• Control charts can often capture full testing

errors, but use caution to be sure ALL significant errors are included

• Repeatability studies or reproducibility studies can often represent the basis for MU determinations

• For any test, consider all potential errors but budgets and calculations may NOT be needed

55

When may MU budgets and calculations not be needed?• If alternative statistics used, eg MDLs, LODs

• If tests qualitative

• If test protocols already define testing variability, precision, repeatability

• This includes 95% confidence determinations for microbial MPN studies

56

Remember …

• No measurement is perfect

• Measurement errors historically combined as Uncertainties … BUT ….

• Uncertainties trigger too often – confusion

• 95% confidence is the basis for uncertainties

57

Final Mantra …

• (not) UNCERTAINTY …

• (but) … CONFIDENCE RANGE !!

58

Thank you

59