AOAC Official Methods of AnalysisSM March 28, 2011, the AOAC INTERNATIONAL Board of Directors approved an alternative path to achieve an Official Method (Official First Action status)

AOAC Official Methods of

AnalysisSM

Darryl Sullivan, Covance Laboratories and Past President, AOAC INTERNATIONAL

On March 28, 2011, the AOAC INTERNATIONAL

Board of Directors approved an alternative path to

achieve an Official Method (Official First Action status)

for methods selected and reviewed using the AOAC

volunteer consensus standards development

processes.

How this change came about…

Rationale for Change

•AOAC’s ability to validate fully collaboratively

studied methods has been steadily on the decline,

approving only three in 2010.

•AOAC was repeatedly disappointing customers

and communities who needed methods to solve

problems.

•AOAC already had a reputation of being slow and

old with cumbersome processes – was potentially

facing decline in the confidence of our brand.


•AOAC has evolved and now acts as a problem solver

through a broader process of consensus building and

standards development.

•AOAC is trying to meet community/customer needs

by gathering the world’s authorities to articulate and

evaluate methods needs through expertise and

judgment.

•AOAC decided to find a way to give proper weight to

the confidence we have in the judgment and collective

knowledge of our experts.

•AOAC decided to align our brand and method output

closer to our proven standards development processes

The “new” or “alternate” path…


How it Works – At a Glance

Funded Stakeholder Panel

Working Groups to establish Standard Method Performance Requirements (SMPRs)

Expert Review Panels to adopt methods as Official First Action based upon performance against SMPRs

How it Works: The Details

Expert Review Panels

Must be supported by relevant stakeholder body

Membership is carefully managed and properly vetted by

the AOAC Official Methods Board

Holds transparent public meetings only

Remains in force to monitor methods as long as method is

in First Action Status.

How it Works: The Details Official First Action Status Decision

Method adopted by ERP must perform adequately against

the SMPR set forth by the stakeholders

Method becomes Official First Action on date when ERP

decision is made.

How it Works: The Details Official First Action Status Decision

Methods to be drafted into AOAC format by a

knowledgeable AOAC staff member or designee in

collaboration with the ERP and method author.

Report of decision complete with ERP report regarding

decision including scientific background (references etc) to

be published concurrently with method in traditional

AOAC publication venues


Transition to Final Action Status

ERP will monitor performance and data submitted for two

years

Further data indicative of adequate method reproducibility

performance to be collected. Data may be collected via a

collaborative study or by proficiency or other testing data

of similar magnitude


Transition to Final Action Status

Removed from Official First Action and OMA if no evidence

of method use or no data indicative of adequate method

reproducibility is forthcoming

ERP makes recommendations to the Official Methods Board

(OMB)

OMB renders decision on transition to Final Action Status

Expected Benefits

More Official Methods of Analysis generated

We can provide solutions faster and take full advantage of

collective expertise of AOAC members

Methods can be put into regular use right away – generating

more useable data to evaluate performance

Expected Benefits

OMA can be more flexible – if a method is not performing

up to ERP expectations it is removed

The transition from first to final action becomes more

meaningful and dynamic, more credence is given to final

action methods.

Time for Change

AOAC will convene its first Expert Review Panel charged

with adopting Official First Action Methods of Analysis this

afternoon.

Many more ERPs will follow, developing out of the many

stakeholder activities going on at AOAC

Questions and Comments?

Thank you!

Standard Method Performance Requirements [SMPR] Guideline

Gaithersburg, Maryland, USA

Thursday June 30, 2011

Background

• First written in 2010.

• First used for the Endocrine Disruptor Compounds (EDC) project in 2010.

• Reviewed and revised by Official Methods Board in 2010.

• Still in review, but also in use.

Background• Resulted when AOAC staff started a project

create a standard SMPR format.

• It was realized that:

• If the SMPR format required certain parameters (i.e. “recovery”) then a definition was needed.

• If we defined parameters then we needed to offer guidance on how to collect data.

Background

• If we defined parameters then we needed to offer guidance on how to collect data.

• If we offered guidance on how to collect data then we needed to provide guidance on acceptance criteria.

• If we offer guidance on acceptance criteria then we needed to explain the concept.

• So 2 pages turned into . . .

Background

. . . into 13 pages with 14 pages of appendices.

But . . . it a single, comprehensive document with good information all in one place for many different kinds of methods.

Philosophical Direction• An attempt to bring to together several

different AOAC technical documents.

• OMB Guidelines

• Microbiology Methods Guideline (Appd. X)

• AOAC Single Laboratory Guideline

• BTAM Guideline

• Best Practices for Microbiological Method (BPMM) Validation

Philosophical Direction

• An attempt to bring to together several different types of methods:

• Chemistry

• Microbiology

• Qualitative

• Quantitative

• Identity

Components of the Guideline

1. SMPR Format

2. Recommended Performance Requirement Parameters

3. Definitions

4. Recommendations for Evaluations

5. Explanations

6. Appendices

Architecture of Performance Requirements in SMPR GuidelineClassification of methods

Quantitative / Qualitative

Main component / trace (contaminant)

Identification method.

Type of data

single laboratory

independent

collaborative study

Big Notes!

• No distinction made between microbiology and chemistry!

• Not intended to require

SLV → independent lab → collaborative

• Identification methods separated from qualitative methods,

Classifications of Methods9

Quantitative Method( main

component1)

Quantitative Method(trace or

contaminant2)

Qualitative Method(main component1)

Qualitative Method(trace or

contaminant2)

Identification Method

Para

met

ers Si

ngle

labo

rato

ry

valid

atio

nIn

de-

pend

ent

Col

labo

rativ

eSt

udy

Classifications of Methods9

Qualitative Method(main component1)

Qualitative Method(trace or

contaminant2)Identification Method

Para

met

ers

Sing

le

Labo

rato

ryva

lidat

ion

Reference Method ComparisonInclusivity/SelectivityExclusivity/Specificity

Environmental InterferenceLaboratory Variance

BiasProbability of Detection

Reference Method ComparisonInclusivity/Selectivity Exclusivity/Specificity

Environmental InterferenceLaboratory Variance

BiasProbability of Detection (POD) at the

AMDL

Reference Method ComparisonInclusivity /SelectivityExclusivity/Specificity

PrecisionEnvironmental Interference

Bias

Inde

pen-

dent TBD5 Probability of Detection (POD)

at the AMD Bias

Col

labo

ra-

tive

Stud

y POD (0)POD (c)

Laboratory Probability of Detection8

POD (0)POD (c)

Laboratory Probability of Detection

POD (0)POD (c)

Laboratory Probability of Detection

Parameters

• Reference Method Comparison• Inclusivity/Selectivity• Exclusivity/Specificity• Environmental Interference• Laboratory Variance• Bias• Probability of Detection (POD)

Inclusivity/Selectivity

• Definition: Strains or isolates or variants of the target agent(s) that the methodcan detect.

• Recommendation: Analyze one test portion containing a specified concentration of one inclusivity panel member. More replicates can be used.

Exclusivity/Specificity• Definition: Strains or isolates or variants of

the target agent(s) that the method must not detect.

• Recommendation: Analyze one test portion containing a specified concentration of one exclusivity panel member. More

Bias

• Definition: The difference between the expectation of the test results and an accepted reference value. Bias is the total systematic error as contrasted to random error. There may be one or more systematic error components contributing to the bias.

• No recommendations.

Probability of Detection (POD)

• Definition: The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given analyte level or concentration. Already discussed.

Probability of Detection (POD)

• Recommendations: Determine the desired Probability of Detection at a critical concentration. Consult with table 7 to determine the number of test portions required to demonstrate the desired Probability of Detection.

No definitions or recommendations

• Reference Method Comparison• Environmental Interference• Laboratory Variance

Summary• SMPR summarized a variety of AOAC

guidelines.

• SMPR is comprehensive, but not detailed.

• SMPR includes chemistry and microbiology; quantitative and qualitative.

• Work in progress.

Chi-Square Statistics in Method Validation

ISPAM Microbiology and Chemistry Working Groups 30 June, 2011

Dan Tholen, M.S.

Chi-Square and Related Issues

Different designs Statistical estimators vs. statistical tests McNemar Chi-Square test in ISO 16140 Related estimators and tests Relationship to POD and dPOD

Estimators vs. Hypothesis Tests

Estimators provide best estimates of parameters of interest, based on design – Accuracy, Sensitivity, Specificity, POD, etc.

Hypothesis tests provide advice on whether differences in estimators could have occurred by chance – Assumes a statistical distribution – Requires consideration of Type 1 and 2 error – Requires decision level

Estimators vs. Hypothesis Tests

Estimators should be accompanied by confidence intervals that show a range of values that could be the „correct‟ value – Similar to measurement uncertainty

Hypothesis tests usually come with a „reject‟ or „do not reject‟ decision, and

perhaps with a „p value‟ which is a

likelihood for the evidence if H0 is not true – Not as informative as C.I. or MU

Chi-Square Analysis

Recommened in ISO 16140, and in proposed CD 16140:2011 – And in protocols influenced by ISO 16140

Used for testing equivalence of methods by looking at discordant results Very powerful technique, often based on only a few discordances out of hundreds of agreements (that is, small differences between methods can be significant)

Comparative Accuracy - 16140

Separate study for several categories of food (up to 5 categories) Select at least 3 types of food from each category Select at least 20 samples representative of each type – Independent samples, not replicates – Ideally 10 negative, 10 positive

Test each sample with both methods

Two Way Designs - 16140

Unpaired: from the same sample, but separate test portions Paired: from the same sample and shared first step in the enrichment procedure

From same enrichment medium (microbiology) From the same extraction (chemistry)

2x2 Layout – Paired (and ISO 16140 unpaired)

Method B

(Reference)

Total

Method A (Alternative)

Present Absent

Present a b NA+

Absent c d NA-

Total NB+ NB-

N

Estimators - ISO 16140 Paired

Relative accuracy: AC = (a+d)/N Relative specificity: SP = (d)/NB- Relative sensitivity: SE = (a)/NB+

Sensitivity (altern): SEalt = (a+b)/(a+b+c) Sensitivity (refrnc): SEref = (a+c)/(a+b+c) Alternative method results confirmed for reference method negatives (cells b & d)

Estimators – 16140 Unpaired

Relative accuracy: AC = (a+d)/N Relative specificity: SP = (d)/NB- Relative sensitivity: SE = (a)/NB+ – These estimators are same as for paired

All alternative method results are confirmed, estimators are listed separately for confirmed and unconfirmed results

Chi-Square Test – ISO 16140

Only McNemar test is discussed 2 = |b-c|2 / (b+c) (1 degree of freedom)

Considers only discordant results

Other estimators not tested, McNemar is considered the most sensitive test to rule out differences due to random error Requires minimum size, b and c (b+c>22) Often the exact test (Binomial) is used for small size samples, or in all cases

Chi-Square Test – ISO 16140

Test for significant difference in proportion of positives – P+A = (a+b)/N – P+B = (a+c)/N

Since P+A and P+B both use a, the proportions are correlated Most sensitive test is for whether b and c are statistically different – Binomial, p=0.5 n=b+c

Equivalence with POD Concept

P+A: PODA = (a+b)/N P+B: PODB = (a+c)/N P+A - P+B = dPOD Test of dPOD using the Binomial (for H0: dPOD=0) is the same as the McNemar – Large and small numbers of tests – Both paired and unpaired – For single lab or multi-laboratory studies

Note on Nordval

Nordval (May 2010), for Qual. Chemistry Uses same 2x2 layout, for paired data Does not use Chi-Square. Uses Kappa, a measure of agreement that “corrects” for random agreement The extent to which agreement exceeds chance agreement is a measure of concordance Nordval recommends agreement > 80%

Note on Nordval

Uses Kappa, a measure of agreement that “corrects” for random agreement – That is, if methods A and B are totally

unrelated, there is a likelihood that they will agree on a lot of results, just by chance

e.g., if A reports 80% positive and B reports 70% positive, then we expect them to agree 56% of the time, just by chance

(0.8x0.7 = 0.56) So the extent to which the methods agree in excess

of 56%, is a measure of concordance

Classic Unpaired Design

When two methods are used on unpaired samples For example, drug studies on “treatment”

and “placebo” groups This is a classic design, not done in method validation studies Mentioned in ISO 16140, but not used

2x2 Layout – Unpaired (not used in ISO 16140)

Result Total

Method Present Absent

Method A e f NA

Method B g h NB

Total N+ N-

N

Estimators

Assume random samples from same population, randomly assigned to A or B Proportion Positive A = (e)/NA (= PODA)

Proportion Negative A = (f)/NA

Proportion Positive B = (g)/NB (= PODB) Proportion Negative B = (h)/NB

Accuracy, sensitivity, specificity not defined unless all results are confirmed

Chi-Square and POD

P+A - P+B = dPOD Chi-Square test is the same as a Binomial test of null hypothesis: H0: dPOD = 0

Chi-Square Test

Checks only for differences between observed and expected numbers of results in each cell “Expected” based on random assignment

of subjects to A or B, so expect same proportion of positives in A and B (and same proportion of negatives) “Expected” calculated from marginal

frequencies

Estimators POD Concept

P+A: PODA = (e)/NA

P+B: PODB = (g)/NB

P+A - P+B = dPOD Chi-Square test is the same as a Binomial test of null hypothesis: H0: dPOD = 0

Thank you

For the Validation of Qualitative Methods

Paul Wehling June 30, 2011

1

Qualitative (Binary) Methods Methods that are restricted to 2 possible

outcomes:

Positive or Negative

Pass or Fail

Heads or Tails

1 or 0

Yes or No

Presence or Absence

Identified or Not Identified 2

POD Parameter – Probability of Detection

General – Designed to be used by any Qualitative (Binary) Method

Microbiological

Chemical

Bio Threat Agent Methods

Botanical Identification

Allergens

3

POD Parameter Method Parameter that describes and

predicts method behavior

Probability of Detection or POD

The probability of getting a positive result at a given concentration of analyte.

POD is a function of concentration

4

POD Curve

5

POD Curve

6

POD A simple descriptive statistic that describes the

method performance at a given concentration.

It is a calculation of proportion of observed positive outcomes per total trials.

This simple statistic is inherent in all other systems, such as Chi-Square, LOD, RLOD.

The “POD Concept” is only new in that it recognizes the POD as a key parameter and plots a graph of POD vs concentration.

7

WHY PLOT POD? Plot of POD Curves are intended to assist

method users To assist users in selecting best method for intended

use.

Understanding POD Curve is crucial for interpretation of results.

The POD curve can be an indicator of the “usefulness” of the method.

If POD were constant across all concentrations, the method would not be useful.

8

VALIDATION The task of validating a qualitative (binary)

method is characterizing the POD curve at critical concentration points. 1. Make up a series of test materials at concentrations

of interest.

2. Analyze with replication

3. Calculate the proportion of positive responses at the concentrations.

4. Plot observed proportions as POD curve by concentration.

9

10

Example POD Response Curve

11

A BIT ABOUT POD POD is a combination of sensitivity,

specificity, false positives, false negatives.

Where did they all go?

12

“Where’s my False Negative?”

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

PO

D

Concentration (ppm)

POD Response vs Concentration

POD(1 ppm)

1-POD(1 ppm)

“Sensitivity at 1 ppm”

“False Negative at 1 ppm”

“Specificity”

1-POD(0)

POD(0)

“False Positive”

13

Some Statistics To do Classical Collab Statistics, Code Results

“Positive” = 1

“Negative” = 0

Use AOAC Calculations from Quantitative Stats to estimate

Mean = POD

Reproducibility Standard Deviation

Repeatability Standard Deviation

Laboratory Standard Deviation

14

POD = Mean LAB1 LAB2 LAB3

Trial1 1 1 0

Trial2 0 1 1

Trial3 1 1 1

Trial4 0 0 0

Trial5 1 0 0

Trial6 1 0 1

Trial7 0 0 1

Trial8 0 1 0

Trial9 1 1 0

Trial10 0 1 0

Mean 0.5 0.6 0.4 LPOD = 0.50

15

Analogous Parameters Method Attribute

Quantitative

Parameter

Quantitative

Estimate

Qualitative

Parameter Qualitative Estimate

General Mean or

Expectation Mean, μ Mean, POD

Repeatability

Variance

Reproducibility

Variance

Laboratory Variance

Expected difference

between Two Methods* Bias, B dPOD

2

r2

r

2

R 2

R

2

L2

L

1 2x x

x

2

rs2

rs

2

Rs 2

Rs

2

Ls 2

Ls

1 2POD POD

or POD LPOD

16

Difference Between Methods - dPOD Compare any two methods by comparing

POD values at a given concentration.

Difference by subtraction

dPOD = PODc – PODr

dPOD is always dependent on concentration

17

dPOD (c = 0.5) = -0.30

dPOD (2) = -0.27

dPOD(3.5) = -0.10

18

dPOD Curve vs Concentration

19

Big Ideas Combine sensitivity, specificity, false

positive, false negative into 1 parameter – Probability of Detection or ‘POD’

Graph POD vs. Concentration with Confidence Intervals

Compare methods by difference of POD at same concentration

Use the classic statistical model and descriptive stats for quantitative methods as the tool for calculating qualitative stats.

20

POD Concept Works for single lab and Multilab experiments.

Works for paired and unpaired designs.

Provides harmonization across qualitative/quantitative methodologies.

Does comparisons and hypothesis tests via confidence interval analysis – equivalent to chi-squared tests.

POD Curve plots mean response and uncertainty on the same graph.

21

Copyright 2011 by Robert A LaBudde 1

Qualitative Method Validation

Studies for Quantal Data: LOD, dPOD, PRE = RLOD and ω

Robert A LaBudde, BS, MS, PhD,

ChDipl ACAFS, PAS

AOAC Statistical Advisor

Least Cost Formulations, Ltd.

Old Dominion University


Summary

• Example POD vs. Concentration curves

• Ideal POD vs. Concentration curve

• Transition range models

• Method performance requirements I

• Method performance requirements II

• Limit of Detection (‘LOD’)

• The ‘Concentration Fallacy’ for micro


Summary (cont’d)

• Method Performance Parameters III

• Difference in POD: dPOD(C,R)

• Odds ratio ω

• Poisson Efficiency Ratio (‘PRE’)

• Relative LOD (‘RLOD’)

• Examples: Micro


Summary (cont’d.)

• Non-micro methods

• Choice of metamer

• Warning for micro studies

• Conclusions & Recommendations


Example POD vs.

Concentration curves


Ideal Response vs.

Concentration curve • POD = ‘Probability of Detection’

= # Positive / # Trials

= mean of 0 or 1 data

• The ideal test method gives POD = 0 at

Concentration = 0, and POD = 1 for all

concentrations > 0.

• For real methods, there is a transition from POD =

0 to POD = 1 over a range of Concentration.


Transition range models

• True shape of transition curve depends upon

underlying model of what happens in test method.

• Symmetric distribution threshold crossing: Probit

and Logit (historically these have been most

commonly used).

• Asymmetric distribution threshold crossing: can

be concave or convex shape.

• ‘Hormesis’: drop-off at high concentrations.



(cont’d)

• There are a dozen or more possible model

forms in common use.

• Choice of a model form is subjective and

subject to controversy.

• Some curves convex, some curves concave,

some symmetric.

• Logit and probit are traditionally used as

middle ground when true shape is unknown.



(cont’d)

• Advantage over individual POD values may

be improved precision by pooling across

concentrations.

• If model form is incorrect, may have worse

precision than individual POD values.

• Generally requires Concentration be known

accurately.


Method Performance

Requirements I: Confirmation • At the most basic level, a qualitative method is

meant to discriminate between the presence and absence of an analyte.

• At zero concentration, POD < PODmax with 95% confidence. (Control false positive fraction.)

• At moderate concentration, POD > PODmin with 95% confidence. (Control false negative fraction.)

• Attainment of these two requirements validates the method as a ‘confirmation’ or ‘identification’ method in testing.


Method Performance

Requirements I (cont’d)

• No real method, despite claims, has POD =

0 at zero concentration or POD = 1.0 even

at high concentration, due to various error

sources, including human-in-the loop .

• One method is better than another if it has

lower POD (FPF) at zero concentration and

higher POD (lower FNF) at moderate

concentration.


Method Performance Requirements

II: Transition region

• The ‘I’ set of requirements does not speak to the

transition range of the POD vs. Concentration

curve.

• A method which satisfies the PODmin performance

requirement at lower Concentration is ‘better’ than

another method does so at higher Concentration.

• A method which has POD < 1 may still be useful

in repeated testing if no better method is available

(e.g., outbreak investigations for micro).


Limit of Detection ‘LOD’ • One way commonly used to characterize a method in the

transitional range is to estimate the concentration at which

a particular POD is attained.

• ‘LOD50’: Concentration for which POD = 0.50

• ‘LOD90’: Concentration for which POD = 0.90

• Various techniques for estimation, including non-

parametric ones, such as linear interpolation and

Spearman-Kaerber (POD-based), or assumed models.

• Requires several points (at least 2, preferably more) in the

transitional or ‘fractional’ range.

• Requires accurately known concentrations!


LOD50


The ‘Concentration Fallacy’ for

Micro Methods • The transition region for qualitative methods for micro

testing typically occurs below 10 CFU/test portion and so cannot be quantified by plate count methods, particularly with other flora present. Instead a ‘MPN’ method is used, based on a reference qualitative method.

• So Concentration is determined from POD (not v.v.), and typically has large error limits (e.g., + 60% or much worse). POD is known more accurately than Concentration.

• Models fitting POD using Concentration as a predictor are invalid.

• LOD50 will be imprecise and unknown to a multiplicative factor (bias) due to clumping of cells.

Micro: 1-Hit-Poisson Model



Method Performance Requirements

III: Comparison of Methods

• Two methods which both satisfy the ‘I’

requirements equally can only be discriminated if

one or the other has data in its transition range.

• There are a number of measures of effect in

common use to compare a candidate method ‘C’

to a reference method ‘R’ (or any two methods) to

each other, based on measured POD values at

different concentrations in the transition range

(i.e., fractional POD range).


Difference in POD:

dPOD(C,R) • The most basic comparison between a reference

method ‘R’ and a candidate method ‘C’ is the

difference in their POD values at a fixed

concentration

dPOD(C,R) = POD(C) – POD(R)

• Non-constant for difference concentrations.

• Expected difference in # positives easily estimated

as n x dPOD(C,R).

• Requires no assumptions, applicable in all cases.


Odds ratio ω

• The most common measure of effect used to

compare two binary methods in scientific

research is the ‘odds ratio’ or ‘ω’

POD(R)/[1-POD(R)]

ω = ---------------------------

POD(C)/[1-POD(C)]

• If a Logit model is appropriate, the ‘odds

ratio’ is a constant across concentrations.


Poisson Relative Efficiency

‘PRE’ or ‘R’ • LaBudde, R.A. (2006). Statistical analysis of interlaboratory validation

studies. X. Poisson-plot and Poisson relative efficiency to compare the

dose-response curves of two presence-absence methods. TR239. Least

Cost Formulations, Ltd., Virginia Beach, VA.

ln [ 1 – POD(C) ] γR

• R = ---------------------- = ----

ln [ 1 – POD(R) ] γC

where the one-hit Poisson model ‘1HPM’ is assumed to hold.


PRE (cont’d)

• If both the reference and candidate methods obey the 1HPM model with cluster sizes γR and γC, resp., the R = γR / γC is the ratio of the two cluster sizes needed. If the reference method is ‘better’ (has a lower γ or LOD50), then R < 1.0.

• If the 1HPM is valid, R will be constant across different concentrations.

• Generally applicable to micro studies only.


Relative LOD ‘RLOD’

• Anon. (2008). ISO 16140.

• If a 1HPM model assumption is made for the

mathematical form of POD, and

log(Concentration) is used as the metamer in the

model, a ‘complementary-log-log’ model results.

• For the complementary-log-log model, RLOD =

R, and γR and γC are the factor coefficients for the

‘Method’ term in the regression model.

• ‘RLOD’ is the same value as ‘PRE’ = ‘R’.

R vs. ω vs. POD

• If POD(C) and POD(R) are both small,

R ~ ω ~ POD(C) / POD(R)

• If POD(C) and POD(R) are both large,

R ~ ω ~ POD(C) / POD(R)

• If POD(C) ~ POD(R),

R ~ ω ~ POD(C) / POD(R) ~ 1

• Differ more otherwise.



Examples: Micro Analyte Matrix Study Level MPN(R) dPOD(C,R) w(C,R) R(C,R)

Salmonella Raw ground turkey Unpaired H 3.66 -0.04 0.38 0.75

Salmonella Raw ground beef #1 Unpaired H 2.18 -0.13 0.40 0.65H 2.33 -0.10 0.45 0.70

Salmonella Raw ground beef #2 Unpaired H 2.11 -0.11 0.47 0.70L 0.58 -0.23 0.34 0.41

Salmonella Dried whole egg Paired H 2.58 -0.05 0.59 0.82L 1.05 -0.03 0.88 0.92

Salmonella Milk chocolate #2 Paired H 1.55 -0.03 0.84 0.91L 0.48 -0.03 0.88 0.90

Salmonella Dry dog food Paired H 1.10 -0.03 0.86 0.91L 0.27 -0.02 0.91 0.92

E. coli O157:H7 Raw ground beef Unpaired H 3.18 0.72 74.41 11.80L 0.84 0.46 11.61 7.93


Method Performance

Requirements III

Possible performance requirements:

|dPOD(C,R)| < dPODmax with 95% confidence

ω(R,C) > ω0 with 95% confidence

R(C,R) > Rmin with 95% confidence


Non-Micro Methods

• Toxins, residues, chemicals, allergens, botanicals.

• There is a large literature associated with POD vs.

Concentration modeling and fitting for toxicology.

• Complementary-log-log is not typically a good

match for non-micro methods, typically logit and

probit have been used successfully.

• None of the standard regression models work for

botanical identification methods where complex

thresholding occurs.


Choice of metamer

• Most models use either Concentration directly or

log(Concentration) as a predictor.

• The transform of Concentration to a new

independent variable is called the ‘metamer’ of

Concentration.

• It should be noted that linear models using

Concentration as metamer and linear models using

log(Concentration) as metamer cannot both be

correct.


Warning for micro studies

• In the case of micro methods, cells are indivisible, and CFUs finitely divisible, so sampling error dominates at low concentrations. Method differences are obscured, if they have low LOD50’s.

• The 1HPM or complementary-log-log model will appear to fit the data very well, and this is fallacious because Concentration is determined assuming 1HPM in the MPN method. Micro study modeling should not use numerical values of Concentration!


Conclusions • Models involving Concentration are flawed at

inception for micro methods. (This applies to Method Performance Requirements III in the transition region and to LOD50.)

• Serial dilution should be considered as a possible method to achieve a POD-independent Concentration estimate.

• Many alternative models exist, each with their own history and literature.

• PRE (aka RLOD or R) depends on the assumption that the 1HPM is correct, which it often is not.


Conclusions (cont’d)

• LOD50 requires Concentration be known reasonably accurately, problematic for micro.

• LOD50 can be nonparametrically estimated from POD, with no model assumptions.

• The choice of statistic used to characterize the transition region of POD vs. Concentration should be made based on the scientific validity of the model assumptions and the ease and usefulness of interpretation of the result.

• Chemical-based methods have accurate Concentrations; Micro studies do not.


Conclusions (cont’d)

• Use of the wrong model form will may give poorer results than using the POD vs. Concentration curve directly, and comparing methods by POD difference (‘dPOD’).

• Model forms that work for one analyte or matrix may not be appropriate for another, even in the same scientific method area.

• Nonparametric methods (POD included) that are distribution and model assumption-free are preferred to unvalidated model assumptions.

Working Group on Statistics

• Provide advisory guidance to Micro and Chem Working groups on aspects of statistical methodologies.

• Advise on strengths, weaknesses and applicability of various models.

• Advise on power of various validation experiment designs.

• Look for potential areas of agreement and encourage flow of ideas across Chem/Micro working groups.

Working Group on Statistics

• Develop scientific consensus on the best statistical techniques to use for validating qualitative methods.

Microbiological Harmonization June 29/30, 2011

Comparison of Method Validation Schemes

Worldwide Validation Schemes

• ISO 16140: internationally accepted standard for

microbiological method validation • AOAC Microbiology Guidelines • Health Canada Part 4 • NordVal (essentially 16140 w/o collaborative) • FDA Draft Guidelines • USDA/FSIS Draft Guidelines

Comparison of Elements

• Comprehensive table constructed • Six schemes compared • Qualitative: 30 topics • Quantitative: 19 topics • Initial effort on Qualitative • 5 of 6 schemes are either new or under

revision

Areas of Divergence

• Microbiology Working Group (WG) had 2 teleconferences

• Identified the 5 most significant areas of divergence among the 6 schemes.

• Nominated a Project Group (PG) with representative from each organization

• ISO/NordVal both use 16140 so only 1 representative

Significant Topics • From 30 topics 5 were chosen as most critical: • Reference method choice • Food/Sample Matrix Applicability Table.

Selection of Food/Category • # of levels/ # of samples/sample size/ # of

laboratories: Method Comparison & Collaborative

• Definition of fractional positive recovery • Data analysis (Chi square, RLOD, LOD, POD) &

performance parameters reported

Today

• 8 page comparative summary table prepared for the 5 significant topics

• PG will hold inaugural meeting to share ideas on harmonization

ISO 16140 AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS

Page 1 of 8

QUALITATIVE

methods

ISO 16140 Doc N 1199 (ISO CD 16140-2)PIV

C2011-04-06

Pending revision of Part 2

AOAC OMA Draft revision document dated 3/24/11

Health Canada Draft Part 4 dated March, 2011

NordVal Protocol for the validation of alternative

microbiological methods

March 2009

FDA Guidelines FDA’s Qualitative

Microbiology Methods

Validation (ORA-LAB.7

version 1.2), pending

revision (proposed revision

marked in red).

Draft USDA/FSIS

Guidelines Disclaimer: The use of the

term “validation” is not

intended to have any

application to the

implementation of 9 CFR

417.4(a)(1) on initial

validation of HACCP plans.

The Draft FSIS Guidelines

deals exclusively with the

evaluation of pathogen test

kit methods.

Pre-Collaborative

Phase(s)

-Reference

Method

-Defined in ISO

16140-1

-1st priority is ISO

method, 2nd

priority is

CEN method, if

neither exists, then 3rd

priority is other

recognized methods

Note: definition still

under discussion at

ISO level to open up

for non ISO/CEN

methods (PIV)

-Can be various pre-

existing recognized

analytical methods

e.g. AOAC OMA,

ISO, FDA BAM,

FSIS MLG and

Health Canada

-If no appropriate Ref

can indicate “NA” in

summary tables for

POD

-Acceptable Ref published

by HC (Part 1)

-May include any methods

from methods organizations,

such as AOAC, BAM,

APHA, ICMSF, IDF, ISO

etc.

-Where no Ref exists, MMC

assess on case by case basis

ISO, CEN, NMKL,

BAM, etc. It is up to

the applicant;

however, as the EU

regulation in EC

2073/2005

Microbiological

criteria states EN ISO

methods, these are

most frequently used.

-Must be BAM, unless there

is no BAM reference

method.

-If these is no BAM

reference method, but if

there is a

nationally/internationally

recognized reference

method, then FSIS MLG,

AOAC, ISO, and Health

Canada are all potential

reference methods. APHA,

ICMSF, and IDF methods

also may be used as

reference methods.

For FSIS regulated products,

the current FSIS method,

which is found in the

Microbiology Laboratory

Guidebook (MLG), is the

most appropriate reference

cultural method for validating

methods used by FSIS-

regulated establishments.

FDA BAM, or methods

referenced by ISO or Codex

Alimentarius may be

appropriate. Non-cultural

methods applicable in some

circumstances.


Page 2 of 8

-Selection of

food RA

-5 categories for all

foods applications, 3

food types per

category (see below)

-Feed, environmental

samples and primary

production samples

(PIV)are additional

categories

RLOD

Same, except 1 food

type per category (if

possible) a different

food type

SLV

-All claimed matrices

must be included in

the study, in other

words, no defined

categories, and “all

foods” claim not

applicable

-Environmental

surfaces claim

require 3-7 different

surfaces (#

required is under

review (RF)

IV

At least 1 matrix that

was tested in the

SLV. For every 5

foods claimed, 1 food

matrix must be

included

-5 categories for all foods

applications, 3 food types

per category (Table 1).

-Environmental samples is

additional category

RA, SE, SP, Kappa

-5 categories for all

foods applications, 3

food types per

category (see below)

-Feed, environmental

samples are

additional categories

LOD

Same, except 1 food

type per category (if

possible) a different

food type

The selection of foods is

determined by FDA’s

regulatory needs.

Matrices commonly sampled

in FSIS regulated

establishments: meat,

poultry, and egg products,

and environmental samples

(sponges, swabs, brines)

All claimed matrices must be

included in the study.

Contains proposal to create

matrix categories based on

intrinsic properties. “All

Foods” claim not applicable


Page 3 of 8

- Food

category/type/

item

Each Food type can be

made of various

relevant food items.

Annex B provides

guidance ( not

mandatory)

These are then

grouped together to

meet the sample

number requirement of

a food type, i.e. 20

samples.

This is allow for the

use of naturally

contaminated samples

(BL)

Only one single food

item is accepted to

meet the sample size

requirement of a food

type, i.e. 20

replicates.

Each food type can be made

of various relevant food

items. Table 1.

CLARIFICATION

NEEDED

Can these be grouped

together to meet the sample

number requirement of a

food type, in this case 20

samples?

Yes, they can be group

together to meet the sample

number requirement. This

notion has been introduce to

allow for heterogeneity with

in a food type. Products in a

type may vary greatly in

origin, composition,

preparation processes,

natural background; all

those small variabilities

could have an influence on

the detectability of the target

organism. (I.I.)

Each Food type can

be made of various

relevant food items.

At NordVal’s

homepage

(www.nmkl.org)

provides a list of food

categories

These are then

grouped together to

meet the sample

number requirement

of a food type, i.e. 20

samples.

Currently, foods are

validated individually and

there are no category

claims. There are no “All

Foods” claims.


Page 4 of 8

-No. of

levels/samples RA

20 samples per food

type or 60 samples per

category

RLOD

3 levels

-negative controls =5

samples

-1 level ( theoretical

LOD, with fractional

positive results (BL))

= 20 samples

-Another level = at

least 5 samples

SLV and IV

3 levels:


samples

-1 level with

fractional positive

results = 20 samples

-Another (high) level

= 5 or 20 samples

(under review (RF)

3 levels:


samples

-1 level with fractional

positive results = 20

samples

-Another level up to 1 log

higher= 20 samples

RA

20 samples per food

type or 60 samples

per category

LOD

3 levels


samples

-1 level ( theoretical

LOD) = 20 samples

-Another level = at

least 5 samples

Level 1, 6 replicates/level,

single level

Level 2, 6 replicates/level, 1

inoculated level + 1

uninoculated level (5

replicates)


1 inoculated level + 1


replicates)


1 inoculated level + 1


replicates)

It is proposed that each of

the 4 levels use 20 replicate

test portions and that all

levels have a negative

control.

For each matrix and analyte:

1) minimum 60 samples

inoculated at fractional

recovery level per

alternative and reference

method

2) 5-10 uninoculated

samples per alternative

and reference method

-Sample size Undefined

MicroVal: Is specified

in the reference

method, other (larger)

samples size is

allowed but specified

in the certificate.

(PIV)

Standard is 25 g or 25

mL, unless Ref

method specified

larger sample size

-25 g, but larger sample

sizes are permitted

-Sample size must be the

same for alternate and Ref

methods, consult MMC if

testing composite samples

Undefined 25 g unless otherwise

specified.

Application dependent.

Portions should not be made

larger without validation.

Validation study conclusions

from larger portions

applicable to smaller

portions.

-Fractional

positive

Can be achieved by

either alternate or Ref.

- All samples should

not be all positive or

all negative.

-Ideal is 10 positive

and 10 negative (50%)

but any fractional

results is acceptable

Can be achieved by

either alternate or

Ref.

-proportion of

positives 25% to

75%, ideal is approx

50%

(10% to 90% is under

review (RF)

Can be achieved by either

alternate or Ref.

-proportion of positives

25% to 75%,

Can be achieved by

either alternate or

Ref.

- All samples should

not be all positive or

all negative.

-Ideal is 10 positive

and 10 negative

(50%) but any

fractional results is

acceptable

Yes, one or both methods

must give 40 – 90% positive

results. It is proposed that

the percentage positive

results be changed to 25 –

75%.

defined as a range of 20-80%

confirmed positive results

using reference method


Page 5 of 8

-Results analysis

and criteria RA

-By type and by

category

-Relative accuracy

AC, relative

specificity SP, relative

sensitivity SE

-First by unconfirmed

results, again by

confirmed results

-McNemar test as

criteria, (for paired

and unpaired) with

caveats i.e. really not

suitable for unpaired

and “never be

interpreted by only

the McNemar test”

RLOD

-by category

-LOD of alternate

method divided by

LOD of Ref

For paired, no lower

limit, but LOD

alternate might not be

> 2 times the LOD Ref

For unpaired samples,

no lower limit, the

LOD alternate might

not be >3 times the

LOD Ref (In the

ISO/CD 16140-2

version, I don’t find

any acceptability

limit settled for

unpaired samples,

only specified for

paired samples BL)

The values of 2

(paired) and 3

(unpaired) are still

tentative values!!!!!

(PIV)

- by level and by

matrix

- by POD Probability

of Detection 95%

confidence interval

for the alternate the

Ref and presumptive

and confirmed results

-then by difference

between POD

alternate and POD

Ref, confidence level

must contain zero for

method to be

considered not

different at 95%

confidence

- Chi Square is not

required but

“interesting”

Method Equivalence:

-POD

-one-tailed POD 95%

confidence interval (I.I.)

Performance parameters:

- by level and by food, but

only calculated for those

that passed POD

successfully

For Unpaired :

-Performance parameters is

the comparison of

presumptive vs. confirmed

results of the alternate

method ( not the Ref

method results)

-Specificity is based on

presumptive results

-Sensitivity is based on final

( confirmed) results

- Equivalence of alternate

method and Ref can only be

determined by the number

of true positives in both sets,

done by POD method

For Paired:

Use “absolute” results

where Ref can have FN

Criteria:

Sensitivity 98%

Specificity 90.4%

False negative rate < 2%

False positive rate ≤ 9.6%

Efficacy 94%

LOD must be comparable or

exceed the lower LOD of

the Ref

RA

-By type and by

category

-Relative accuracy

AC,

-Relative specificity

SP,

-Relative sensitivity

SE

- Kappa

-First by unconfirmed

results, again by

confirmed results

Criteria:

SE ≥ 95%

Kappa ≤ 0.80

LOD: fit for purpose

By level/individual

experiment for each matrix.

Per AOAC Microbiology

guidelines, McNemar Chi

Square statistics are used.

Performed for each matrix.

Unpaired study: One sided

chi-square test with alpha =

0.05. Criterion:

indistinguishable or better

performance than reference

method. Paired study:

Evaluate sensitivity with

minimum 29 confirmed

positive results. Zero false

negative results from 29

confirmed positives would be

consistent with a test having

a sensitivity that met or

exceeded 90% and zero

negative results from 50

confirmed positives would be

consistent with a test with a

sensitivity that met or

exceeded 94%. Criterion:

none proposed


Page 6 of 8

Inter-

laboratory

Study

Applicable to alternative

methods with a major

modification, defined as any

significant change in the

design or the component

reagents for a screening test,

for example, the introduction

of a new antibody or

oligonucleotide primer.

Follow guidance provided by

the AOAC International

Official Methods of Analysis

Program

- minimum no.

of valid data

sets/collaborators

-10; defined as

individuals working

independently using

different sets of

samples; from a min.

of 5 different

organizations,

including organizing

lab and different

locations from same

company

-10 valid lab data sets

required

- Specifies that 12

labs should start

Minimum of 8 labs

reporting valid data, labs

should be accredited per

17025 or demonstrate is

functioning under

equivalent quality system

-10; defined as

individuals working

independently using

different sets of

samples; from a min.

of 5 different

organizations,

including organizing

lab and different

locations from same

company

2 for a Level 2 study, 3 for a

Level 3 study, and 10 for a

level 4 study.

-Sample size NA

Is defined by the

protocol of the

reference method

(PIV)

Standard is 25 g or 25

mL, unless Ref

method specified

larger sample size

CLARIFICATION

NEEDED

Consistent with Pre-

collaborative?

Sample size is 25g unless

otherwise specified by the

method or need for larger

size ( to achieve enhance

detectability, regulatory

purpose or compositing)

(I.I.)

NA 25 g unless otherwise

specified.


Page 7 of 8

- number of

foods

1; relevant food item,

inoculated with target,

using a challenging

enrichment protocol

1 At least 1 1; relevant food

item, inoculated with

target, using a

challenging

enrichment protocol

One or more.

- number of

levels

3; negative control,

one level which

produce fractional

positive and another

level


one level which

produce fractional


level

3; negative control, one

level which produce

fractional positive and

another level about 10 times

greater than the detection

level


one level which

produce fractional


level

2 for a Level 2 study (1

inoculated and 1

uninoculated.

3 for Levels 3 & 4 (high,

low, and uninoculated.

- number of

replicates

8; per level of

contamination

-minimum of 48

results per collaborator

= 8 replicates x 3

levels x 2 methods

-minimum of 480

results (48 from each

collaborator) = ( 240

per method) for

statistical analysis

12 per level of

contamination

- 72 results per

collaborator = 12

replicates x 3 levels x

2 methods = 72

- minimum of 720

results ( 360 per

method) for statistical

analysis

8 per level

- min of 24 results per

collaborator ( 8 x3 levels )

per method

8 laboratories;

- 3 levels in

duplicates

8 labs x 3 levels x 2

replicates x 2

methods

6

-Confirmation for Paired, only

confirm the + Alt/-

Ref, for Unpaired,

confirm all

enrichments

Matched or

unmatched, confirm

all samples

Confirm all samples for Paired, only

confirm the + Alt/-

Ref, for Unpaired,

confirm all

enrichments

Yes.


Page 8 of 8

- Comparisons Analyzed two ways:

1.Unconfirmed

Alternate method

results vs. confirmed

Ref

2. Confirmed

Alternate method

results vs. confirmed

Ref

By level and by

matrix analyzed and

reported separately

CLARIFICATION

NEEDED


collaborative?

By level and by matrix, all

result confirmed

Confirmed alternate method

results vs reference (I.I.)

Alternative to reference

method (if available).

-Parameters

Calculated

Specificity ( only for

Neg controls)

Sensitivity ( only for

inoculated levels )

Relative Accuracy

(%of agreements )

RLOD of the

different participants

(BL)

Cross Lab Probability

of Detection (LPOD)

Difference between

Alternate LPOD and

Ref LPOD

CLARIFICATION

NEEDED


collaborative?

Yes,

POD , dPOD determined

for each matrix-level . All

dPOD data is then used to

assess the comparative

performance of both

methods

All 5 method

parameter(specificity,

selectivity, FP, FN and

method efficacy) calculated

in one of two ways,,

depending if sample is

paired or un paired. (I.I.)

Rel Specificity

Rel Sensitivity

Rel Accuracy

Kappa

Per AOAC guidelines,

Sensitivity, Specificity,

False Negative, and False

Positive Rates.

- Interpretation

McNemar test (chi

square)

RLOD is for

information only

: analysis of deviance

test to assess the

laboratory effect on

RLOD then

acceptability of

RLOD global value

(BL)

If confidence interval

of dLPOD does not

contain zero, then the

diff is statistically

significant

CLARIFICATION

NEEDED


collaborative?

Yes , dPOD one-tailed and

method parameter

requirement must be met.

(I.I.)

Criteria:

SE ≥ 95%

Kappa ≤ 0.80

[LOD: fit for its

purpose]

Per AOAC guidelines,

McNemar Chi Square

statistics.

AOAC ISPAM Small Group Micro Working Groups Meeting Agenda nlm PRE-DECISIONAL Page 1

International Stakeholder Panel on Alternative Methods

Microbiology Working Group for Harmonized Matrix Comparison

Thursday, June 30, 2011 at 1:00pm – 3:00pm

Twinbrook HILTON WASHINGTON D.C./ROCKVILLE EXECUTIVE MEETING CENTER

DRAFT MATRIX TABLES FROM:

EN ISO 1614:2008 – NORDVAL - AOAC BPMM – AOAC BPMM WORKING GROUP MATRIX EXTENTION DRAFT

ISPAM

1) EN ISO 16140:2008 E

2) NORDVAL MATRIX TABLES

a. Salmonella

b. Listeria

c. Campylobacter

d. E.coli O157

3) Annex A AOAC OMA Microbiological Guidelines

4) Appendix B – BPMM AOAC Microbiological Working Group for Matrix Extension

EN ISO 16140:2008 (E) nlm

EN ISO 16140:2008 (E) nlm

Matrix for Salmonella (NORDVAL)

Matrix group

Matrix

Food examples

1. Meat 1.1 Raw red meat Minced meat, offal

1.2 Raw white meat Chicken, turkey, duck

1.3 Raw smoked salted products Bacon

1.4 Heat treated products Sliced meat and poultry products

1.5 Fermented products Salami

2. Fish 2.1 Raw fish and shelfish Raw two-shelled mollusc, raw

shrimps

2.3 Heat treated fish products

and shelfish

Heat treated shrimps

3. Milk 3.1 Milk Raw milk

3.5 Desserts, ice-cream Ice-cream

3.6 Dry milk products Milk powder

4. Eggs 4.1 Raw egg Whole egg

4.2 Egg products Manufactured egg

4.3 Dried products Dried whole eggs

5. Vegetable products 5.1 Raw vegetables Sprouts

5.2 Dried products Spices

5.4 Fatty products Chocolate, mayonnaisesalads

7. Environment tests 7.1 Environment tests Swab tests

8. Feed

8.1 Animal feed Meat bone meal, fish meal, fish food

9. Animal faeces

10. Miscellaneous

NORDVALnlm

Matrix for Listeria (NORDVAL)

Matrix group Matrix Food Examples 1. Meat 1.1 Raw red meat Minced meat, (tatar – type)

1.3 Raw smoked salted

meat-products

Bacon, smoked filet

1.4 Heat treated products Sliced meat and poultry products


2. Fish 2.1 Raw fish, shelfish and

Fish products

Cold smoked salmon

2.3 Heat treated fish products Heat treated shrimps


3.4.1 Firm cheese Yellow cheese

3.4.2 Soft cheese Mould cheese

3.5 Desserts, ice-cream Ice-cream


5. Vegetable products 5.1 Raw vegetables Cut salads, sprouts

7. Environment tests 7.1 Environment tests Swab tests, Cleaning water

10. Miscellaneous

Matrix for Campylobacter (NORDVAL)

Matrix group Matrix Food Examples

1. Meat 1.1 Raw red meat Minced meat, offal

1.2 Raw white meat Chicken, turkey, duck

1.3 Raw smoked salted products Sliced smoked turkey meat

1.4 Heat treated products Sliced poultry meat

2. Fish 2.1 Raw fish and shelfish Raw two-shelled mollusc, raw shrimps



4.2 Egg products Manufactured eggs

5. Vegetable products 5.1 Raw vegetables Sprouts


9. Animal faeces

10. Miscellaneous

Matrix for E. coli O 157 (NORDVAL)

Matrix group Matrix Food Examples 1. Meat 1.1 Raw red meat Minced meat, cut meat, offal

1.3 Raw smoked salted

meat-products

Bacon, smoked filet

1.4 Heat treated products, ready

to eat smoked products

Sliced meat and poultry products,

smoked turkey filet



3.2 Sour milk products Yoghurt with fruit

3.4 Cheese Mould cheese

3.5 Desserts/ice-cream Ice cream

5. Vegetable products 5.1 Raw vegetables Cut salads, sprouts

5.4 Fatty products Mayonaise-salads


9. Animal feces

10. Miscellaneous

Documents

AOAC Official Methods of AnalysisSM March 28, 2011, the AOAC INTERNATIONAL Board of Directors approved an alternative path to achieve an Official Method (Official First Action status)