Upload
others
View
28
Download
1
Embed Size (px)
Citation preview
Charles Charles CaraguelCaraguelCentre for Veterinary Epidemiological Research
Centre for Aquatic Health SciencesCentre for Aquatic Health SciencesAtlantic Veterinary CollegeAtlantic Veterinary College
University of Prince Edward IslandUniversity of Prince Edward Island
Diagnostic testsDiagnostic tests
evaluation & usesevaluation & uses
Why is it so exciting?“Test performance is
critical to start a new life.” Me (2008)
1
Basics (1): Variation & errors in estimation
SourcesSources• Sampling error (selection bias)
• Biological variation (time & space)
• Measurement error (diagnostics)
SemanticsSemantics
Health status (Binary outcome)
Test result (Binary outcome)
POSITIVE (T+)
NEGATIVE (T-)
DISEASED (D+)
NON-DISEASED (D-)
2
Phase 1 - ”Laboratory“ validation
• Optimisation & standardisation of reagents
• Analytical sensitivity (ASe) & specificity (ASp)
Validation ObjectivesValidation Objectives““evaluation of a process to determine its evaluation of a process to determine its fitness for a particular usefitness for a particular use””
Dilution
3
Phase2 - “Field” validation
• “Precision” (agreement sensus stricto)
Repeatability (overall intra-lab)
Reproducibility (overall inter-lab)
• “Accuracy” (performances)
Diagnostic Sensitivity (DSe)
Diagnostic Specificity (DSp)
Degree of mistakes made by the test
Phase 1 - ”Laboratory“ validation
• Optimisation & standardisation of reagents
• Analytical sensitivity (ASe) & specificity (ASp)
Validation ObjectivesValidation Objectives““evaluation of a process to determine its evaluation of a process to determine its fitness for a particular usefitness for a particular use””
4
Basics (2): Basics (2):
Conditional ProbabilityConditional ProbabilityPr(Pr(AA//BB) = given ) = given BB, the probability of , the probability of AA
i.i. First set the primary information or condition (First set the primary information or condition (BB))
Health status (Health status (Se&SpSe&Sp); Test result (Predictive values)); Test result (Predictive values)
ii.ii. Then defined the component of interest (Then defined the component of interest (AA))
Health status (Predictive values); Test result (Health status (Predictive values); Test result (Se&SpSe&Sp))
5
Tested positiveTested positive
Tested negativeTested negative
““ Non diseasedNon diseased ”” ““ DiseasedDiseased ””
Sp = Sp = Negative & Non dis. Negative & Non dis.
Non diseasedNon diseased
Se = Se = Positive & Diseased Positive & Diseased
DiseasedDiseased
““ True True PositivePositive ””
““ False False NegativeNegative ””
““ True True NegativeNegative ””
““ FalseFalsePositivePositive ””
6
Accuracy DefinitionAccuracy DefinitionSe & SpSe & Sp
Diagnostic Sensitivity Diagnostic Sensitivity (true positive)(true positive)
Pr(Pr(TT++//DD++) =) =
Diagnostic SpecificityDiagnostic Specificity (true negative)(true negative)
Pr(Pr(TT--//DD--) =) =
the probability of the probability of TT++given given DD++, ,
the probability of the probability of TT--given given DD--, ,
7
Legend of theLegend of the
2X2 TABLE2X2 TABLE
Sensitivity Sensitivity (true positive fraction)(true positive fraction)
Pr(Pr(TT++//DD++) = ) = a / (a + b)a / (a + b)
Specificity Specificity (true negative fraction)(true negative fraction)
Pr(Pr(TT-- //DD--) =) = d / (c + d)d / (c + d)
HealthStatus
Test Results
DD++
DD--
TT++ TT--
aa bb
cc dd
a + ca + c b + db + d
a + ba + b
c + dc + d
n = n = a+b+c+da+b+c+d
TP
FP
FN
TN
8
Real applications in the Real applications in the ““fieldfield””
1. 1. Prevalence estimatePrevalence estimate
““farmers & veterinariansfarmers & veterinarians””
““ApparentApparent””
prevalence (prevalence (APrAPr))““TrueTrue””
prevalence (prevalence (TPrTPr))
Se / SpSe / Sp
Prevalence definitionPrevalence definition
APrAPr == TPrTPr = =
True prevalence formulaeTrue prevalence formulae
TPrTPr ==APrAPr + Sp + Sp -- 11
Se + Sp Se + Sp -- 11 True positive among D+
False positive among D-
Apparent prevalence formulaeApparent prevalence formulae
APrAPr = Pr Se + (1 = Pr Se + (1 -- TPrTPr) (1 ) (1 –– Sp)Sp)
# Positive# Positive
# Total (n)# Total (n)# Diseased# Diseased
# Total (n)# Total (n)
9
Exercise (1):True prevalence estimation
During a surveillance screening, 18 cows tested positive
for Staphylococcus aureus out of 100 tested
(Test performances: Se = 72 % & Sp = 98 %)
Compute the following value:
Apparent Prevalence =
True Prevalence =
10
Real applications in the Real applications in the ““fieldfield””
1. 1. Prevalence estimatePrevalence estimate
2. 2. Decision makingDecision making
““farmers & veterinariansfarmers & veterinarians””
““ApparentApparent””
prevalence (prevalence (APrAPr))
““TrueTrue””
prevalence (prevalence (TPrTPr))
Se / SpSe / Sp
Positive Predictive Value (PPV)Positive Predictive Value (PPV)
Given test is positive, what probability to be diseased?Given test is positive, what probability to be diseased?
Negative Predictive Value (NPV)Negative Predictive Value (NPV)
Given test is negative, what probability to be nonGiven test is negative, what probability to be non--diseased?diseased?
Primary Information: TEST RESULTS!Primary Information: TEST RESULTS!
(condition)(condition)
11
NegNeg NegNeg NegNeg NegNeg NegNeg
NegNeg NegNeg NegNeg NegNeg NegNeg
NegNeg NegNeg NegNeg NegNeg NegNeg
NegNeg NegNeg NegNeg NegNeg NegNeg
NegNeg NegNeg NegNeg NegNeg NegNeg
PosPos PosPos PosPos PosPos PosPos
PosPos PosPos PosPos PosPos PosPos
PosPos PosPos PosPos PosPos PosPos
PosPos PosPos PosPos PosPos PosPos
PosPos PosPos PosPos PosPos PosPos
DiseasedDiseased
NonNon--diseaseddiseased
““ True True PositivePositive ””
““ False False PositivePositive ””
““ True True NegativeNegative ””
““ FalseFalseNegativeNegative ””
PPV = PPV = Positive & Diseased Positive & Diseased
PositivePositive
NPV = NPV = Negative & Non dis. Negative & Non dis.
NegativeNegative
NegativeNegative PositivePositive
12
Positive Predictive value Positive Predictive value (PPV)(PPV) Strongly influenced by SpStrongly influenced by Sp
Pr(Pr(DD++//TT++) =) =
Negative Predictive value Negative Predictive value (NPV)(NPV) Strongly influenced by SeStrongly influenced by Se
Pr(Pr(DD--//TT--) =) =
the probability of the probability of DD++Given Given TT++, ,
the probability of the probability of DD--Given Given TT--, ,
Predictive Values DefinitionsPredictive Values Definitions
13
Predictive Values FormulaePredictive Values Formulae
Positive Predictive ValuePositive Predictive ValuePr(Pr(DD++//TT++) = ) = a / (a + c)a / (a + c)
Negative Predictive ValueNegative Predictive ValuePr(Pr(DD-- //TT--) =) = d / (b + d)d / (b + d)
Test Results
DD++
DD--
TT++ TT--
aa bb
cc dd
a + ca + c b + db + d
a + ba + b
c + dc + d
n = n = a+b+c+da+b+c+d
TP
FP
FN
TN
14
You test 100 cows for Staph. aureus infection. Sixty-five cows tested
negative by PCR and 35 tested positive. However, using culture as
a gold standard, 5/35 of the PCR-positives were disease-free and
14/97 of the seronegatives were diseased.
Fill the 2X2 table and compute AP, TP, Se, Sp, PPV, NPV:
Se =
Sp =
PPV =
NPV =
Exercise (2):2X2 table and more
TT++ TT--
DD++
DD--
15
n*n*PrPr*Se*Sen*n*Pr(Pr(DD++&&TT++) )
n*n*Pr(Pr(DD--&&TT++) ) n*n*Pr(Pr(DD--&&TT++) )
n*n*Pr(Pr(DD++&&TT--) ) aa bb
cc dd
a + ca + c b + db + d n = n = a+b+c+da+b+c+d
Positive Predictive ValuePositive Predictive Value
Pr(Pr(DD++//TT++) =) =
Negative Predictive ValueNegative Predictive Value
Pr(Pr(DD-- //TT--) =) =
DD++
DD--
TT++ TT--
PrPr*(1*(1--Se)Se)
(1(1--Pr)Pr)*(1*(1--Sp)Sp) (1(1--Pr)Pr)*Sp*Sp
Pr(Pr(DD++) = ) = Prevalence(Prevalence(PrPr))
Pr(Pr(DD--) = ) = 1 1 -- PrPr
Pr(Pr(TT++//DD++) = Se) = SePr(Pr(TT-- //DD--) =) = SpSp
Bayesian Theorem: Bayesian Theorem: Pr(Pr(AA//BB) = ) = Pr(Pr(AA&&BB) / ) / Pr(Pr(BB)) Pr(Pr(AA&&BB)) = = Pr(Pr(BB) *) *Pr(Pr(AA//BB))
n take any value even 1n take any value even 1
PrPr*Se*Se
11
PrPr*Se*Se
PrPr*Se + *Se + (1(1--Pr)Pr)*(1*(1--Sp)Sp)
(1(1--Pr)Pr)*Sp*Sp
(1(1--Pr)Pr)*Sp*Sp + + PrPr*(1*(1--Se)Se)
When no a, b, c or d?When no a, b, c or d?but Se, Sp and Prevalencebut Se, Sp and Prevalence
16
Real applications in the Real applications in the ““fieldfield””
1. 1. Prevalence estimate (risk analysis) Prevalence estimate (risk analysis)
2. 2. Decision makingDecision making
““farmers & veterinariansfarmers & veterinarians””
““ApparentApparent””
prevalence (prevalence (APrAPr))
““TrueTrue””
prevalence (prevalence (TPrTPr))
Se / SpSe / Sp
DSeDSe
DSpDSp
PrevalencePrevalence
Positive Predictive Value (PPV)Positive Predictive Value (PPV)
Given test is positive, what probability to be diseased?Given test is positive, what probability to be diseased?
Negative Predictive Value (NPV)Negative Predictive Value (NPV)
Given test is negative, what probability to be nonGiven test is negative, what probability to be non--diseased?diseased?
17
in determining if she is infected with Staph. aureus. The PCR test
used assures 99 % accuracy (assumed 99% of DSe and 99% DSp).
The prevalence of cows infected with Staph. aureus and
presenting with clinical signs of mastitis of is 25%.
Define and compute the following values:
PPV =
NPV =
Exercise (3):Predictive value
Clarice is a 3 year-old cow showing
clinical signs of mastitis. The farmer
and the vet are interested
18
Real applications in the Real applications in the ““fieldfield””
1. 1. Prevalence estimate (risk analysis) Prevalence estimate (risk analysis)
2. 2. Decision makingDecision making
3. 3. Surveillance ProgramSurveillance Program
““farmers & veterinariansfarmers & veterinarians””
““ApparentApparent””
prevalence (prevalence (APrAPr))
““TrueTrue””
prevalence (prevalence (TPrTPr))
Se / SpSe / Sp
Computation of sample size to declare a population Computation of sample size to declare a population ‘‘free of diseasefree of disease’’
DSeDSe
DSpDSp
PrevalencePrevalence
n / Sen / Se
Positive Predictive Value (PPV)Positive Predictive Value (PPV)
Given test is positive, what probability to be diseased?Given test is positive, what probability to be diseased?
Negative Predictive Value (NPV)Negative Predictive Value (NPV)
Given test is negative, what probability to be nonGiven test is negative, what probability to be non--diseased?diseased?
19
QUESTIONS ?QUESTIONS ?
20
21