2005 Syndromic Surveillance 1

Estimating the Expected Warning Time of Outbreak-Detection AlgorithmsYanna Shen, Weng-Keen Wong,

Gregory F. Cooper

RODS Laboratory, Center of Biomedical Informatics, University of Pittsburgh

2005 Syndromic Surveillance 2

Overview

Objective Background Methods Experimental Results Conclusions Future Work

2005 Syndromic Surveillance 3

Objective

A new measure for evaluating alerting algorithms, which is called Expected Warning Time (EWT).

It is a generalization of the standard AMOC curve.

2005 Syndromic Surveillance 4

Why Useful?

Can compare expected clinician detection time to expected computer-based algorithm detection time

Can provide a promising new approach for optimizing and comparing outbreak detection algorithms

2005 Syndromic Surveillance 5

Background

Maximum meaningful detection timeMaximum meaningful WT

Time

Warning Time Computer Detection Time

False Alerts (in red)

Incubation

Time

Release occurs

Computers raise alert

Clinicians detect outbreak

Last outbreak case appears

… … …hit

hithit

hit

2005 Syndromic Surveillance 6

Model

A simple model of clinician outbreak detection Assumes that

People with disease D are diagnosed independently of each other.

The probability of a person with disease D being diagnosed is constant (p).

2005 Syndromic Surveillance 7

Equation

Definitions: p – probability that a person with D is diagnosed as

having D upon presentation with that disease time(i) – maps patient case i to the time at which that

patient presented with D to clinicians M – total # of patient cases with D t – time at which the alerting score first exceeds a

given threshold

M

i

iM pptitimeptMtimetEWT1

11,0max1,0max

2005 Syndromic Surveillance 8

Equation

0

tMtime Mp1 11 ippx x+

WT if clinicians

never detect the outbreak

Probability that clinicians

will never detect the outbreak

WT if clinicians first

detect the outbreak on the ith case

Probability that clinicians will

detect the outbreak on the

ith case

0

titime

M

i 1

M

i

iM pptitimeptMtimetEWT1

11,0max1,0max

2005 Syndromic Surveillance 9

Experiment Setup

Apply PANDA to simulated cases of inhalational anthrax

For various value of p, derive EWT for PANDA

2005 Syndromic Surveillance 10

PANDA

An outbreak detection system Uses causal Bayesian networks to model spatio-

temporal patterns of a non-contagious disease in a population (Cooper, 2004)

Contains a model to detect inhalational anthrax

2005 Syndromic Surveillance 11

BARD

BARD simulator produces the simulated cases of anthrax.

It models the effects of an outdoor airborne anthrax release using the Gaussian plume model of atmospheric dispersion and a model of inhalational anthrax (Hogan, 2004).

2005 Syndromic Surveillance 12

Performance of Clinicians

p P(CD)ECDT (hours)

0.001 0.992 167.9

0.005 0.999 104.1

0.01 0.9999 88.6

0.05 0.99999 64.5

1 1 0

p – Clinician detection proficiency

P(CD) – Probability that clinicians will detect the outbreak at all

ECDT – Expected clinician detection time given that clinicians detect the outbreak

2005 Syndromic Surveillance 13

0

20

40

60

80

100

120

140

160

180

0 100 200 300 400 500 600 700

False Alerts Per Month

Exp

ecte

d W

arn

ing

Tim

e (H

ou

rs)

p = 0.001

p = 0.005

p = 0.01

p = 0.05

p = 1

Experimental Results

p increases, EWT decreases p = 1, EWT = 0

?

2005 Syndromic Surveillance 14

Experimental Results

Clinician-detection-

proficiency (p)

Expected Warning Time

(EWT)

0.001 76 hours

0.005 13 hours

0.01 5 hours

0.05 34 minutes

1 0

False alert rate = 1 per month

If and false alert rate = 1 per month, then EWT minutes.

05.0p 34

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11

False Alerts Per Month

Exp

ecte

d W

arn

ing

Tim

e (H

ou

rs)

p = 0.001

p = 0.005

p = 0.01

p = 0.05

p = 1

2005 Syndromic Surveillance 15

Conclusions

The Expected Warning Time (EWT) is a useful concept for evaluating outbreak-detection algorithms.

We illustrated the general idea of EWT using a simple model of clinician detection and simulated cases of inhalational anthrax.

Our example analysis suggests that PANDA is most helpful when clinicians’ detection proficiency < 5%.

2005 Syndromic Surveillance 16

Future Work

Extend the model: Instead of a constant (p), use the function p(t),

where t is time Develop and apply more disease-specific models

of clinician detection (please see the poster by Christina Adamou)

2005 Syndromic Surveillance 17

Acknowledgements

This research was supported by grants from the National Science Foundation (IIS-0325581), the Department of Homeland Security (F30602-01-2-0550), and the Pennsylvania Department of Health (ME-01-737).

We thank members of the Bayesian Biosurveillance Project for helpful comments.