Verbal Autopsy Modules in Surveys _______________________________ Henry Kalter, MD, MPH Johns Hopkins Bloomberg School of Public Health Baltimore, MD,

Verbal Autopsy Modules in Surveys

______________________________________________________________

Henry Kalter, MD, MPHHenry Kalter, MD, MPHJohns Hopkins Bloomberg School of Public HealthJohns Hopkins Bloomberg School of Public Health

Baltimore, MD, USABaltimore, MD, USA

Sneak Preview

Some available VA questionnairesSome available VA questionnaires

Use of VA in national surveys (DHS)Use of VA in national surveys (DHS)

Misclassification error and possible solutionsMisclassification error and possible solutions

Available VA tools and ongoing developments WHO standard VA for infants and children (1999)WHO standard VA for infants and children (1999)

Developed by WHO/JHSPH/LSHTM

Tanzania Adult M&M Project (AMMP)Tanzania Adult M&M Project (AMMP)

INDEPTH Network standardized VA (2003)INDEPTH Network standardized VA (2003) Built on WHO infant/child and AMMP adult formats

SAVVY VA for neonates, children and adultsSAVVY VA for neonates, children and adults Developed by Measure Evaluation in collaboration with

HMN for use in a nationally representative sample or selected sentinel area

Also can be used in surveys or censuses Used by India SRS, which claims better ascertainment of laims better ascertainment of

births and deaths than single surveysbirths and deaths than single surveys

Available VA tools and ongoing developments

WHO consensus group VA for neonates, children WHO consensus group VA for neonates, children and adults (2003–2007)and adults (2003–2007) Sponsored by HMN: to replace SAVVY VA tool and be part

of ‘HMN’s Stepping Stones’ resource kit for strengthening national vital statistics systems

Modules: birth-27 days, 28 days-14 years, 15+ years

Ongoing Harvard/JHBSPH/Queensland neonatal, Ongoing Harvard/JHBSPH/Queensland neonatal, child and adult VA validation study (Gates GC13)child and adult VA validation study (Gates GC13) Tanzania, Philippines, India (2 sites) Modules: birth-27 days, 28 days-11 years, 12+ years Will compare results of three analytic methods

Individual causes, by algorithms and physician readers All causes of death at once, by symptom profiles

National surveys that use VA

MACRO’s DHS (16/187 surveys, 1987–2007)MACRO’s DHS (16/187 surveys, 1987–2007) Stillbirths, child (NN, 1-11 mo, 12-59 mo), maternal deaths No non-maternal adult deaths

UNICEF’s MICS (limited COD information)UNICEF’s MICS (limited COD information) AIDS: ‘anyone aged 18-59 years who died in past 12 months

and was seriously ill for 3/12 months before death’ MM: sisterhood method + ‘death during pregnancy,

childbirth or within 6 weeks after the end of pregnancy’

Other national health surveys, e.g., Turkey 2003Other national health surveys, e.g., Turkey 2003

Country DHS with VA Subsequent DHS without VA

Morocco 1987 1992, 2003-04

Egypt 1988 1992, 1995, 2000, 2005

Cameroon 1991 1998, 2004

Namibia 1992 2000, 2006

Bolivia 1994 1998, 2003

CAR 1994-95 --

Haiti 1994-95 2000, 2005

Chad 1996-97 2004

Nigeria 1999 2003

Bangladesh 20042007 (but 1993-94 & 1996-97 w/VA: showed declines in most causes)

Cambodia 2005 --

Honduras 2005 --

Nepal 2006 --

Pakistan 2006 --

Angola 2006 --

Uganda 2007 --

Country ModulesReference period Analysis

Morocco<6 years: identified death but not cause -- --

Egypt<5 years: accident, 8 Sxs, 2 Dxs Child: 5 years

NN, 1-11 mo, 12-59 mo: Sxs (sum >100%)

Cameroon<5 years: accident or illness type, 9 Sxs Child: 5 years

NN, 1-59 mo: combine mother’s opinion &/or algorithm, e.g., diarrhea=algorithm, malaria= mother or algorithm (sum>100%)

Namibia

NN: COD, 9 Sxs1-59 mo: COD, 19 SxsMAT: preg/deliver/6wks

Child: 5 years MAT: sisterhood

NN, 1-59 mo: non-specific algorithms (e.g., measles: age >4 mo + rash (sum>100%)MAT: direct (age & year of death)

Bolivia MAT: preg/deliver/6wks MAT: sisterhood MAT: direct

Central African Republic

NN: COD, 9 Sxs1-35 mo: COD, 19 SxsMAT: preg/deliver/6wks

Child: 3 yearsMAT: sisterhood

NN, 1-35 mo: combine mother’s opinion &/or algorithm (sum>100%)MAT: direct

HaitiNN: COD, 7 Sxs1-59 mo: COD, 16 Sxs Child: 5 years

NN, 1-11 mo, 12-59 mo: combine mother’s opinion &/or algorithm (sum>100%)


Chad

NN: COD, 9 Sxs1-59 mo: COD, 19 SxsMAT: preg/deliver/2mo

Child: 5 yearsMAT: sisterhood

NN, 12-59 mo: combine mother’s opinion &/or algorithm (sum>100%)MAT: direct

Nigeria MAT: preg/deliver/2mo MAT: sisterhood MAT: direct (but data quality prblm)

BangladeshNN, 1-59 mo: detailed format for each group Child: 5 years

NN, 1-11 mo, 12-59 mo: detailed algorithms, w/hierarchical assignment of cause(s) (sum=100%)

Cambodia MAT: preg/deliver/2moChild: 3 yearsMAT: sisterhood

NN, 1-59 mo: mother’s opinion & algorithm (separately); MAT: direct

HondurasNN: COD, 31 Sxs1-59 mo: COD, 28 Sxs Child: 5 years NN, 1-11 mo, 12-59 mo: ?

Nepal

Stillbirth, NN, 1-59 mo: detailed format for each groupMAT: preg/deliver/2mo MAT: sisterhood

Stillbirth, NN, 1-11 mo, 12-59 mo: detailed algorithms, w/hierarchical assignment of cause(s) + MD review of undetermined cases (sum=100%)MAT: direct

Pakistan

Stillbirth, NN, 1-59 mo: detailed module for eachMAT: detailed VA

Child: ?MAT: ?

Child: ?MAT: MD review to determine if maternal, direct/indirect, cause & CS


AngolaNN, 1-59 mo: 3-page format for each group Child: ? Child: ?

UgandaNN, 1-59 mo: 6-8-page format for each group Child: ? Child: ?

Verbal autopsy in DHS surveys

Survey design issues (‘Deaths of children born in Survey design issues (‘Deaths of children born in the last 3-5 years’)the last 3-5 years’) 1-year maximum recall recommended for child deaths Variable recall depending on age (shorter for older children) Age and cause distributions distorted

Disproportionately captures deaths of younger children• This also distorts the all-ages cause distribution

Nepal design does not distort:• All child deaths in past 5 years


Most use a sparse questionnaireMost use a sparse questionnaire No adult (non-maternal) deaths Based on validation studies, but perhaps too few items with

insufficient detail 21-60% (high end for NN deaths) of cases with

undetermined cause of death Bangladesh & Nepal VAs longer, based on standard formats

Require re-visit to administer 1.1-3.4% (Bangladesh) and 4.5-11.4% (Nepal) of cases

with undetermined cause of death by algorithm Consider developing a standard DHS VA questionnaire based

on current best practices and what’s practical


Unusual methods for coding VA diagnosesUnusual methods for coding VA diagnoses Usual methods: physician readers or one algorithm/COD Most DHSs examine multiple algorithms for each cause Most DHSs combine maternal opinion with algorithms

Unclear decision tree (when to use which method?) Diagnoses add to >100%, but usually unclear as to which

children have >1 diagnosis Consider developing a standard DHS coding method based

on current best practices (change may be on the horizon)

VA diagnosis misclassification

ALRI (sensitivity/specificity)ALRI (sensitivity/specificity) Cough >3 days and difficult breathing >3 days

Bangladesh: 64%/84% Uganda: 51%/68%

Malaria (sensitivity/specificity)Malaria (sensitivity/specificity) Fever and convulsions or loss of consciousness

Namibia: 45%/87% Uganda: 44%/77%

Measles (sensitivity/specificity)Measles (sensitivity/specificity) Age >120 days, rash and fever >3 days, rash on face (Nam.)

or rash anywhere except extremities (Phil.) Namibia: 83%/86% Uganda: 98%/93%

VA diagnosis misclassification

Methods of dealing with misclassificationMethods of dealing with misclassification Do nothing (usual method)

False positives and false negatives may counter-balance each other to produce an accurate cause-specific mortality estimate, but this is uncertain

Despite uncertainties, there is some evidence that VA can usefully measure changes in cause-specific mortality

Improve VA performance Accommodate for misclassification Adjust for misclassification Go around misclassification

VA diagnosis misclassification – Improve VA performance

Attempts to improve WHO standard neonatal VAAttempts to improve WHO standard neonatal VA Addition of stillbirth module Additional details on pregnancy and L&D complications New signs of NN illnesses strive for increased specificity

Breathed ‘immediately’ after birth Breathed ‘immediately’ after birth (was ‘able to breathe’)(was ‘able to breathe’) Sucked normally ‘during the first day’ Sucked normally ‘during the first day’ (was ‘after birth’)(was ‘after birth’)

Attempt to improve coding of VA diagnosesAttempt to improve coding of VA diagnoses Compare physician readers to algorithms

Improvements may be elusiveImprovements may be elusive Best sensitivity and specificity depend on disease prevalence Disease mix can affect specificity (and perhaps sensitivity)

VA diagnosis misclassification – Accommodate for misclassification

1 0 % M o r t a l i t y F r a c t i o n

S n/ Sp 60% / 7 0% 60 % / 90% 80 % / 90%D i s e a s e D i s e a s e D i s e a s e

VA dx yes no yes no yes no + 18 81 33% 18 27 15% 24 27 17%

- 12 189 12 243 6 243

30 270 300 30 270 300 30 270 300

D i s e a s e D i s e a s e D i s e a s eVA dx yes no yes no yes no + 18 81 33% 18 27 15% 24 27 17%

- 12 189 12 243 6 243

30 270 300 30 270 300 30 270 300


40% Mortality Fraction

Sn/Sp 60%/70% 60%/90% 80%/90%

Disease Disease DiseaseVA dx yes no yes no yes no + 72 54 42% 72 18 30% 96 18 38%

- 48 126 48 162 24 162 120 180 300 120 180 300 120 180 300

Disease Disease DiseaseVA dx yes no yes no yes no + 72 54 42% 72 18 30% 96 18 38%

- 48 126 48 162 24 162 120 180 300 120 180 300 120 180 300


Sensitivity/Specificity that can Estimate Cause-Specific

Mortality Fraction Within + 20% of the True Level

5% MF

70-100%/100%

10% MF

80-100%/100%

50-70%/95%

20% MF

80-100%/100% 60-100%/95% 50-80%/90% 50-60%/85%

30% MF

80-100%/100% 70-100%/95% 60-95%/90% 50-85%/85% 50-70%/80% 50-60%/75% 50%/70%

40% MF

80-100%/100% 80-100%/95%

70-100%/90% 60-95%/85% 50-90%/80% 50-80%/75% 50-70%/70% 50-60%/60%


ObjectivesObjectives Determine how different (cultural and disease) settings affect VA

performance Identify algorithms with consistent and appropriate performance in similar

settings

Method: Conduct validation studies with standardized Method: Conduct validation studies with standardized methods in multiple settingsmethods in multiple settings Much of the apparent variability in VA performance may be due to

inconsistent study methods Determine the effects of site characteristics on performance

Different cultural settings Different disease mixes (e.g., with and without malaria) Malaria sites with different transmission intensities

VA diagnosis misclassification – Adjust for misclassification

‘‘Back-calculate’ to adjust for misclassificationBack-calculate’ to adjust for misclassification

Uses sensitivity/specificity estimates of VA algorithms from hospital-based validation studies

Very sensitive to inaccurate estimates caused by: Hospital-based study biases:

• Differences in hospital/community disease mix• Medical exposure• Cultural, SES, etc. differences

Differences between validation study and survey sites Basis of the problem: composite nature of specificity vs.

“yes/no” classification

CSMF = (VA + Sp – 1) / (Sn + Sp – 1)

VA diagnosis misclassification – Back-calculate to adjust for misclassification

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Uganda Bangladesh Nicaragua

Back-calcVA estimateTrue MF

Diarrhea-Specific True Fraction and Estimated Levels Determined by Verbal Autopsy and Back-Calculation

(using the average sn/sp from the other two sites)

VA diagnosis misclassification – Go around misclassification

Calculate disease probability from symptom profilesCalculate disease probability from symptom profilesP(S) = P(S|D) P(D) 2K x 1 2K x J J x 1

P(S|D)

Symptom profile COD_1 COD_2 COD_3 P(S)

000 0.09 0.08 0.04 0.04

001 0.37 0.27 0.11 0.32

010 0.14 0.12 0.04 0.11

011 0.00 0.00 0.00 0.00

100 0.12 0.29 0.38 0.27

101 0.00 0.18 0.15 0.09

110 0.10 0.06 0.00 0.07

111 0.18 0.00 0.28 0.10

All profiles 1.00 1.00 1.00 1.00

P(D) ? ? ?


Calculate disease probability from symptom profilesCalculate disease probability from symptom profiles

Does not require VA algorithms, sensitivity/specificity estimates or physician readers

Estimates mortality fractions of all CODs at once [P(D)] Eliminates biases caused by dichotomizing COD

Uses P(S|D) estimates from hospital-based study Less sensitive to inaccurate estimates than VA algorithms:

‘Symptom’ profiles can be manipulated at will to find differences in P(S|D) between diseases

Does not require big differences in P(S|D)s Allows multiple P(S|D)s for each disease (vs. one “yes/no”

algorithm) Still liable to bias due to inaccurate P(S|D) estimates

P(S) = P(S|D) P(D) 2K x 1 2K x J J x 1


Validation in Tanzania for adults (left graph), children (middle), and infants (right). In each graph, a direct estimate of cause-specific mortality is plotted horizontally by our verbal autopsy estimate plotted vertically (G. King, Y. Lu, 2006; data: Setel et al. 2006)

Summary and Conclusions –

DHS often uses sub-optimal VA methods DHS often uses sub-optimal VA methods Sparse modules, no adult (non-maternal) module, Sparse modules, no adult (non-maternal) module,

problematic analytic methodproblematic analytic method Convene a study group to improve modules and analytic Convene a study group to improve modules and analytic

methods based on current knowledge and practicesmethods based on current knowledge and practices

Ongoing research holds promise for improvementsOngoing research holds promise for improvements Identify algorithms with increased sensitivity/specificity Identify algorithms with increased sensitivity/specificity Gain better understanding of how cultural and diseases Gain better understanding of how cultural and diseases

settings affect VA performancesettings affect VA performance New (experimental) analytic method decreases bias in VA New (experimental) analytic method decreases bias in VA

estimates due to disease misclassificationestimates due to disease misclassification

THANK YOU

Documents

Verbal Autopsy Modules in Surveys _______________________________ Henry Kalter, MD, MPH Johns Hopkins Bloomberg School of Public Health Baltimore, MD,