Validity of ”antibiotic treatment for pneumonia” indicator

Professor Harry Campbell, Centre for Global Health Research,

Usher Institute of Population Health Sciences and Informatics, University of Edinburgh

JOGH Collection on Improving Coverage Measurement (ICM), 20 November 2018

Harry.Campbell@ed.ac.uk

Pneumonia still a major cause of child death840k child pneumonia deaths in 2017:• Wahl B et al. Lancet GH 2018

60% of global child pneumonia deaths occurred outside hospitals:• Nair H at al. Lancet 2015

The Integrated Global Action Plan for the Prevention and Control of Pneumonia and Diarrhoea

GAPPD Framework: Protect, Prevent, Treat

Reporting of population-based coverage data“Coverage of core interventions remains low”

Source: UNICEF's State of the World's Children 2013

Coverage indicator validation: Study design

Step 1: Observe intervention delivery

Step 2: Wait, based on recall period in DHS/MICS.

Step 3: Conduct household interviews

1) Standard DHS/MICS questions2) Additional or modified questions3) Inclusion of strategies to aid recall

Step 4: Compare, determining validity of respondents’ reports

Evaluation of “pneumonia treatment indicator”DESIRED INDICATOR: proportion of young children with pneumonia who receive a recommended antibiotic • Numerator: children with pneumonia who receive an antibiotic• Denominator: children with pneumonia

ACTUAL INDICATOR:• Numerator: children reported by caretaker to have symptoms of respiratory

infection consistent with pneumonia* who report receiving an antibiotic• Denominator: children reported by caretaker to have symptoms of

respiratory infection consistent with pneumonia* in the past 2 weeks

• * [cough and] short, rapid breathing or difficult breathing that is chest-related

Evaluation of “pneumonia treatment indicator”PROJECT AIMS: to provide information to assess the validity of the indicator as measured by DHS and MICS

Study in Ibadan, Nigeria led by Prof Falade / Dr Ayede, Department of Paediatrics, University of Ibadan• recruited ~300 children with “true pneumonia” and

~300 children with respiratory symptoms (cough / difficulty breathing) but in whom pneumonia was excluded (based on examination by trained doctor)

• conducted follow up survey with current DHS / MICS questions

Validity of the denominator of the indicator

What is the sensitivity and specificity of the DHS and MICS questionnaire measure of “reported short, rapid breathing or difficult breathing due to a problem in the chest ” for confirmed pneumonia ?

Validity of the numerator of the indicator

What is the sensitivity and specificity of the DHS and MICS questionnaire measure of reported caregiver recall of antibiotic treatment for antibiotic treatment recorded at the consultation? •does this improve with the use of a pill board use?

Validity of the denominator* of the indicator* children with fast/difficult breathing due to a problem in the chest past 2 weeks

Sensitivity, specificity and AUC for true pneumonia, by study site

Bangladesh urban

Nigeria urban

Pakistan urban

Bangladesh rural

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Sens

itivi

ty (%

)

False positive (%)

Validity of the denominator* for true pneumonia*children with fast/difficult breathing due to a problem in the chest past 2 weeks

The caregiver report is a poor predictor of true pneumonia [AUC ~ 0.6]

The sensitivity / specificity values vary widely so the % of true pneumonia will vary widely across surveys making comparisons difficult

TRUE PNEUMONIA

REPORTED ARI SYMPTOMS

Present Absent TotalPresentAbsentTotal 170 1700 1870

Validation results applied to a typical household survey10,000 children aged < 5 years

• ~170 cases true pneumonia expected in last 2 weeks (based on 300 cases /1000 child-yrs)• ~1700 cases of cough / difficult breathing

TRUE PNEUMONIA

REPORTED ARI SYMPTOMS

Present Absent TotalPresent 119 425 544Absent 51 1275 1326Total 170 1700 1870

Validation results applied to a typical household survey10,000 children aged < 5 years

• Apply 70% sensitivity and 75% specificity of DHS / MICS questions for true pneumonia

• 544 children with “ARI symptoms”* of whom 119 (22%) had true pneumonia

• * % of children with reported [cough and] short, rapid breathing or difficult breathing that is chest-related

TRUE PNEUMONIA

REPORTED ARI SYMPTOMS

Present Absent TotalPresent 119 425 544Absent 51 1275 1326Total 170 1700 1870

• All 119 children with pneumonia were treated and all children that did not have pneumonia were not treated – [all children with pneumonia treated correctly]

• indicator value 119/544 = 22%• No children with pneumonia were treated and 119 children that did not have

pneumonia were treated [no children with pneumonia treated correctly]• indicator value 119/544 = 22%

• This indicator does not give useful information on programmeperformance and 90% is not an appropriate target value – could result in very substantial over-treatment and promote antibiotic resistance

Scenarios with “pneumonia antibiotic treatment rate” indicator

Validity of the numerator of the indicator

• “pill board” was not labour intensive to develop• “pill board” significantly improved antibiotic treatment

recall• correct recall increased from 63% to 89% [overall kappa

statistic for agreement increased by ~ 0.15]• replicated now in 4 diverse Asian / African settings

Conclusions• poor discriminative power of DHS/MICS

questions for identifying true childhood pneumonia confirmed in African setting

• as most identified cases are NOT pneumonia, this does not serve as an adequate basis for asking about pneumonia treatment rate

• if programmes aim to increase this “treatment rate” they will drive over-prescription of antibiotics

“Coverage of core interventions remains low”

“careful assessment of how well countries are progressing on the delivery of proven interventions”

“only a core set of proven indicators are analysed”

Kruk ME et al. Lancet Global Health 2018

Global pneumonia report released on World Pneumonia Day 2018

Conclusions and Recommendations• low discriminative power + low pneumonia prevalence so most DHS/ MICS reported episodes don’t have pneumonia• invalid denominator for pneumonia antibiotic treatment indicator

Conclusions and Recommendations

• Do not attempt to use data on reported antibiotics given for symptoms of ARI as a proxy indicator of antibiotic treatment rate of pneumonia

• DHS and MICS now make their ARI questions very clear and do NOT attempt to measure “pneumonia treatment rate”;

• however, some other agencies still interpret these data as “pneumonia treatment rate”

• this is misleading and likely to lead to incorrect conclusions / decisions

Global pneumonia report released on World Pneumonia Day 2018

Conclusions and Recommendations

Conclusions and Recommendations

• Measures of effective coverage of pneumonia treatment should be explored by linking data on care-seeking coverage from household surveys with data on quality of care from health provider surveys

• Indicator validation studies are useful in determining which household survey questions to include and how data should be interpreted; and can give information on how indicators can be improved

For more details:see JOGH

Thanks & Acknowledgements

Department of Paediatrics, University of Ibadan and University College Hospital, Ibadan, Nigeria• Adejumoke I. Ayede; Kayode R. Fowobaje; Adebayo A

Bakare; Oladapo B. Oyewole; Oluwaseun B. Olorunfemi; Oluwaseun Kuna; Nkechi T. Iwuala; Abolanle Oguntoye; O. Simeon; Mofeyisade E. Okunlola; Adegoke G. Falade

University of Edinburgh• Amir Kirolos; Linda Williams; Harish Nair; Harry CampbellWHO• Shamim A. Qazi

• Funding for the Improving Coverage Measurement project was provided by the Bill and Melinda Gates Foundation through a grant to the Johns Hopkins University Bloomberg School of Public Health (OPP1084442).

Some additional slides

.

Caregiver recall of antibiotic treatment