Validity of Bronchiolitis OutcomeMeasuresRicardo M. Fernandes, MDa,b, Amy C. Plint, MD, MScc, Caroline B. Terwee, PhDd,e, Cristina Sampaio, MD, PhDb,Terry P. Klassen, MD, PhDf, Martin Offringa, MD, PhDg, Johanna H. van der Lee, MD, PhDh

abstractBACKGROUND: The Respiratory Distress Assessment Instrument (RDAI) and Respiratory AssessmentChange Score (RACS) are frequently used in bronchiolitis clinical trials, but evidence is limited ontheir measurement properties. We investigated their validity, reliability, and responsiveness.

METHODS: We included data from up to 1765 infants with bronchiolitis enrolled in 2 studies conductedin pediatric emergency departments. We assessed RDAI construct validity by testing hypotheses ofassociations with physiologic measures (respiratory rate, oxygen saturation) and with constructsrelated to hospitalization, using correlation coefficients, and multivariable analysis. RDAI/RACSresponsiveness was evaluated by using anchors of change based on these constructs; measures ofresponsiveness included the area under the curve. RDAI test-retest agreement and interrater reliabilitywere evaluated by using limits of agreement and intraclass correlation coefficients.

RESULTS: Baseline RDAI scores were weakly correlated with respiratory rate (r = 0.38, P , .001), andscores increased in lower oxygen saturation categories (P , .001). Higher RDAI scores wereassociated with hospitalization (odds ratio: 1.36; 95% confidence interval: 1.26–1.47); scoresdiffered between participants who were discharged, admitted, or stayed in the emergencydepartment (P, .001). Our hypotheses were met, but the magnitude of associations was below ourpredefined thresholds. RDAI test-retest limits of agreement were 23.80 to 3.64 (20% of the range),whereas interrater reliability was good (intraclass correlation coefficient = 0.93). Formulatedhypotheses for responsiveness were confirmed, with moderate responsiveness (area under thecurve: RDAI, 0.64–0.70; RACS, 0.72).

CONCLUSIONS: RDAI has poor to moderate construct validity, with good discriminative properties butconsiderable test-retest measurement error. The RDAI and RACS are responsive measures ofrespiratory distress in bronchiolitis but do not encompass all determinants of disease severity.

WHAT’S KNOWN ON THIS SUBJECT: TheRespiratory Distress Assessment Instrument(RDAI) and the Respiratory Assessment ChangeScore (RACS) are the most frequently usedmeasurement instruments in bronchiolitisclinical trials. Evidence is scarce regarding theirmeasurement properties and their suitability foruse as evaluative instruments in clinical trials.

WHAT THIS STUDY ADDS: The RDAI is anincomplete measure of respiratory distress inbronchiolitis, with poor to moderate constructvalidity. It has adequate discriminativeproperties but considerable test-retestmeasurement error. The RDAI and RACS weremoderately responsive, but methodologic issueslimit the interpretation of this finding.

aDepartment of Pediatrics, Santa Maria Hospital, Lisbon Academic Medical Centre, Lisbon, Portugal; bClinicalPharmacology Unit, Instituto de Medicina Molecular, University of Lisbon, Lisbon, Portugal; cUniversity of Ottawaand Children’s Hospital of Eastern Ontario, Ottawa, Canada; dDepartment of Epidemiology and Biostatistics andeEMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, Netherlands; fManitobaInstitute of Child Health, University of Manitoba, Winnipeg, Canada; gChildHealth Evaluative Sciences, Hospital forSick Children, Toronto, Canada; and hDivision of Woman and Child, Pediatric Clinical Research Office, AcademicMedical Centre, Amsterdam, Netherlands

Dr Fernandes conceptualized and designed the study, carried out all analyses, and drafted theinitial manuscript; Dr Plint was the principal investigator for both original studies on which thisstudy is based; she reviewed the study protocol and critically reviewed the manuscript; Drs Terweeand Sampaio critically reviewed the manuscript; Dr Klassen reviewed the study protocol andcritically reviewed the manuscript; Dr Offringa conceptualized and designed the study and criticallyreviewed the manuscript; Dr van der Lee, conceptualized and designed the study, supervised allanalyses, and critically reviewed the manuscript; and all authors approved the final manuscript assubmitted.

PEDIATRICS Volume 135, number 6, June 2015 ARTICLE by guest on July 16, 2018www.aappublications.org/newsDownloaded from

Acute viral bronchiolitis is the mostcommon lower respiratory tractinfection in infants and carriessubstantial clinical and financialburden.1,2 There is wide practicevariation in its management, withheterogeneous evidence for manytherapeutic approaches.3–7

Systematic reviews have highlightedvarious shortcomings in randomizedcontrolled trials (RCTs) in thisfield.8–10 One of the major issues isthe heterogeneous choice of outcomemeasures. There has beeninconsistency in selectedmeasurement instruments, whosemeasurement properties have oftennot been adequately studied.8,11

Respiratory status is an importantdimension and determinant ofseverity in bronchiolitis. TheRespiratory Distress AssessmentInstrument (RDAI) and RespiratoryAssessment Change Score (RACS) areoften used to measure this domain inbronchiolitis.8 Lowell et al12 firstdescribed them in an RCT ofepinephrine in wheezing infants. TheRDAI includes items on retractionsand wheezing, whereas the RACS isa change score based on the RDAI andrespiratory rate. Evidence is limitedregarding RDAI and RACSmeasurement properties and theirsuitability for use as evaluativeinstruments in clinical trials.13–15

Previous RCTs have reported somedata on which reliability and validitycan be assessed, whereas the firstformal validation study is recent.16

The aim of this study was to assessand compare the measurementproperties of RDAI and RACS, that is,validity, reliability, andresponsiveness.

METHODS

Population

We used data from 2 related studiesconducted simultaneously in8 Canadian pediatric emergencydepartments (EDs) during3 bronchiolitis seasons (2004–2007): a2 3 2 factorial RCT (CanadianEpinephrine/Steroid Trial, CanBEST;N = 800)17 and a prospective cohortstudy (N = 1554 infants, 584 of whomalso participated in CanBEST).18 Bothstudies included infants aged ,12months with acute bronchiolitis (firstepisode of wheezing) and excludedthose with previous asthma, wheezing,or use of bronchodilators. Additionalexclusion criteria in CanBEST were asfollows: prematurity with corrected age,6 weeks, chronic cardiopulmonarydisease or immunodeficiency, recentcorticosteroid use or exposure tovaricella, very mild or severe distress(pulse rate .200 beats per minute,respiratory rate .80 breaths perminute, or RDAI score ,4 or .15), orlethargy.

Participants in CanBEST wererandomly assigned to receive oraldexamethasone or placebo andnebulized epinephrine or placebo,both administered in the ED (Fig 1).During the first 90 minutes, only

supplemental oxygen oracetaminophen was allowed. Otherparticipants in the cohort study weregiven standard treatment as decidedby the attending physicians. In bothstudies, written informed consentwas obtained from the parents orguardians of the infants, and bothwere approved by ethics committeesat each site and by Health Canada.

Instruments and Outcome Measures

We assessed the RDAI as describedby Lowell et al12 and a modificationof the RACS as reported by Schuhet al19 (Tables 1 and 2). Thefollowing measurements wereperformed at baseline for bothstudies and every 30 minutes untiladmission/discharge or 240 minutesfor CanBEST: RDAI, respiratory rate,heart rate, oxygen saturation (SaO2),and activity status (Fig 1). Fever wasalso assessed at baseline. Researchnurses performed all measurementsafter formal training and usingwritten instructions. SaO2 wasmeasured by using pulse oximetersavailable locally. In both studies, theattending physician independentlydetermined whether to admit ordischarge the infant; RDAI was notused clinically at any site. InCanBEST, physicians and nurseswere blinded to treatmentinterventions, and by protocol anydecisions regarding admission,discharge, or continued stay in theED were to be made only after thestudy interventions (ie, after90 minutes).

FIGURE 1Timing of intervention, measurements, and clinical decisions in the CanBEST trial. Dex, dexamethasone; Epi, epinephrine; Pla, placebo; RR, respiratoryrate.

e1400 FERNANDES et al by guest on July 16, 2018www.aappublications.org/newsDownloaded from

Statistical Analysis

We used the Consensus-basedStandards for the selection of healthMeasurement Instruments initiative(COSMIN) definitions ofmeasurement domains andproperties regarding validity,reliability, and responsiveness.20

Construct Validity of the RDAI

There is no “gold standard” to assessbronchiolitis severity or respiratorydistress. We assessed constructvalidity of the RDAI by formulatinghypotheses about the direction andmagnitude of the association of RDAIscores with both physiologicmeasures (respiratory rate, SaO2)and clinical decision-makingconstructs (decision to admit/discharge and time to admission/discharge).21 We studied bothconvergent and discriminativevalidity.

We hypothesized that baseline RDAIscores and respiratory rate wouldhave a strong positive correlation(Pearson’s r $ 0.7). We used multiplelinear regression analysis to explorepossible confounding of thisassociation, by activity status andfever (data from both studies) andage and weight (data from CanBEST).We further hypothesized a negativeassociation between RDAI and SaO2,which we expected to be weaker andnonlinear (Spearman’s r#20.5), andwe compared RDAI scores between3 categories of SaO2 (,92%, 92%–95%,.95%; data from both studies).

We hypothesized that a higher RDAIscore would increase the risk ofadmission (expected odds ratio [OR]for admission: $1.5 for RDAI scoresabove the median). For this analysis,we used the last RDAI score assessedor registered before the time ofadmission/discharge (data from

CanBEST). Multiple logistic regressionanalysis was used to evaluate whetherthat association was confounded bycenter, treatment group, age, and SaO2.Furthermore, we expected participantswho stayed in the ED longer to haveintermediate scores compared withthose who were admitted ordischarged sooner (data from CanBESTafter the main trial interventions).

Reliability of the RDAI

For RDAI test-retest reliability, weconsidered that the group of CanBESTparticipants who had received bothplacebo interventions was stablebetween the 90- and 120-minutemeasurements. The same researchnurse assessed the same childunblinded to the previousassessment. In a convenience sampleof participants from each study,2 nurses performed baseline RDAImeasurements independently toobtain interrater assessments.

For both test-retest and interraterconditions, we distinguishedmeasures of measurement error fromreliability measures.22 To evaluatemeasurement error we calculated thestandard error of measurement(SEM) and, the smallest detectablechange (SDC), and we obtaineda Bland-Altman plot and the 95%limits of agreement (LoA; formulasare shown in SupplementalInformation). The Bland-Altman plotshows the mean differences betweenthe test and retest scores (expressed

TABLE 1 Respiratory Distress Assessment Instrument (RDAI)

Variable Score Range

0 1 2 3 4

Wheezing (auscultation)Expiration None End 1/2 3/4 All 0–4Inspiration None Part All 0–2Location None Segmental: #2 of 4 lung fields Diffuse: $3 of 4 lung fields 0–2

Partial sum score 0–8Retractions (visual assessment)Supraclavicular None Mild Moderate Marked 0–3Intercostal None Mild Moderate Marked 0–3Subcostal None Mild Moderate Marked 0–3Partial sum score 0–9

Sum score (higher scores indicatemore severe disease)

0–17

The original RDAI as reported by Lowell et al12 also included respiratory rate, ie, it did not differentiate between RDAI and RACS and only used RACS as an outcome measure.

TABLE 2 Respiratory Assessment Change Score (RACS)

Variable Formula Range

Wheezing change score Final partial sum score 2 baseline partial sumscore

28 to +8

Retractions change score Final partial sum score 2 baseline partial sumscore

29 to +9

Respiratory rate “standardized”change score

5% change: 0 units 2n to +n6% to 15% change: 21/+1 units16% to 25%: 22/+2 units

etcSum score (negative change scores

indicate improvement)2172 n to +17 + n

As modified by Schuh et al.19 In the original RACS as reported by Lowell et al,12 final scores were subtracted from baselinescores (ie, positive change scores indicated improvement), and cutoffs to define respiratory rate change were defined at10% intervals.

PEDIATRICS Volume 135, number 6, June 2015 e1401 by guest on July 16, 2018www.aappublications.org/newsDownloaded from

in the unit of the scale) along therange of the scale. The LoA show thescores between which 95% of thesedifferences lie. We assessed reliabilityby calculating the intraclasscorrelation coefficient (ICC) in a2-way random-effects model, includingpatient, observer, and residual variancecomponents (formula shown inSupplemental Information).

Responsiveness of the RDAI/RACS

As with validity, we studied theresponsiveness of the RDAI and RACSthrough testing hypothesesconcerning the expected associationsof change scores.21 There is no clearcriterion for change in bronchiolitisand none of the studies includedexplicit assessments of change. Webased our hypotheses on physiologicand clinical constructs of changeusing a priori–defined criteria toidentify groups of participants whoimproved versus those who hadremained stable or deteriorated,irrespective of interventions. Criteriawere based on respiratory rate andSaO2, and we used combined groupdata from CanBEST (Table 3).

We used different measures ofresponsiveness assessing statisticalchange or clinically important change,focusing on comparing the improvedgroup with the stable/deterioratedgroup.23 These included thefollowing: testing differences in RDAIchange scores and/or RACS within

and between groups, and calculatingstandardized/Cohen’s effect size (ES)and the responsiveness ratio(formulas shown in SupplementalInformation). We hypothesized thatpatients who had improved wouldhave larger change scores and ESsthan patients who had not improved.We also used the area under thecurve (AUC) of the receiver operatingcharacteristic curve of improvedversus stable groups (AUC .0.70considered appropriate).

For all analyses, we excludedparticipants with nonvalid or missingdata, with no imputation. Statisticalsignificance was set at P, .05, and wecalculated 95% confidence intervals(CIs) when applicable. We used SPSSversion 19 (IBM SPSS Statistics, IBMCorporation, Armonk, NY).

RESULTS

Figure 2 shows data sources andparticipants included in the analysisof each measurement property. Thebaseline characteristics ofparticipants and selected outcomesfrom both studies are presented inTable 4. Participants in CanBESTwere older than those in the cohortstudy, whereas baseline severity wasgreater in the latter study.

Construct Validity of the RDAI

We found a weak positive correlationbetween RDAI score and respiratory

rate at baseline with data from bothstudies (Pearson’s r = 0.38; 95% CI:0.35 to 0.45; P , .001; N = 1765).Correlations for retractions andwheezing subscores were r = 0.41and r = 0.17, respectively. By usingsimple linear regression, thecoefficient estimate was a 1.55 (95%CI: 1.38 to 1.73) increase inrespiratory rate (breaths per minute)per increase in RDAI unit (P , .001).The estimate was comparable whenadjusting for fever and activity status(adjusted estimate: 1.52). Whenrestricting the analysis to CanBESTdata, the correlation was weaker(Pearson’s r = 0.22; unadjusted linearregression estimate: 0.98; n = 800).The association was not confoundedby age, weight, fever, or activity status(adjusted estimate: 0.92).

There was a weak negativecorrelation between baseline RDAIscores and SaO2 levels (Spearman’sr = 20.24; P , .001; n = 1761).Correlations for retractions andwheezing were r = 20.25 and r =20.14, respectively. RDAI scoresincreased in lower SaO2 categories(Fig 3). The median (interquartilerange) RDAI scores were 10 (8–12),8 (6–10), and 7 (5–10) for SaO2 ,92%,92%–95%, and .95%, respectively(Kruskal-Wallis test, P , .001).

We found an association between thedecision to admit or discharge andthe last RDAI score of CanBEST

TABLE 3 Constructs of Change and Criteria Used to Assess Responsiveness of RDAI and RACS

Anchor of Change Population Timing of Measurements Improved Group Stable/Deteriorated Group

Inclusion n Criteria n Criteria n

Change in respiratory rate(relative change)a

All participants 796 At baseline and lastmeasurement beforeadmission or discharge

$25% reduction inrespiratory rate

204 #25% reduction inrespiratory rate

592

Change in respiratory rate(tachypnea)a

Participants with tachypneaat baseline (respiratoryrate .50 breaths/minute[,6 mo] or .40 breaths/minute [6–12 mo])

305 At baseline and lastmeasurement beforeadmission or discharge

Reduction in respiratoryrate below tachypneacut-off

96 No reduction inrespiratory rate belowtachypnea cutoff

209

Probability of admission atbaseline (versus actualdecision)

Participants with highbaseline probability ofadmission (respiratoryrate .60 breaths/minuteor SaO2 ,90%)

209 At baseline and lastmeasurement beforeadmission or discharge

Discharge 154 Admission 55

a Only for RDAI; not used to measure RACS responsiveness because respiratory rate is included in the RACS formula.

e1402 FERNANDES et al by guest on July 16, 2018www.aappublications.org/newsDownloaded from

participants. The preceding RDAIscore was higher in admitted patientsthan in those who were discharged(mean difference: 2.28; 95% CI: 1.75to 2.81; t test, P , .001; n = 798).A higher RDAI score was associatedwith higher risk of admission(OR:1.36; 95% CI: 1.26 to 1.47) perincrease in RDAI unit; and 2.54(95% CI: 1.65 to 3.92) when the RDAIwas . 8). Adjusted analyses forcenter, treatment group, age, and SaO2revealed no relevant changes in theseassociations.

In addition, we found that RDAIscores measured after CanBESTinterventions differed between thegroups of participants who weredischarged (median [interquartilerange]: 5 [2–6]), hospitalized(8 [5–10]), or who stayed in the ED(6 [4–8]) (n = 695; Kruskal-Wallis test,P , .001) (Fig 4). Differences betweendischarged participants and the 2latter groups were statisticallysignificant (P = .01 and P , .01,

respectively; Bonferroni post hoc test).Patients with a higher RDAI score at90 minutes had higher risk of ED stay.240 minutes (OR: 1.33; 95% CI: 1.24to 1.43 per increase in RDAI unit).Overall, although results were inaccordance with our validityhypotheses, the magnitude of theassociations was mostly below ourpredefined thresholds.

Reliability of the RDAI

Test-retest assessments wereavailable from 79 CanBESTparticipants. The mean differencebetween the 2 repeated assessmentswas 0.08 (95% CI: 20.35 to 0.5; P =.72). The SEM was 1.34 and the SDCwas 3.72 RDAI units. The test-retest95% LoA were 23.8 to 3.64 RDAIunits (95% CI: 24.53 to 23.07, to2.91 to 4.37). This finding means thatif a child is assessed twice, the secondscore could be between 3.64 pointslower and 3.8 points higher than thefirst score, just because of

measurement error. The magnitudesof differences between repeatedmeasurements remained the sameover the whole range of mean valuesas shown in the Bland-Altman plot(Fig 5). The test-retest ICC was 0.80(95% CI: 0.70 to 0.87).

Interrater assessments wereperformed in 107 participants. Therewas no significant difference between2 repeated assessments (meandifference: 20.06; 95% CI: 20.28 to0.15; P = .54). The SEM was 0.78, andthe interrater LoA were 22.1 to 2.22RDAI units (95% CI: 22.46 to 21.74,to 1.86 to 2.58]. The ICC was 0.93(95% CI: 0.9 to 0.94).

Responsiveness of the RDAI andRACS

Measures of responsiveness for RDAIand RACS based on the differentconstructs of change are presented inTable 5. By using both anchors, themean RDAI scores decreased in bothimproved and stable groups (pairedt test, P , .001; for all within-groupcomparisons), with larger meanchanges in scores of the improvedgroup (unpaired t test, P , .001; forall between-group comparisons).These results were in accordancewith our predefined hypotheses.Between-group differences in meanRDAI change scores ranged from21.31 (95% CI: 21.85 to 20.77) forthe 25% respiratory rate reductioncriterion to 22.03 (95% CI: 22.9 to21.16) for the probability ofadmission criterion. Standardized ESsfor the improved group ranged from1.43 to 1.71, whereas responsivenessratios ranged from 1.54 to 1.61, andAUCs from 0.64 to 0.7. The RACS waslarger in the improved group(between-group difference: 22.81;95% CI: 23.92 to 21.7), witha responsiveness ratio of 1.96 and anAUC of 0.72.

DISCUSSION

This study of measurementproperties of RDAI and RACS in acutebronchiolitis identifies strengths and

FIGURE 2Sources of data and number of participants included in the analysis of each measurement property.

PEDIATRICS Volume 135, number 6, June 2015 e1403 by guest on July 16, 2018www.aappublications.org/newsDownloaded from

limitations of their use as outcomemeasures. The RDAI was evaluated in3 systematic reviews of measurementproperties of asthma or wheezingseverity scales in children.13–15

Limited data on its reliability andresponsiveness were provided in theoriginal description of the scale andin later reports of RCTs.12,19,24

However, none of these wereadequately designed measurementstudies, and no formal assessment ofvalidity was found. Destino et al16

recently reported the first validationstudy on RDAI in bronchiolitis,showing poor construct validity,interrater reliability, andresponsiveness. Findings on validitywere fairly consistent with ourresults; differences in setting, raters,and methods may explain why resultson reliability and responsivenesswere distinct.

Our results show that the RDAI haspoor to moderate construct validity.The RDAI was developed ad hoc withno elaboration on the underlyingconceptual model, item selection,scoring, or weighting. Although inthe original report only the RACSwas used as an outcome measure,later trials used RDAI scoresseparately for single or repeatedassessments.12 In our conceptualframework, respiratory distress wasputatively reflected by RDAI items(ie, reflective model) andcontributed to the multidimensionalconstruct of bronchiolitis.21 Wefound poor convergent validity withrespiratory rate, but RDAI scoresdiscriminated well between clinicallymeaningful SaO2 subgroups.Measurement properties from otherrespiratory scales or their individualitems, which often includerespiratory rate or SaO2, are seldomavailable.13–15 When they arereported, there is substantialheterogeneity in correlations withSaO2, ranging from poor to moderate.Thus, our predefined cutoffs mayhave been too strict. Most, but notall, studies are consistent with ourfindings of weaker correlations

FIGURE 3Box plot displaying baseline RDAI scores by categories of SaO2. The box spans the interquartile range(IQR), the solid horizontal line through the box is the median value, and the whiskers denote valueswithin 1.5 IQRs lower than the first quartile and 1.5 IQRs higher than the third quartile.

TABLE 4 Baseline Characteristics of Participants and Selected Outcomes From the CanBEST Trialand the Cohort Study

Baseline Characteristics CanBEST Trial (N = 800) Cohort Study (N = 1554)a

Demographic characteristicsAge, median (IQR), mo 5 (3–7) 4 (2–7)Male gender, n (%) 493 (62) 948 (61)White, n (%) 654 (82) 1243 (80)

HistoryPersonal history of atopy, n (%) 89 (11) 157 (10)Prematurity, n (%) 83 (10) 202 (13)Household smoking, n (%) 305 (38) 575 (37)Symptom length, median (IQR), d 4 (2–5) 4 (2–5)

Clinical characteristicsRespiratory rateBreaths/minute, median (IQR) 48 (42–58) 48 (42–60).60 Breaths/minute, n (%) 196 (25) 441 (28)

Oxygen saturation%, median (IQR) 97 (95–98) 97 (95–98),90%, n (%) 24 (3) 121 (8)

Heart rateBeats/minute, median (IQR) 150 (139–160) 152 (140–164).180 Beats/minute, n (%) 33 (4) 143 (9)RDAI score

Median (IQR) 8 (6–10) 8 (6–10).12, n (%) 76 (10) 211 (14)

Patient outcomesAdmission during enrollment visit, n (%) 119 (15) 370 (24)Time to admission/discharge, n (%),90 minutes 103 (13) NA90–120 minutes 261 (33)120–240 minutes 248 (31).240 minutes 188 (23)

IQR, interquartile range; NA, not applicable.a Data from the severity (cohort) study include n = 584 participants who were also included in CanBEST.17,18

e1404 FERNANDES et al by guest on July 16, 2018www.aappublications.org/newsDownloaded from

between SaO2 and auscultatory itemswhen compared with work ofbreathing.13–15,25

These results reflect thepathophysiology and clinicalcorrelates of respiratory distress in

bronchiolitis. It is known that asdisease progresses and severityincreases, so do the disturbances inventilation and ventilation-perfusionmatching.26,27 Many patients haveeffective compensatory mechanismsfor these disturbances, althoughothers do not. However, clinical signsof respiratory distress may notcapture hypoxemia/hypercapniabalance equally. Furthermore, thecorrelation between SaO2 (reflectingoxygenation), and respiratory rate(also dependent on respiratory driveand ventilation) varies acrossconditions.25 Therefore, the RDAIlikely does not represent alldimensions of respiratory distress inbronchiolitis, and a combination ofparameters may be more relevant forthe measurement of respiratorydistress, as seen in formallydeveloped scales.28,29 However, mostother scales were not developedspecifically for bronchiolitis, and theirmeasurement properties cannot betransferred between differentrespiratory conditions withoutfurther validation.

We found that the RDAI hadreasonable predictive validity basedon its association with hospitalizationand length of stay in the ED. Ourfindings are consistent with those ofCorneli et al,30 who identified RDAIscore, SaO2, and respiratory rate aspredictors of hospitalization inbronchiolitis. On the contrary, Destinoet al16 found that RDAI sum scoresdid not discriminate well betweenadmitted and discharged patients, butthe item on retractions did. Two largeprognostic studies have alsoidentified retractions as predictors ofsevere disease in ED and hospitalizedpatients.31,32 Decisions regardinghospitalization and length of stay inthe ED are multifactorial.Nonrespiratory severity parameters(eg, feeding), prognostic factors(eg, age), social issues, clinicaljudgment, available resources, andlocal practices influence decision-making.30 Furthermore, there arelimits to the validity of static

FIGURE 4Box plot displaying RDAI scores and clinical decisions at 90 minutes. The box spans the interquartilerange (IQR), the solid horizontal line through the box is the median value, and the whiskers denotevalues within 1.5 IQRs lower than the first quartile and 1.5 IQRs higher than the third quartile.

FIGURE 5Bland-Altman plot of the difference between test-retest RDAI scores at t1 (90 minutes) and t2 (120minutes) plotted against the mean value of both scores. The central line corresponds to the averagedifference between 2 RDAI scores (which reflects systematic error), whereas the lower and upperdotted lines correspond to lower and upper 95% LoA (which reflect random error), respectively.

PEDIATRICS Volume 135, number 6, June 2015 e1405 by guest on July 16, 2018www.aappublications.org/newsDownloaded from

measurements of respiratory distressin a highly dynamic condition. Froman outcome measure perspective, theRDAI does not encompass alldeterminants of bronchiolitis severity.

Interrater reliability measured by theICC was good, both at the group andindividual level, as was interratermeasurement error. These findingsmean that RDAI scores canadequately discriminate participantsassessed by different raters at thesame time point in both clinical andresearch settings. Data from previousRCT reports also showed goodinterrater reliability, but Destinoet al16 found a strikingly low ICC.Differences may relate to training,familiarity with the instrument,raters, and population heterogeneity.On the other hand, we foundconsiderable test-retest measurementerror at the individual level, becausea patient should change at least closeto 4 points (approximately one-fifthof the scale) before a change isdetectable beyond measurementerror. Thus, in clinical practice,changes in individual patients shouldbe interpreted with caution. For theRACS, we must also considermeasurement error for respiratoryrate.33,34 The SDC is paramount tointerpretability parameters such asthe minimal important change (MIC),because a large SDC relative to theMIC means that observed change may

be caused by measurement errorrather than change per se.35 At theindividual level, taking repeatedmeasurements and averaging thevalue would reduce the measurementerror with a factor √k (k is thenumber of measurements). Althoughreassessment is a key componentwhen evaluating children withrespiratory distress, many repeatedmeasurements might not be practicalin clinical practice. At the group level,the SDC of a mean change is equal toSDC/√n, which reduces its impact.21

Because the ICC was high, the RDAI isreliable for use in studies. Overall,these results suggest that the RDAIhas adequate discriminativeproperties, but test-retestmeasurement error should beminimized.

The RDAI was responsive accordingto our predefined hypotheses basedon 2 distinct constructs of change.Previous data on RDAIresponsiveness are scarce.13–15

Hardly any intervention can beconsidered clearly effective inbronchiolitis in the ED setting, andthus none is a reasonable goldstandard to assess change. Destinoet al16 reported a mild correlationbetween the change in the RACS andthe Children’s Hospital of WisconsinRespiratory Score, but data onresponsiveness of this latter scale arealso missing. We anchored our

constructs of change on physiologicchange and change in clinical statuslikely to be relevant for decisionsregarding patient disposition at theED. Measures of responsiveness thattook into account both improved andstable groups (responsiveness ratioand AUC) were comparable betweenanchors for the RDAI. The AUC valuewas close to the frequently usedcutoff of acceptability (0.7) for boththe RDAI and RACS, with the RACSbeing slightly more responsive. Thesedata suggest that the RDAI and RACSare moderately responsive, but anycomparison with other respiratoryscales is limited.

Our study has limitations related todesign constraints of both includedstudies. First, less heterogeneity ofRDAI scores in the selected sample ofCanBEST participants may explainwhy we found a weaker correlationwith respiratory rate and lower test-retest ICC scores. Further validationis needed when considering childrenwith very mild or severe disease, whowere excluded in CanBEST. Second,our results are applicable to infantswith a first episode of wheezing andno relevant comorbidities and shouldbe interpreted with caution whendefining bronchiolitis differently inother populations.36 Third,concurrent factors that affectdecisions of hospitalization were notcollected, and the exact timing of this

TABLE 5 Measures of Responsiveness for RDAI and RACS Using Different Anchors of Change

Responsiveness Measure Anchor

RDAI Probability of Admission (n = 209)

Respiratory Rate 25%Reduction (n = 796)

Respiratory Rate TachypneaCutoff (n = 305)

RDAI RACS

Within-group mean change (RDAI) orchange score (RACS) 6 SDImproved group 24.17 6 2.86 24.48 6 3.07 23.63 6 2.96 25.94 6 3.76Stable group 22.86 6 2.71 22.8 6 2.78 21.6 6 2.36 23.13 6 3.03

Between-group difference in mean change(RDAI) or change score (RACS) (95% CI)

21.31 (21.85 to 20.77) 21.67 (22.37 to 20.97) 22.03 (22.9 to 21.16) 22.81 (23.92 to 21.7)

Standardized ESImproved group 1.54 1.71 1.43 NAStable group 1.22 1.15 0.67 NA

Responsiveness ratio 1.54 1.61 1.54 1.96AUC (95% CI) 0.64 (0.59–0.68) 0.65 (0.59–0.71) 0.7 (0.63–0.78) 0.72 (0.64–0.8)

Measurements were made at baseline and before admission or discharge. NA, not applicable.

e1406 FERNANDES et al by guest on July 16, 2018www.aappublications.org/newsDownloaded from

decision was not known. Although, inideal conditions, managing physicianswould be blinded to RDAI/RACSscores, blinding to their individualitems is not expected. Finally, definingstability and change can beproblematic and time-dependent dueto the dynamic nature ofbronchiolitis. When assessingresponsiveness by using datacollected at different time points(mostly between 90 and 240minutes), we observed significantimprovements in RDAI scores ingroups that we considered a priori tobe stable. This finding is likelya limitation of our anchors and may

also reflect the effect of supportivemeasures and the nebulized“placebo.” These limitations should beconsidered when calculating the MICof the RDAI, which will be the focus offuture work.

In conclusion, we found the RDAI tobe an incomplete measure ofrespiratory distress in bronchiolitis,with poor to moderate constructvalidity and adequate interraterreliability. The RDAI hadconsiderable test-retestmeasurement error, and althoughboth the RDAI and RACS weremoderately responsive,

methodologic issues may limit theinterpretation of this finding. Finally,the RDAI does not encompass alldeterminants of bronchiolitisseverity.

ACKNOWLEDGMENTS

We thank the children and familieswho participated in CanBEST and theprognostic cohort study, the siteresearch coordinators, thecoordinating site staff, the datamanager, all of the research nursesand participating site physicians andnurses, and Pediatric EmergencyResearch Canada members.

This work was presented at the annual international congress of the European Respiratory Society; September 6–10, 2014; Munich, Germany.

www.pediatrics.org/cgi/doi/10.1542/peds.2014-3557

DOI: 10.1542/peds.2014-3557

Accepted for publication Mar 18, 2015

Address correspondence to Ricardo M. Fernandes, MD, Clinical Pharmacology Unit, Instituto de Medicina Molecular, University of Lisbon, Avenida Professor Egas

Moniz, 1649-028 Lisbon, Portugal. E-mail: rmfernandes@campus.ul.pt

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).

Copyright © 2015 by the American Academy of Pediatrics

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

FUNDING: Funding for the data collection for this study was provided by a grant from the Canadian Institutes of Health Research and the Ontario Thoracic Society.

Dr Fernandes was supported by the Programme for Advanced Medical Education, sponsored by Fundação Calouste Gulbenkian, Fundação Champalimaud, Ministério

da Saúde and Fundação para a Ciência e Tecnologia, Portugal. Dr Plint is supported by a University of Ottawa Research Chair in Pediatric Emergency Medicine.

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

REFERENCES

1. Smyth RL, Openshaw PJ. Bronchiolitis.Lancet. 2006;368(9532):312–322

2. Zorc JJ, Hall CB. Bronchiolitis: recentevidence on diagnosis and management.Pediatrics. 2010;125(2):342–349

3. Bialy L, Foisy M, Smith M, Fernandes RM.The Cochrane Library and the treatmentof bronchiolitis in children: an overviewof reviews. Evid Based Child Health. 2011;6(1):258–275

4. Hartling L, Fernandes RM, Bialy L, et al.Steroids and bronchodilators for acutebronchiolitis in the first two years of life:systematic review and meta-analysis.BMJ. 2011;342:d1714

5. Plint AC, Johnson DW, Wiebe N, et al.Practice variation among pediatricemergency departments in the

treatment of bronchiolitis. Acad EmergMed. 2004;11(4):353–360

6. Mansbach JM, Emond JA, Camargo CA Jr.Bronchiolitis in US emergencydepartments 1992 to 2000: epidemiologyand practice variation. Pediatr EmergCare. 2005;21(4):242–247

7. Florin TA, Byczkowski T, Ruddy RM, ZorcJJ, Test M, Shah SS. Variation in themanagement of infants hospitalized forbronchiolitis persists after the 2006American Academy of Pediatricsbronchiolitis guidelines. J Pediatr. 2014;165(4):786–792

8. Fernandes RM, Bialy LM, Vandermeer B,et al. Glucocorticoids for acute viralbronchiolitis in infants and youngchildren. Cochrane Database Syst Rev.2013;6:CD004878

9. King VJ, Viswanathan M, Bordley WC,et al. Pharmacologic treatment ofbronchiolitis in infants and children:a systematic review. Arch PediatrAdolesc Med. 2004;158(2):127–137

10. Gadomski AM, Scribani MB.Bronchodilators for bronchiolitis. CochraneDatabase Syst Rev. 2014;6:CD001266

11. Klassen TP. Determining the benefit ofbronchodilators in bronchiolitis: when isthere enough benefit to warrant adoptioninto clinical practice? Arch PediatrAdolesc Med. 1996;150(11):1120–1121

12. Lowell DI, Lister G, Von Koss H, McCarthy P.Wheezing in infants: the response toepinephrine. Pediatrics. 1987;79(6):939–945

13. van der Windt DA, Nagelkerke AF, BouterLM, Dankert-Roelse JE, Veerman AJ.

PEDIATRICS Volume 135, number 6, June 2015 e1407 by guest on July 16, 2018www.aappublications.org/newsDownloaded from

Clinical scores for acute asthma in pre-school children: a review of the literature.J Clin Epidemiol. 1994;47(6):635–646

14. Birken CS, Parkin PC, Macarthur C.Asthma severity scores for preschoolersdisplayed weaknesses in reliability,validity, and responsiveness. J ClinEpidemiol. 2004;57(11):1177–1181

15. Bekhof J, Reimink R, Brand PL.Systematic review: insufficient validationof clinical scores for the assessment ofacute dyspnoea in wheezing children.Paediatr Respir Rev. 2014;15(1):98–112

16. Destino L, Weisgerber MC, Soung P, et al.Validity of respiratory scores in bronchiolitis.Hosp Pediatr. 2012;2(4):202–209

17. Plint AC, Johnson DW, Patel H, et al;Pediatric Emergency Research Canada(PERC). Epinephrine and dexamethasonein children with bronchiolitis. N Engl JMed. 2009;360(20):2079–2089

18. Plint AC, Johnson DW, Correll R, et al.Predicting the Development of SevereBronchiolitis Among an EmergencyDepartment Cohort: A Multi-CenterStudy. Pediatric Academic Societies;2008. Available at: www.abstracts2view.com/pasall/. Accessed November 2014

19. Schuh S, Coates AL, Binnie R, et al.Efficacy of oral dexamethasone inoutpatients with acute bronchiolitis.J Pediatr. 2002;140(1):27–32

20. Mokkink LB, Terwee CB, Patrick DL, et al.The COSMIN study reached internationalconsensus on taxonomy, terminology,and definitions of measurementproperties for health-related patient-reported outcomes. J Clin Epidemiol.2010;63(7):737–745

21. De Vet H, Terwee C, Mokkink L, Knol D.Measurement in Medicine: A Practical

Guide. Cambridge, United Kingdom:Cambridge University Press; 2011

22. de Vet HC, Terwee CB, Knol DL, Bouter LM.When to use agreement versus reliabilitymeasures. J Clin Epidemiol. 2006;59(10):1033–1039

23. Terwee CB, Dekker FW, Wiersinga WM,Prummel MF, Bossuyt PM. On assessingresponsiveness of health-related qualityof life instruments: guidelines forinstrument evaluation. Qual Life Res.2003;12(4):349–362

24. Klassen TP, Sutcliffe T, Watters LK, WellsGA, Allen UD, Li MM. Dexamethasone insalbutamol-treated inpatients with acutebronchiolitis: a randomized, controlledtrial. J Pediatr. 1997;130(2):191–196

25. Zhang L, Mendoza-Sassi R, Santos JC, LauJ. Accuracy of symptoms and signs inpredicting hypoxaemia among youngchildren with acute respiratory infection:a meta-analysis. Int J Tuberc Lung Dis.2011;15(3):317–325

26. Javouhey E, Pouyau R, Massenavette B.Pathophysiology of Acute RespiratoryFailure in Children with Bronchiolitis andEffect of CPAP. In: Esquinas AM, ed.Noninvasive Ventilation in High-RiskInfections and Mass Casualty Events:Vienna, Austria: Springer; 2014:233–249

27. Thorburn K, McNamara P. Bronchiolitis.In: Wheeler DS, Wong HR, Shanley TP, eds.Pediatric Critical Care Medicine. London,United Kingdom: Springer; 2014:75–85

28. Chalut DS, Ducharme FM, Davis GM. ThePreschool Respiratory AssessmentMeasure (PRAM): a responsive index ofacute asthma severity. J Pediatr. 2000;137(6):762–768

29. Gorelick MH, Stevens MW, Schultz TR,Scribano PV. Performance of a novel

clinical score, the Pediatric AsthmaSeverity Score (PASS), in the evaluationof acute asthma. Acad Emerg Med. 2004;11(1):10–18

30. Corneli HM, Zorc JJ, Holubkov R, et al;Bronchiolitis Study Group for thePediatric Emergency Care AppliedResearch Network. Bronchiolitis: clinicalcharacteristics associated withhospitalization and length of stay.Pediatr Emerg Care. 2012;28(2):99–103

31. Mansbach JM, Piedra PA, Stevenson MD,et al; MARC-30 Investigators. Prospectivemulticenter study of children withbronchiolitis requiring mechanicalventilation. Pediatrics. 2012;130(3).Available at: www.pediatrics.org/cgi/content/full/130/3/e492

32. Damore D, Mansbach JM, Clark S,Ramundo M, Camargo CA Jr. Prospectivemulticenter bronchiolitis study:predicting intensive care unitadmissions. Acad Emerg Med. 2008;15(10):887–894

33. Simoes EA, Roark R, Berman S, Esler LL,Murphy J. Respiratory rate: measurementof variability over time and accuracy atdifferent counting periods. Arch Dis Child.1991;66(10):1199–1203

34. Rusconi F, Castagneto M, Gagliardi L,et al. Reference values for respiratoryrate in the first 3 years of life. Pediatrics.1994;94(3):350–355

35. Terwee CB, Roorda LD, Knol DL, De BoerMR, De Vet HC. Linking measurementerror to minimal important change ofpatient-reported outcomes. J ClinEpidemiol. 2009;62(10):1062–1067

36. Everard ML. Acute bronchiolitis andcroup. Pediatr Clin North Am. 2009;56(1):119–133, x–xi

e1408 FERNANDES et al by guest on July 16, 2018www.aappublications.org/newsDownloaded from

DOI: 10.1542/peds.2014-3557 originally published online May 18, 2015; 2015;135;e1399Pediatrics 

Klassen, Martin Offringa and Johanna H. van der LeeRicardo M. Fernandes, Amy C. Plint, Caroline B. Terwee, Cristina Sampaio, Terry P.

Validity of Bronchiolitis Outcome Measures

ServicesUpdated Information &

http://pediatrics.aappublications.org/content/135/6/e1399including high resolution figures, can be found at:

Referenceshttp://pediatrics.aappublications.org/content/135/6/e1399#BIBLThis article cites 30 articles, 6 of which you can access for free at:

Subspecialty Collections

http://www.aappublications.org/cgi/collection/bronchiolitis_subBronchiolitishttp://www.aappublications.org/cgi/collection/pulmonology_subPulmonologysubhttp://www.aappublications.org/cgi/collection/emergency_medicine_Emergency Medicinefollowing collection(s): This article, along with others on similar topics, appears in the

Permissions & Licensing

http://www.aappublications.org/site/misc/Permissions.xhtmlin its entirety can be found online at: Information about reproducing this article in parts (figures, tables) or

Reprintshttp://www.aappublications.org/site/misc/reprints.xhtmlInformation about ordering reprints can be found online:

by guest on July 16, 2018www.aappublications.org/newsDownloaded from

DOI: 10.1542/peds.2014-3557 originally published online May 18, 2015; 2015;135;e1399Pediatrics 

Klassen, Martin Offringa and Johanna H. van der LeeRicardo M. Fernandes, Amy C. Plint, Caroline B. Terwee, Cristina Sampaio, Terry P.

Validity of Bronchiolitis Outcome Measures

http://pediatrics.aappublications.org/content/135/6/e1399located on the World Wide Web at:

The online version of this article, along with updated information and services, is

http://pediatrics.aappublications.org/content/suppl/2015/05/12/peds.2014-3557.DCSupplementalData Supplement at:

ISSN: 1073-0397. 60007. Copyright © 2015 by the American Academy of Pediatrics. All rights reserved. Print the American Academy of Pediatrics, 141 Northwest Point Boulevard, Elk Grove Village, Illinois,has been published continuously since 1948. Pediatrics is owned, published, and trademarked by Pediatrics is the official journal of the American Academy of Pediatrics. A monthly publication, it

by guest on July 16, 2018www.aappublications.org/newsDownloaded from