Quality Indicators for High Acuity Pediatric Conditions

abstractOBJECTIVE: Identifying gaps in care and improving outcomes for se-verely ill children requires the development of evidence-basedperformance measures. We used a systematic process involvingmultiple stakeholders to identify and develop evidence-based qualityindicators for high acuity pediatric conditions relevant to anyemergency department (ED) setting where children are seen.

METHODS: A prioritized list of clinical conditions was selected by anadvisory panel. A systematic review of the literature was conducted toidentify existing indicators, as well as guidelines and evidence thatcould be used to inform the creation of new indicators. A multiphase,Rand-modified Delphi method consisting of anonymous questionnairesand a face-to-face meeting of an expert panel was used for indicatorselection. Measure specifications and evidence grading were createdfor each indicator, and the feasibility and reliability of measurementwas assessed in a tertiary care pediatric ED.

RESULTS: The conditions selected for indicator development were di-abetic ketoacidosis, status asthmaticus, anaphylaxis, status epilepti-cus, severe head injury, and sepsis. The majority of the 62 selectedindicators reflect ED processes (84%) with few indicators reflectingstructures (11%) or outcomes (5%). Thirty-seven percent (n = 23)of the selected indicators are based on moderate or high qualityevidence. Data were available and interrater reliability acceptablefor the majority of indicators.

CONCLUSIONS: A systematic process involving multiple stakeholderswas used to develop evidence-based quality indicators for high acuitypediatric conditions. Future work will test the reliability and feasibilityof data collection on these indicators across the spectrum of EDsettings that provide care for children. Pediatrics 2013;132:752–762

AUTHORS: Antonia S. Stang, MDCM, MBA, MSc,a,b Sharon E.Straus, MD, MSc, FRCPC,c Jennifer Crotts, RN,a,d David W.Johnson, MD,a,e and Astrid Guttmann, MDCM, MScf

aDivision of Emergency Medicine, Alberta Children’s Hospital,Alberta Children’s Hospital Research Institute, Departments ofbPediatrics and Community Health Sciences, dPediatrics, andePediatrics, Physiology and Pharmacology, University of Calgary,Calgary, Alberta, Canada; cDepartments of Medicine andGeriatric Medicine, University of Toronto, and KnowledgeTranslation Program, Li Ka Shing Knowledge Institute, St.Michael’s Hospital and University of Toronto, Toronto, Ontario;and fDivision of Pediatric Medicine, Hospital for Sick Children,Department of Pediatrics and Health Policy, Management andEvaluation, University of Toronto and Institute for ClinicalEvaluative Sciences, Toronto, Ontario, Canada

KEY WORDSquality improvement, quality indicators, performancemeasurement, emergency department

ABBREVIATIONSAHRQ—Agency for Healthcare Research and QualityCNS—central nervous systemCT—computed tomographyDKA—diabetic ketoacidosisED—emergency departmentGRADE—Grading of Recommendations Assessment, Developmentand EvaluationICC—intraclass correlationICD-10—International Classification of Diseases, 10th RevisionIO—intraosseousIV—intravenousNQF—National Quality ForumSRs—systematic reviews

Dr Stang conceptualized and designed the study, securedfunding, designed and piloted surveys and data extractionforms, screened articles, extracted data, interpreted data, anddrafted and revised the manuscript; Dr Straus providedmethodological advice, reviewed and revised surveys, reviewedand revised tables and figures, and revised the manuscript; MsCrotts screened articles, extracted data, coordinated expertpanel meeting, created Access database review charts, andrevised tables and figures; Dr Johnson provided methodologicaladvice, reviewed and revised surveys, reviewed and revisedtables and figures, and revised the manuscript; and DrGuttmann provided methodological advice, reviewed and revisedsurveys, facilitated expert panel meeting, providedmethodological advice, interpreted data, reviewed and revisedtables and figures, and revised the manuscript.

www.pediatrics.org/cgi/doi/10.1542/peds.2013-0854

doi:10.1542/peds.2013-0854

Accepted for publication Jul 30, 2013

Address correspondence to Antonia S. Stang, MDCM, MBA, MSc,Alberta Children’s Hospital, 2888 Shaganappi Trail, Calgary AB,T3B 6A8. E-mail: antonia.stang@albertahealthservices.ca

(Continued on last page)

752 STANG et al by guest on April 10, 2020www.aappublications.org/newsDownloaded from

Assessing thequalityofhealthcare isaninternational priority.1–4 Research hasrevealed that performance measurementimproves health care outcomes.5,6 Ac-cording to the commonly referencedDonabedian framework, quality indicatorsare explicitly defined and measurableitems referring to the structures (staff,equipment, and facilities), processes(prescribing, investigations, interactionsbetween professionals and patients), oroutcomes (mortality, morbidity, or patientsatisfaction) of care.4,7,8 Quality indicatorshave been developed for a number ofhealth care settings, including emergencydepartments (EDs).9–13

However, despite the fact that childrenarefrequentusersofemergencycare,14

there is a lack of research on indicatorsspecific to the pediatric population. Forexample, ,5% of children are affectedby the 3 conditions most frequentlyaddressed in adult outcomes re-search (diabetes, heart disease, andarthritis).15 Similarly, quality measuresthat are part of pediatric emergencypractice have not been systematicallydeveloped or validated.15 Performancemeasures specific to pediatrics and pe-diatric emergency medicine have beenidentified as a research priority.16–18

Evidence indicates that there is sub-stantial practice variation for pediatricpatients among emergency care pro-viders, and that many providers do notoptimally manage seriously injured orill children.19–21 Most of the recent workon practice variation and lack of adher-ence to practice guidelines in the pedi-atric ED setting has been done oncommon, often lower acuity condi-tions,21–26 despite evidence of a similargap between knowledge and practice inseverely ill and injured children.20 Iden-tifying gaps in care for high acuity con-ditions, where improvement is likely tohave the largest impact on quality of lifeand longevity,19 requires valid and reli-able quality indicators. The objective ofthis project was to use a systematic

process involving multiple stakeholdersto review existing indicators and de-velop new indicators for high acuity pe-diatric conditions relevant to any EDsetting where children are seen.

METHODS

Weused a systematic,multiphase, Rand-modified Delphi method congruent withthe process for quality indicator de-velopment as outlined by the Agencyfor Healthcare Research and Quality(AHRQ)27 and used in previous indicatorwork.4,7,10,11,13,28 Ethics approval for thisstudy was obtained by the ConjointHealth Research Ethics Board of theUniversity of Calgary.

Phase 1: Selection of TargetConditions

We convened a 32-member advisorypanel to select target conditions. Thepanel included representatives fromstakeholder organizations, emergencymedicine clinicians, administrators, anddecision-makers from the UnitedStatesandCanada.Panelmemberswereidentified by contacting stakeholderorganizations and the ED directors of allpediatric EDs, and a sample of rural andgeneral EDs across Canada, and askingthem to provide names of individualswith expertise in pediatrics, ED care, orquality improvement.Weanalyzedhealthadministrativedata from2006 to2008onthe main diagnosis for high acuity pe-diatric patients, defined as patients age0 to 19 years old prioritized as re-suscitation and emergent by using theCanadian Triage and Acuity Scale29

(Supplemental Information 3). We usedthese data on the most frequent maindiagnoses seen in all EDs in Ontario andAlberta, representing 50% of Canada’spopulation, to construct an initial list ofpotential conditions. We provided pan-elists with frequency data on the initiallist and invited them to suggest addi-tional conditions. In an e-mail survey,panelists were asked to use a scale

from 1 (strongly disagree) to 9 (stronglyagree)27,30 based on National QualityForum (NQF) measure evaluation crite-ria31 to rate the final list of conditions onthe following: importance (potential formorbidity or mortality associated withthe condition), impact (potential to ad-dress gap between current and bestpractice), and validity (adequacy of sci-entific evidence linking performance ofcare to patient outcome). The surveywas tested for face validity before dis-semination and e-mail reminders weresent at weeks 2 and 3 to optimize re-sponse. It was decided a priori thatconditions withmean scores$7 acrossall 3 criteria by $70% of respondentswould be retained.

Phase 2: Indicator Identificationand Development

We conducted a systematic review of theliterature to identify existing indicatorsand high quality national and inter-national guidelines, systematic reviews(SRs), and randomized controlled trialsthat could be used to generate newindicators for the selected conditions.The search strategy (Supplemental Ta-ble 5) was developed by a medical re-search librarian in consultation withthe research team. We searched thefollowing bibliographic databases:PubMed, Cumulative Index to Nursingand Allied Health Literature, Embase, theCochrane Collaboration and Evidence-Based Emergency Medicine, Databaseof Abstracts of Reviews of Effects forSRs, the National Guideline Clearing-house, the Canadian Medical Associa-tion InfoBase, the National ElectronicLibrary for Health, Turning Researchinto Practice, and Best Bets from 1980to September 2010. Targeted handsearches of relevant journals and con-ference proceedings (Supplemental Ta-ble 6) were conducted for a 3-yearperiod (2007–2010). Due to resourceconstraints, only articles in Englishwere included. Guidelines and existing

SPECIAL ARTICLE

PEDIATRICS Volume 132, Number 4, October 2013 753 by guest on April 10, 2020www.aappublications.org/newsDownloaded from

indicators were also identified bysearching specialty society Web sites,internationally recognized guidelinessuch as Pediatric Advanced Life Supportand Advanced Trauma Life Support, andWeb sites that focus on quality andperformance improvement (Supple-mental Table 7).

Two research teammembers (Dr StangandMs Crotts) independently screenedall titles, abstracts, and guidelines. Thereviewers included for full text reviewany articles or guidelines that eitherreviewer thought might provide exist-ing indicators for the target conditionsor relevant clinical recommendationsthat could be used to guide indicatordevelopment. Two reviewers (Dr Stangand Ms Crotts) then independentlyreviewedall full textarticlesandselectedfor final inclusion any articles thatreported on quality indicators for theidentified conditions. The Appraisal ofGuidelines Research and Evaluation in-strumentwasapplied independentlyby2reviewers (Dr Stang and Ms Crotts) toassess the quality of the guidelines.32 Wedeveloped new indicators from highquality national or international guide-lines. We defined high quality as rec-ommended or strongly recommendedby both reviewers by using the Appraisalof Guidelines Research and Evaluationinstrument. Criteria for indicator de-velopment included the following: (1) thestrength of the recommendation withonly strong recommendations consid-ered33; (2) the consistency of the rec-ommendation between guidelines; and(3) the strength of the evidence linkingED structure or care process to patientoutcome.8 We developed new indicatorsbased on consensus between 2 re-searchers (Dr Stang and Ms Crotts),and review by remaining authors (DrsGuttmann, Straus, and Johnson). Tworeviewers (Dr Stang and Ms Crotts)independently assessed the quality ofthe evidence upon which each indicatoris based by using the Grading of

Recommendations Assessment, Develop-ment and Evaluation (GRADE)34 systemwith 1 = very low quality (expert opinion),2 = low quality, 3 = moderate quality, and4 = high quality.34 We used SRs and ran-domized controlled trials identified in theliterature search to grade the strength ofthe evidence supporting a link betweenthe performance of care specified by theindicator and patient outcomes.

Phase 3: Indicator Selection

We convened an expert panel of 14individuals, consisting of general andpediatric emergencymedicine clinicians,a nurse manager, a pediatric intensivist,quality improvement and safety re-searchers, and ED administrators (Sup-plemental Information 2). The panelistswere selected based on recommenda-tions by members of the advisory paneland represented the full spectrum of EDsettings where children are seen.

We used a modified Delphi techniqueconsisting of 2 rounds of anonymousquestionnaires and a face-to-face meet-ing of the expert panel to generate a finallist of indicators. Before completingthe first questionnaire, panelists weree-mailed a description of the goals of theresearch project, the full list of existingandnewlydevelopedindicatorsclassifiedaccording to Donabedian’s framework(structure, process, and outcome),8 andthe grading of the quality of evidence. Thefirst e-mailed questionnaire, sent in De-cember 2010, was pilot tested for facevalidity, and e-mail reminders were sentat weeks 3 and 4 to optimize response.Panelists were asked to rate the identi-fied indicators on the criteria of (1) rel-evance to the care of high acuitypediatric patients seen in any ED settingand (2) the degree to which measure-ment of the indicator would impact thequality of care provided. Panelists wereasked to rate each indicator on bothcriteria by using a Likert scale from 1(strongly disagree) to 9 (stronglyagree).27,30 We used a predetermined

decision rule that any indicators rated#3 on both criteria by all panelistswould be discarded from further con-sideration.

Theexpert panelmet in-person inJanuary2011aftercompletionof thefirst survey. Atthe meeting, the panelists reviewed ano-nymized ratings for each indicator. Pan-elists were also provided with theirindividualratings fromthefirstsurveyandgiventheopportunitytosuggestadditionalindicators. At the end of the meeting,panelists were asked to independently re-rate each indicator by using the samecriteria of relevance and impact and pri-oritize the top 5 indicators for each con-dition considered the most important tomeasure to improve quality of care andpatient outcomes. Based on previous in-dicator development work28 and con-sensus by the expert panel, we used apredetermined decision rule that indica-tors rated$7 (moderately agree) acrossboth criteria by$70% of panelists wouldbe included in the final list. Transcripts ofthe meeting were documented.

Phase 4: Feasibility of IndicatorMeasurement

Data from January 2009 to December2010 on the selected indicators werecollected retrospectively from a tertiarycare, pediatric ED with 65 000 annualvisits. The goal of this phase was todetermine the feasibility, described bythe NQF as the extent to which the dataare readily available, retrievable with-out undue burden, and can be imple-mented for performance measurement,and reliability of data collection.31 Basedon the NQF description and previouspediatric indicator development work,35

we defined feasibility as measures thatcould be generated by using existingdata sources including chart review,physician order entry, and ED patienttracking systems. Performance mea-surement in a pediatric ED also pro-vided an initial estimate of practicevariation and provider compliance with

754 STANG et al by guest on April 10, 2020www.aappublications.org/newsDownloaded from

care processes or structures specifiedby the indicators. We created a standardprofile of measure specifications, themethods by which the target populationis identified and the data actually col-lected.36 This included the InternationalClassification of Diseases, 10th Revision(ICD-10) diagnostic codes and inclusioncriteria for each condition, and thespecific data elements, such as numer-ator, denominator, and exclusions, foreach indicator (Supplemental Infor-mation 3). Data abstractors (3 experi-enced chart reviewers) used astandardized database (Access 2010)that was piloted for accuracy and clarityon a sample of 10 patient visits for eachcondition. Interrater reliability was cal-culated for a random sample (10%) ofcharts by using intraclass correlation(ICC), Cohen’s unweighted k, or pro-portion agreement depending on dataelements abstracted.37–40

RESULTS

Figure 1 summarizes the process ofindicator selection.

Phase 1: Selection of TargetConditions

Ninety-one percent (32/35) of invitedadvisory panel members agreed toparticipate and identified 13 potentialconditions. The number of high acuitypediatric ED visits for these conditionsover a 2-year period ranged from 24 formeningitis to 1458 for burns (Table 1).Eighty-one percent (26/32) of the advi-sory panel members completed thecondition prioritization survey, and 6conditions had a mean score of$7 by$70% of respondents (Table 1).

Phase 2: Indicator Identificationand Development

Table 2 reveals the results of the searchof the literature and quality improve-ment Web sites. We identified 46 existingindicators for the 6 targeted conditions.We derived 51 new indicators from rec-

ommendations contained in national andinternational guidelines. The interraterreliability for determining the level ofevidence upon which each indicator wasbased was acceptable (k. 0.6)39 for allconditions except status epilepticus (k =0.15). The overall interrater reliability forthe GRADE rating was k = 0.68.

Phase 3: Indicator Selection

We presented 97 indicators to the ex-pert panel for initial rating, and none ofthe indicators were discarded after thefirst survey. The expert panel suggestedan additional 17 indicators for discus-sion at the face-to-face meeting. Theexpert panel selected 62 quality indi-cators. In addition to the indicators foreach condition, the panel selected 2general measures relevant for allhigh acuity pediatric patients (Table 3).The majority of the indicators reflectED processes (84%, n = 52), withfew indicators reflecting structures(11%, n = 7) or outcomes (5%, n = 3).8

Thirty-seven percent (n = 23) of theindicators selected are based on moder-ate or high quality evidence.

Phase 4: Feasibility of IndicatorMeasurement

Table 4 reveals the age and proportionof patients who met the inclusion cri-teria for each of the conditions. A total of1681 unique visits were identified basedon age, acuity, and ICD-10 codes. Theproportion of patients meeting the in-clusion criteria for each conditionranged from 22% for severe head injuryto 84% for anaphylaxis. The interraterreliability for determiningwhich patientsmet the inclusion criteria was accept-able (k. 0.6)39 for all conditions exceptfor severe sepsis (k = 0.23).

Results of indicator measurement ina pediatric ED are shown in Table 3. Forthe indicators reflecting timeliness ofcare, ED arrival (first time recorded)was used as time zero based on con-sensus of the expert panel. For diabetic

FIGURE 1Process of indicator selection.

SPECIAL ARTICLE

PEDIATRICS Volume 132, Number 4, October 2013 755 by guest on April 10, 2020www.aappublications.org/newsDownloaded from

ketoacidosis (DKA), data were acces-sible from chart review and a physicianorder entry system for all of the appli-cable indicators with high interraterreliability. Compliance with the pro-cesses of care specified by the indica-tors was good with minimal practicevariation identified. For example, nopatients received bicarbonate (n = 62),and the expert panel agreed that thisnumber should be low (ie, ,1%). Themajority of patients received potassiumreplacement (91%, n = 65) and weretreated with the appropriate insulindose and route (88%, n = 59). Requireddata were available for the statusasthmaticus indicators, but interraterreliability was more variable. The ma-jority of patients received a systemiccorticosteroid during the visit (99%,

n = 180), and b2-agonists and systemicsteroids were provided in a median of19 and 27 minutes, respectively. Re-liability was lower for indicators thatrelied on information written in thechart by a clinician (compared withdata from physician order entry or pa-tient tracking systems), such as the“percentage of admitted patients withobjective assessment of severity of theircondition” (49%, k = 0.08) and “patientsreferred to an asthma education pro-gram” (32%, k = 0.48). From the cur-rently available data sources, it is notpossible to determine if the perfor-mance on these indicators was low dueto practice variation and poor providercompliance or lack of documentation.For anaphylaxis, 68% of patients re-ceived epinephrine in the ED, and 94% of

patients who received epinephrine inthe ED were treated by the appropriateroute. Interrater reliability was good forthe anaphylaxis indicators.

For the status epilepticus indicators,the median time to second line anti-convulsant administration was 31minutes (ICC = 0.89), with 87% (k = 1.0)of patients receiving a benzodiazepineas initial therapy and 86% (k = 1.0) ofpatients with rapid bedside glucosedocumented. Interrater reliability wasnot calculated for “attainment of sei-zure control within 30 minutes” and“receipt of an antiepileptic within 10minutes” due to the small number ofcases available for reliability compar-ison.38 Four of the severe head injuryindicators were specific to referring(nontrauma) centers (Table 3) andwere not applicable to the centerwhere data were collected. Data for 2of the indicators “head computed to-mography (CT) scan performed andanalyzed within 1 hour of request” and“neurosurgeon response time .30minutes” were not available by usingexisting data sources. Compliance washigh with respect to documentation ofcentral nervous system (CNS), bloodpressure, and oxygen saturation mon-itoring. However, 9.5% (n = 21) ofpatients were not intubated beforeleaving the ED, and only 46% (n = 13) ofintubated patients had documentedend tidal CO2 monitoring. Interraterreliability was high for the head injury

TABLE 1 Volume of Pediatric Patients (Age 0–19 Years) With Identified Conditions Seen in Albertaand Ontario in 2006–2008 and Results of Advisory Panel Condition Selection

Condition Numberof Visits

Ratea (Per Thousand#Age 19)

% Panelists With MeanScore for Condition$7

Anaphylaxis 1334 0.34 92Cardiac arrest 170 0.04 54DKA 1138 0.29 100Meningitis 24 0.01 65Neutropenia 406 0.10 65Septic shock 240 0.06 85Severe head injury 941 0.24 92Sickle cell anemia crisis 770 0.19 50Status asthmaticus 489 0.12 96Status epilepticus 439 0.11 88Supraventriculartachycardia

545 0.14 58

Burnb 1458 0.37 62Tylenol ingestionb 1390 0.35 58a Denominator = population of Alberta and Ontario # age 19 = 3 967 400.b Data from Ontario only.

TABLE 2 Results of the Systematic Review of the Literature and Indicator Selection Process

Condition ExistingIndicators

New GuidelineBased

Indicators

Additional IndicatorsSuggested byExpert Panel

Total IndicatorsConsidered byExpert Panel

FinalIndicatorsSelected

InterraterReliability (k)a forGRADEb Rating

DKA 6 4 6 16 10 0.74Status asthmaticus 16 11 1 28 16 0.73Anaphylaxis 1 6 2 9 3 1.00Status epilepticus 2 13 2 17 6 0.15Severe head injury 14 7 4 25 16 0.66Sepsis/septic shock 2 9 2 13 9 0.55General measuresc 5 1 0 6 2 1.00Total 46 51 17 114 62 0.68a k(k) = Cohen’s unweighted k.b Grade = level of evidence upon which indicator is based (1 = very low quality [expert opinion], 2 = low quality, 3 = moderate quality, 4 = high quality).34c Measures identified in literature search that were not condition specific.

756 STANG et al by guest on April 10, 2020www.aappublications.org/newsDownloaded from

TABLE 3 Final List of Indicators Selected by Expert Panel and Results of Indicator Measurement at a Pediatric ED

Quality Indicators (First 5 Listedfor Each Category Were Rated

Highest Priority by Expert Panel)

Type ofIndicator

Gradea Directionb Results of Measurement,% (Denominator) orMedian Min (IQR)

Interrater Reliability,k,c ICC, or %Agreement (n)

DKA% of EDs with clinical guidelines for the treatment of DKA in

childrenS 1 ↑ NA (single site) NA

% of patients receiving IV fluids within 60 min of ED arrival P 1 ↑ 36% (n = 47) k = 1.00 (n = 9)Time from triage to initiation of IV therapy P 1 NA 78 min (69) ICC = 1.00 (n = 11)% of patients treated with appropriate insulin dose and route P 4 ↑ 88% (n = 59) k = 1.00 (n = 11)Time from ED arrival to insulin administration P 1 NA 115 min (88) ICC = 1.00 (n = 11)% of patients treated initially with IV normal saline P 2 ↑ 72% (n = 47) k = 1.00 (n = 11)% of patients who receive potassium replacement P 4 ↑ 91% (n = 65) k = 1.00 (n = 11)% of patients who receive bicarbonate P 3 ↓ 0% (n = 65) Agree = 1.0 (n = 9)% of patients administered IV insulin P 4 ↑ 88% (n = 65) k = 1.00 (n = 11)Time from arrival to expert consultation P 1 NA 161 min (71) ICC = 1.00 (n = 11)

Status asthmaticus/severe asthmaTime from arrival to systemic steroid administered P 4 NA 27 min (23) ICC = 0.88 (n = 68)% of EDswith clinical guidelines for the treatment of asthma in

childrenS 2 ↑ NA (single site) NA

Time from arrival to first inhaled b2-agonist treatment P 4 NA 19 min (16) ICC = 0.90 (n = 69)% of admitted patients with objective assessment of severity

on initial presentationP 1 ↑ 49% (n = 94) k = 0.08 (n = 14)

Unplanned return visit within 72 h for same/related asthmaexacerbation

O 1 ↓ 5.4% (n = 95) Agree = 0.93 (n = 68)

% of patients with asthma who received respiratoryassessment score

P 3 ↑ 98% (n = 190) k = 1.0 (n = 69)

% of patients receiving systemic corticosteroid during visit P 4 ↑ 99% (n = 180) k = 0.87 (n = 68)% of patients who received ipratropium bromide in ED P 4 ↑ 99% (n = 180) k = 0.80 (n = 68)% of patients receiving combination b2-agonist/

anticholinergic therapyP 4 ↑ 98% (n = 180) k = 0.22 (n = 68)

Time from arrival to expert consult for patients not improvingafter conventional treatment

P 1 NA 253 min (197) ICC = 0.66 (n = 13)

% of patients not improving who receive at least 1 second-linetherapy

P 3 ↑ 65% (n = 49) k = 0.38 (n = 68)

% of patients discharged from the hospital from ED witha prescription/supply of oral steroids

P 4 ↑ 92% (n = 95) k = 0.62 (n = 53)

% of patients discharged from the hospital with a controllermedication

P 4 ↑ 76% (n = 95) k = 0.50 (n = 53)

% of discharged patients referred to an asthma educationprogram

P 2 ↑ 32% (n = 95) k = 0.48 (n = 53)

% of patients discharged from the hospital from the ED witha written action plan

P 3 ↑ 49% (n = 95) k = 0.83 (n = 53)

% of patients discharged with follow-up instructions P 3 ↑ 87% (n = 95) k = 0.51 (n = 53)Anaphylaxis% of EDs with clinical guidelines for the treatment of

anaphylaxis in childrenS 2 ↑ NA (single site) NA

% of patients with anaphylaxis who received epinephrine in ED P 3 ↑ 68% (n = 211) k = 0.89 (n = 19)% of patients treated with epinephrine in ED treated by the

appropriate routeP 3 ↑ 94% (n = 82) k = 0.68 (n = 19)

Status epilepticus% of EDs with clinical guidelines for the treatment of status

epilepticus in childrenS 1 ↑ NA (single site) NA

Time from arrival to administration of second-lineanticonvulsant

P 2 NA 31 min (44) ICC = 0.89 (n = 7)

Failure to achieve seizure control within 30 min of ED arrival O 2 ↓ 54% (n = 59) NCd (n = 4)% of patients receiving antiepileptic drug within 10 min of ED

arrivalP 1 ↑ 58% (n = 59) NCd (n = 4)

% of patients in whom initial drug therapy in ED includeda benzodiazepine

P 3 ↑ 87% (n = 76) k = 1.00 (n = 12)

% of patients with rapid bedside glucose documented P 2 ↑ 86% (n = 85) k = 1.00 (n = 12)

SPECIAL ARTICLE

PEDIATRICS Volume 132, Number 4, October 2013 757 by guest on April 10, 2020www.aappublications.org/newsDownloaded from

indicators with the exception of “hourlyCNS monitoring” (k = 0.41) and “CTwithin 1 hour of arrival” (Agreement= 0.44). For severe sepsis/septic shock,the median time from ED arrival to

isotonic fluid bolus was 68 minutes, 63minutes to intravenous (IV)/intraosseous(IO) insertion, and 189 minutes to anti-biotic administration. Sample size wasalso small, and interrater reliability not

calculated, for the severe sepsis indi-cators that measured fluid refractoryshock (n = 7), dopamine resistant shock(n = 3), and patients treated withpressors who had not received 60 cc/kg

TABLE 3 Continued

Quality Indicators (First 5 Listedfor Each Category Were Rated

Highest Priority by Expert Panel)

Type ofIndicator

Gradea Directionb Results of Measurement,% (Denominator) orMedian Min (IQR)

Interrater Reliability,k,c ICC, or %Agreement (n)

Severe head injury% of EDs with clinical guidelines for the treatment of severe

head injury in childrenS 2 ↑ NA (single site) NA

Time from arrival to definitive airway management/rapidsequence intubation

P 1 NA 20 min (19) 0.99 (n = 6)

% of patients with documented, untreated hypotension P 3 ↓ 5% (n = 21) k = 0.77 (n = 9)Time from ED arrival to definitive care/trauma centere P 1 NA NA NALocal guidelines for referral of patients to definitive care/

trauma centereS 1 ↑ NA NA

Time from ED arrival to initiation of transfer to definitive care/trauma centere

P 1 ↓ NA NA

Time from ED arrival to arrival of specialized transporte P 1 ↓ NA NA% of patients with hourly CNS monitoring/charting P 1 ↑ 90% (n = 21) k = 0.41 (n = 21)% of patients without sequential blood pressure monitoring P 3 ↓ 0% (n = 21) k = 1.00 (n = 21)% of patients without sequential oxygen saturation and

respiratory rate monitoringP 2 ↓ 0% (n = 21) k = 1.00 (n = 21)

% of patients who are not intubated before leaving ED (ie, foradmission, imaging)

P 1 ↓ 9.5% (n = 21) k = 0.63 (n = 13)

% of patients with CT scan analyzed within 1 h of request P 1 ↑ NCf NCf

% of patients who do not have a CT scan performed within 1 hof ED arrival

P 1 ↓ 24% (n = 21) Agree = 0.44 (n = 9)

Neurosurgeon response time .30 min from time of call-inrequest

P 1 ↓ NCf NCf

% of patients without spinal cord immobilization P 2 ↓ 14% (n = 21) k = 1.00 (n = 21)% of intubated patients with end tidal CO2 monitoring P 1 ↑ 46% (n = 13) k = 1.00 (n = 9)

Severe sepsis/septic shockTime from ED arrival to isotonic fluid bolus P 3 NA 68 min (104) ICC = 0.99 (n = 12)% of EDs with clinical guidelines for the treatment of severe

sepsis/septic shock in childrenS 1 ↑ NA (single site) NA

Time from ED arrival to antibiotic administration in suspectedsevere sepsis/septic shock

P 1 NA 189 min (212) ICC = 1.0 (n = 12)

% of patients treated with appropriate volume of isotonic fluidin first hour

P 3 ↑ 57% (n = 28) k = 0.85 (n = 16)

% of patients with fluid refractory shock treated withappropriate vasopressor/inotrope

P 2 ↑ 43% (n = 7) NCd (n = 3)

% of patients with dopamine resistant shock treated withepinephrine/norepinephrine

P 2 ↑ 0% (n = 3) NCd (n = 3)

% of patients treated with pressors who have not received 60cc/kg fluid

P 2 ↓ 25% (n = 8) NCd (n = 3)

Time from ED arrival to IV/IO insertion P 3 NA 63 min (78) ICC = 0.98 (n = 9)Glucose measured in first hour of resuscitation P 1 ↑ 47% (n = 30) Agree = 0.59 (n = 17)

General measuresTime from ED arrival to first provider for all urgent and

emergency patientsP 1 NA 34 min (32) NCf

Return visits within 48 h resulting in admission for all urgentand emergency patients

O 1 ↓ 1.3% (n = 12 636) NCf

IQR, interquartile range; NA, results not reported as denominator requires multiple sites or indicator refers only to referral centers; O, outcome; P, process; S, structure.a Grade denotes level of evidence upon which indicator is based (1 = very low quality [expert opinion], 2 = low quality, 3 = moderate quality, 4 = high quality).34b Direction denotes directionality of proportion-based indicators with a higher (↑) or lower (↓) number indicative of better performance.c k(k) = Cohen’s unweighted k.d NC denotes reliability not calculated due to sample size (,5 observations).e This indicator was specific to referring (nontrauma) centers and was not applicable to the center where data were collected.f NC denotes data not available from existing data sources.

758 STANG et al by guest on April 10, 2020www.aappublications.org/newsDownloaded from

of fluid (n = 8). The median time to firstprovider for all resuscitation and em-ergent patients was 34 minutes. In ad-dition, 1.3% of 12 636 resuscitation andemergent patients discharged from thehospital returned within 48 hours andwere admitted. Data on time to providerand return visits were from an admin-istrative database, and interreliabilitydata could not be assessed.

DISCUSSION

This rigorous process provides 62 evi-dence and expert consensus basedquality indicators for high acuity con-ditions relevant for any ED setting wherechildren are seen. Previous work onindicators for pediatric ED patients hasfocused on administrative and clinicalmeasures, such as length of stay in the EDafteradmission, that are not presentationor condition specific41,42; common con-ditions seen in any ED setting28; and cre-ation of a balanced scorecard to reflectall facets of pediatric emergency care.43

None of the previous work targets highacuity conditions. A recently publishedanalysis of existing pediatric measuresrelevant to emergency care revealed thatmost disease specific measures addressa few common pediatric conditions andsuggested that future measures shouldconsider illness severity.44

The 4 phases of this study followed theprocess for indicator development andassessment as outlined by the AHRQ.27

These phases included the following:expert engagement of an advisory

panel to identify conditions for in-dicator development, identification ofcandidate indicators including litera-ture review and summary of evidence(using GRADE), expert panel review andselection of indicators by using a mod-ified Delphi process, and assessmentof feasibility of candidate indicatorsincluding empirical analyses.

One of the strengths of this project wasthe comprehensive search for existingindicatorsandhighqualityguidelines forthe development of new indicators, andthe systematic application of GRADE inassessing the level of evidence uponwhich each indicator is based. Althoughthe GRADE systemwas a useful means ofsummarizing the evidence for the expertpanel, we identified a number of chal-lengeswith itsuse, including theneed forsignificant time,resources,andresearchmethodology expertise. Even for raterswith clinical and research backgrounds(physician with masters level epidemi-ology training and an experienced re-search nurse), the interrater reliabilityfor GRADE assignment was variable. Notsurprisingly, the k was lower for con-ditions with a less developed evidencebase, such as status epilepticus, ascompared with asthma or DKA.

Given the large number of indicatorsconsidered and the variable quality ofevidence available, the opportunity forthe expert panel to discuss the indi-cators in personwas an integral part ofthe indicator selection process. Pre-vious work has also emphasized the

importance of a face-to-face meeting ofthe expert panel.28 Another similaritywith previous work on pediatric qualitymeasurement was that the majority ofindicators selected by the expert panelreflected ED processes.28,44 The onlystructural indicators retained by theexpert panel assessed the presence ofclinical guidelines for each of the con-ditions, despite a paucity of evidencelinking guidelines to patient outcome.These findings illustrate the need forfurther work developing outcomeindicators and establishing links be-tween structure and process indica-tors and patient outcome.

A final strength of the project was theinclusion of a data collection phase toassess the feasibility and reliability ofindicator measurement. Previous workon quality measures for the pediatricpopulation has emphasized the im-portanceof testingmeasures in therealworld settings where care will be as-sessed.45 Themeasurement stage of thisstudy highlighted a number of issuesthat are relevant to the interpretationand application of the indicators. Forexample, a challenge identified in thedata collection phase was the difficultyassigning time zero for complex con-ditions such as severe sepsis, DKA, andstatus epilepticus. We decided a priori touse ED arrival as time zero but recognizethat this may not be accurate as a septicchild may decompensate while in the EDand not have met the criteria for severesepsis at presentation. Similarly, a child

TABLE 4 Demographic Data and Proportion of Patients Who Met Inclusion Criteria for High Acuity Conditions

Condition Age, Median (Range), y Charts Reviewed, n Meet Criteria, % (n)a Reliability, k (n)b

DKA 11.7 (1.3–17.9) 78 83% (n = 65) 1.0 (n = 11)Severe asthma/status asthmaticus 5.25 (1.75–18) 1058 18% (n = 190) 1.0 (n = 69)Anaphylaxis 6.9 (0.3–17.9) 250 84% (n = 210) 1.0 (n = 23)Status epilepticus 3.75 (0.5–18) 127 67% (n = 85) 0.67 (n = 12)Severe head injury 6.4 (0–16.9) 94 22% (n = 21) 0.83 (n = 13)Severe sepsis/septic shock 2.2 (0.3–17.2) 74 41% (n = 30) 0.23 (n = 17)

The definition of DKA was based on biochemical criteria51; severe asthma/status asthmaticus: score of$9 on a validated clinical pediatric asthma score52; anaphylaxis criteria were based onthe National Institute of Allergy and Infectious Disease and Food Allergy and Anaphylaxis Network consensus definition for anaphylaxis53; status epilepticus based on$1 of the following: (1)a 30-min duration of seizures; (2) 2 or more sequential seizures without full recovery of consciousness between seizures; or (3) ongoing seizure for.10 min in the ED54; severe head injury:Glasgow Coma Scale $ 955; severe sepsis/septic shock criteria were based on the international consensus definitions for sepsis and organ dysfunction in pediatrics.46a Inclusion criteria outlined in Supplemental Information 3.b k = Cohen’s unweighted k for the number of patients who met inclusion criteria for each condition.

SPECIAL ARTICLE

PEDIATRICS Volume 132, Number 4, October 2013 759 by guest on April 10, 2020www.aappublications.org/newsDownloaded from

may start seizing while in the ED suchthat the time from ED arrival to firstanticonvulsant treatment may not ac-curately describe the timeliness of sei-zure treatment. Difficulty in assigninga time zero may account in part for therelatively long length of time from EDarrival to isotonic fluid bolus; IV/IO in-sertion; and antibiotic administration(Table 3) for patients with severe sepsis.These results highlight the need for datacollection across multiple centers toestablish reasonable benchmarks forthese indicators, especially for con-ditions such as severe sepsis whereidentifying the denominator is a chal-lenge, as illustrated by our poor inter-rater reliability (k = 0.23) in applying anoperational definition based on an in-ternational consensus definition.46

Another challenge we encountered wasthesmall samplesize foranumberof theindicators. The combination of low eventrates and small numbers of eligiblepatients is a recognized issue in per-formance measurement,47 particularlyin pediatrics.48 The conventional mini-mum sample size is $30 eligiblepatients.48 Our experience collecting 2years of feasibility data suggests thateven tertiary care pediatric centersmaynot be able to accrue sufficient num-bers to adequately measure perfor-mance for conditions such as severehead injury or severe sepsis (Table 4).

A number of methods have been sug-gested to address the issue of smallsample size in indicator reporting. Oneapproach is to only report on institu-tionswith adequate numbers of eligiblepatients ($30).48 However, applyingmeasures only to institutions with

a particular volume of high acuity caseswould miss a significant portion ofpatients who are seen in smaller cen-ters, and it is in these centers wherepractice variation and the potential forimprovement may be greatest.21,24–26,49

Many of the indicators developed herewould be useful even for smaller volumecenters for local quality improvementinitiatives, such as measuring the im-pact of a new clinical pathway. Anothersolution already in use by the US De-partment of Health and Human Servicesis to aggregate data over 3 years.48 Inaddition to changing the time frame ofreporting, other methods that could beused to adapt certain indicators forpublic reporting and accountability forsmaller volume centers include the fol-lowing: changing reporting conventionsto reflect uncertainty when it exists48;using a composite measure of multipleoutcomes48; application of statisticalmethods such as indirect estimationand hierarchical modeling47; and utiliz-ing selected measures on a regionalrather than an institutional level.

In addition to the challenges identifiedabove, the results of this study aresubject to a few limitations. First, de-spite our efforts to incorporate the bestpossible evidence, less than half of thefinal indicators selected by the expertpanel are based on moderate or highquality evidence. Unfortunately, this islikely a reflection of gaps in the overallquantity and quality of child health re-search50 and reinforces the need forfurther high quality research in pedi-atrics. Second, information on thevolume of patients seen and the avail-ability of data may not be generalizable

to other pediatric institutions and iscertainly not applicable to smaller,nonpediatric hospitals. However, themeasure specifications and reliabilitydata should be applicable to mostsettings. A final limitation is the de-pendence on written documentationfor some of the indicators. In general,interrater reliability and compliancewith the process or structure specifiedby the indicatorwas lower for indicatorsthat required clinician documentation,as comparedwith indicatorswhere datawere available fromapatient tracking orphysician order entry system. This limi-tation needs to be taken into account infuture applications of the indicators,either at the data collection phasethrough the use of alternate data suchas physician billing, electronic healthrecords, and pharmacy data or in thebenchmarking/reporting phase.

CONCLUSIONS

This evidence and expert consensusbased process provides indicators forhigh acuity pediatric conditions poten-tially suitable for a range of applicationsfromlocalquality improvement initiativesto public reporting. The results of thisstudy contribute significantly to theexisting body of quality indicators for theemergency care of pediatric patients.Future work will focus on multicenterbenchmarking and data collection to testthe validity and feasibility of these indi-catorsacross thespectrumofEDsettingsthat provide care for children. This re-search provides clinicians, researchers,and policy makers with tools to improvethequalityofpediatriccare forseverely illchildren seen in any ED setting.

REFERENCES

1. Institute of Medicine. Statement onQuality of Care: National Roundtableon Health Care Quality–The UrgentNeed to Improve Health Care Quality.

Washington, DC: National Academies Press;1998

2. Institute of Medicine. Crossing the QualityChasm: A New Health System for the 21st

Century. Washington, DC: National Acade-mies Press; 2001

3. Beal AC, Co JP, Dougherty D, et al.Quality measures for children’s health

760 STANG et al by guest on April 10, 2020www.aappublications.org/newsDownloaded from

care. Pediatrics. 2004;113(1 pt 2):199–209

4. Campbell SM, Cantrill JA, Roberts D. Pre-scribing indicators for UK general practice:Delphi consultation study. BMJ. 2000;321(7258):425–428

5. Lee TH. Eulogy for a quality measure. N EnglJ Med. 2007;357(12):1175–1177

6. Bradley EH, Holmboe ES, Mattera JA, RoumanisSA, Radford MJ, Krumholz HM. A qualita-tive study of increasing beta-blocker useafter myocardial infarction: why do somehospitals succeed? JAMA. 2001;285(20):2604–2611

7. Campbell SM, Braspenning J, Hutchinson A,Marshall MN. Research methods used indeveloping and applying quality indicatorsin primary care. BMJ. 2003;326(7393):816–819

8. Donabedian A. The quality of care. How canit be assessed? JAMA. 1988;260(12):1743–1748

9. Graff L, Stevens C, Spaite D, Foody J. Mea-suring and improving quality in emergencymedicine. Acad Emerg Med. 2002;9(11):1091–1107

10. Lindsay P, Schull M, Bronskill S, Anderson G.The development of indicators to measurethe quality of clinical care in emergencydepartments following a modified-delphiapproach. Acad Emerg Med. 2002;9(11):1131–1139

11. Mourad SM, Hermens RP, Nelen WL, BraatDD, Grol RP, Kremer JA. Guideline-baseddevelopment of quality indicators for sub-fertility care. Hum Reprod. 2007;22(10):2665–2672

12. McGory ML, Shekelle PG, Ko CY. De-velopment of quality indicators for patientsundergoing colorectal cancer surgery. JNatl Cancer Inst. 2006;98(22):1623–1633

13. Guru V, Anderson GM, Fremes SE, O’ConnorGT, Grover FL, Tu JV; Canadian CABG Sur-gery Quality Indicator Consensus Panel. Theidentification and development of Canadiancoronary artery bypass graft surgeryquality indicators. J Thorac CardiovascSurg. 2005;130(5):1257

14. National Center for Health Statistics.Health, United States, 2012: With SpecialFeature on Emergency Care. Hyattsville,MD: National Center for Health Statistics;2013

15. Bordley WC. Outcomes research andemergency medical services for children:domains, challenges, and opportunities.Ambul Pediatr. 2002;2(suppl 4):306–310

16. Clancy CM, Dougherty D, Walker E. The im-portance of outcomes research in pediat-ric emergency medicine. Ambul Pediatr.2002;2(suppl 4):293–300

17. Moody-Williams JD, Krug S, O’Connor R,Shook JE, Athey JL, Holleran RS. Practiceguidelines and performance measures inemergency medical services for children.Ann Emerg Med. 2002;39(4):404–412

18. Dougherty D, Simpson LA. Measuring thequality of children’s health care: a pre-requisite to action. Pediatrics. 2004;113(1pt 2):185–198

19. Institute of Medicine, Committee on theFuture of Emergency Care in the UnitedStates Health System. Emergency Care forChildren: Growing Pains. Washington, DC:National Academies Press; 2007

20. Zebrack M, Dandoy C, Hansen K, Scaife E,Mann NC, Bratton SL. Early resuscitation ofchildren with moderate-to-severe trau-matic brain injury. Pediatrics. 2009;124(1):56–64

21. Hampers LC, Trainor JL, Listernick R, et al.Setting-based practice variation in themanagement of simple febrile seizure.Acad Emerg Med. 2000;7(1):21–27

22. Knapp JF, Simon SD, Sharma V. Quality ofcare for common pediatric respiratory ill-nesses in United States emergency depart-ments: analysis of 2005 National HospitalAmbulatory Medical Care Survey Data.Pediatrics. 2008;122(6):1165–1170

23. Schweich PJ, Smith KM, Dowd MD, WalkleyEI. Pediatric emergency medicine practicepatterns: a comparison of pediatric andgeneral emergency physicians. PediatrEmerg Care. 1998;14(2):89–94

24. Petrack EM, Christopher NC, Kriwinsky J.Pain management in the emergency de-partment: patterns of analgesic utilization.Pediatrics. 1997;99(5):711–714

25. Plint AC, Johnson DW, Wiebe N, et al. Practicevariation among pediatric emergency de-partments in the treatment of bronchiolitis.Acad Emerg Med. 2004;11(4):353–360

26. Isaacman DJ, Kaminer K, Veligeti H, JonesM, Davis P, Mason JD. Comparative practicepatterns of emergency medicine physiciansand pediatric emergency medicine physi-cians managing fever in young children.Pediatrics. 2001;108(2):354–358

27. Fabian LA, Geppert J. Quality IndicatorMeasure Development, Implementation,Maintenance, and Retirement Summary(Prepared by Battelle, under Contract No.290-04-0020). Rockville, MD: Agency forHealthcare Research and Quality; May 2011

28. Guttmann A, Razzaq A, Lindsay P, ZagorskiB, Anderson GM. Development of measuresof the quality of emergency departmentcare for children using a structured panelprocess. Pediatrics. 2006;118(1):114–123

29. Canadian Paediatric Triage and Acuity Scale:implementation guidelines for emergency

departments. Canadian Journal of Emer-gency Medicine. 2001;3(4)

30. Fitch K, Bernstein SJ, Aguilar MD, et al. TheRAND/UCLA Appropriateness Method User’sManual. Santa Monica CA; RAND, SantaMonica CA: 2001

31. National Quality Forum. Measure evaluationcriteria. December 2009. Available at: www.qualityforum.org/docs/measure_evaluation_criteria.aspx. Accessed August 13, 2013

32. Appraisal of Guidelines for Research andEvaluation (AGREE) Instrument. Available at:www.agreetrust.org/wp-content/uploads/2013/06/AGREE_II_Users_Manual_and_23-item_Instrument_ENGLISH.pdf. AccessedAugust 22, 2013

33. Guyatt GH, Oxman AD, Kunz R, et al; GRADEWorking Group. Going from evidence torecommendations. BMJ. 2008;336(7652):1049–1051

34. Atkins D, Best D, Briss PA, et al; GRADEWorking Group. Grading quality of evidenceand strength of recommendations. BMJ.2004;328(7454):1490

35. Nkoy FL, Fassl BA, Simon TD, et al. Quality ofcare for children hospitalized with asthma.Pediatrics. 2008;122(5):1055–1063

36. Rubin HR, Pronovost P, Diette GB. Froma process of care to a measure: the de-velopment and testing of a quality in-dicator. Int J Qual Health Care. 2001;13(6):489–496

37. Reeves MJ, Mullard AJ, Wehner S. Inter-raterreliability of data elements from a prototypeof the Paul Coverdell National Acute StrokeRegistry. BMC Neurol. 2008;8:19

38. Walter SD, Eliasziw M, Donner A. Samplesize and optimal designs for reliabilitystudies. Stat Med. 1998;17(1):101–110

39. Worster A, Haines T. Advanced statistics:understanding medical record review(MRR) studies. Acad Emerg Med. 2004;11(2):187–192

40. Cohen J. A coefficient of agreement fornominal scales. Educ Psychol Meas. 1960;20(1):37–46

41. Hung GR, Chalut D. A consensus-establishedset of important indicators of pediatricemergency department performance. PediatrEmerg Care. 2008;24(1):9–15

42. Shaw KN, Ruddy RM, Gorelick MH. Pediatricemergency department directors’ bench-marking survey: fiscal year 2001. PediatrEmerg Care. 2003;19(3):143–147

43. Alessandrini E, Alpern E, Varadarajan K,et al. Developing quality performancemeasures for pediatric emergency care(Abstract). E-PAS. 2009;2725.1

44. Alessandrini E, Varadarajan K, Alpern ER,et al; Pediatric Emergency Care Applied

SPECIAL ARTICLE

PEDIATRICS Volume 132, Number 4, October 2013 761 by guest on April 10, 2020www.aappublications.org/newsDownloaded from

Research Network. Emergency departmentquality: an analysis of existing pediatric mea-sures. Acad Emerg Med. 2011;18(5):519–526

45. Lannon C, Peterson LE, Goudie A. Qualitymeasures for the care of children withotitis media with effusion. Pediatrics. 2011;127(6). Available at: www.pediatrics.org/cgi/content/full/127/6/e1490

46. Goldstein B, Giroir B, Randolph A; In-ternational Consensus Conference on Pe-diatric Sepsis. International pediatricsepsis consensus conference: definitionsfor sepsis and organ dysfunction in pedi-atrics. Pediatr Crit Care Med. 2005;6(1):2–8

47. Zaslavsky AM. Statistical issues in report-ing quality data: small samples and case-mix variation. Int J Qual Health Care. 2001;13(6):481–488

48. Bardach NS, Chien AT, Dudley RA. Smallnumbers limit the use of the inpatient pe-diatric quality indicators for hospital com-parison. Acad Pediatr. 2010;10(4):266–273

49. Grol R. Successes and failures in theimplementation of evidence-based guide-lines for clinical practice. Med Care. 2001;39(8 suppl 2):II46–II54

50. Zylke JW, Rivara FP, Bauchner H. Challengesto excellence in child health research: callfor papers. JAMA. 2012;308(10):1040–1041

51. Dunger DB, Sperling MA, Acerini CL, et al;ESPE; LWPES. ESPE/LWPES consensus state-ment on diabetic ketoacidosis in childrenand adolescents. Arch Dis Child. 2004;89(2):188–194

52. Chalut DS, Ducharme FM, Davis GM.The Preschool Respiratory Assessment

Measure (PRAM): a responsive index ofacute asthma severity. J Pediatr. 2000;137(6):762–768

53. Sampson HA, Muñoz-Furlong A, CampbellRL, et al. Second symposium on the defi-nition and management of anaphylaxis:summary report—Second National In-stitute of Allergy and Infectious Disease/Food Allergy and Anaphylaxis Networksymposium. J Allergy Clin Immunol. 2006;117(2):391–397

54. Epilepsy Foundation. Prolonged or serialseizures (status epilepticus). Available at:www.epilepsyfoundation.org/about/types/types/statusepilepticus.cfm. Accessed Au-gust 13, 2013

55. Ghajar J. Traumatic brain injury. Lancet.2000;356(9233):923–929

(Continued from first page)

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).

Copyright © 2013 by the American Academy of Pediatrics

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

FUNDING: All stages of this work were funded by the Canadian Institute for Health Research (CIHR) MOP-102676.

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

762 STANG et al by guest on April 10, 2020www.aappublications.org/newsDownloaded from

DOI: 10.1542/peds.2013-0854 originally published online September 23, 2013; 2013;132;752Pediatrics 

GuttmannAntonia S. Stang, Sharon E. Straus, Jennifer Crotts, David W. Johnson and Astrid

Quality Indicators for High Acuity Pediatric Conditions

ServicesUpdated Information &

http://pediatrics.aappublications.org/content/132/4/752including high resolution figures, can be found at:

Referenceshttp://pediatrics.aappublications.org/content/132/4/752#BIBLThis article cites 44 articles, 13 of which you can access for free at:

Subspecialty Collections

subhttp://www.aappublications.org/cgi/collection/quality_improvement_Quality Improvemente_management_subhttp://www.aappublications.org/cgi/collection/administration:practicAdministration/Practice Managementsubhttp://www.aappublications.org/cgi/collection/emergency_medicine_Emergency Medicinefollowing collection(s): This article, along with others on similar topics, appears in the

Permissions & Licensing

http://www.aappublications.org/site/misc/Permissions.xhtmlin its entirety can be found online at: Information about reproducing this article in parts (figures, tables) or

Reprintshttp://www.aappublications.org/site/misc/reprints.xhtmlInformation about ordering reprints can be found online:

by guest on April 10, 2020www.aappublications.org/newsDownloaded from

DOI: 10.1542/peds.2013-0854 originally published online September 23, 2013; 2013;132;752Pediatrics 

GuttmannAntonia S. Stang, Sharon E. Straus, Jennifer Crotts, David W. Johnson and Astrid

Quality Indicators for High Acuity Pediatric Conditions

http://pediatrics.aappublications.org/content/132/4/752located on the World Wide Web at:

The online version of this article, along with updated information and services, is

http://pediatrics.aappublications.org/content/suppl/2013/09/18/peds.2013-0854.DCSupplementalData Supplement at:

ISSN: 1073-0397. 60007. Copyright © 2013 by the American Academy of Pediatrics. All rights reserved. Print the American Academy of Pediatrics, 141 Northwest Point Boulevard, Elk Grove Village, Illinois,has been published continuously since 1948. Pediatrics is owned, published, and trademarked by Pediatrics is the official journal of the American Academy of Pediatrics. A monthly publication, it

by guest on April 10, 2020www.aappublications.org/newsDownloaded from