Upload
donovan-ruddy
View
235
Download
0
Tags:
Embed Size (px)
Citation preview
The Development of Quality Indicators for High Acuity Pediatric Conditions: Challenges in the Translation of
Knowledge into Performance Measurement
Antonia Stang MDCM MBA MSc
Assistant Professor
University of Calgary
Departments of Pediatrics and Community Health Sciences
Disclosure
• I do not have an affiliation (financial or otherwise) with any commercial organization that may have a direct or indirect connection to the content of my presentation.
“If we want healthy citizens-as opposed to citizens who have ready access to sickness care-we need a profound philosophical shift in what we should expect from medical professionals. We need to reward and incent quality, not quantity”
Andre Picard, Globe and Mail, March 20, 2012
Background
Quality of Care: “the degree to which health services for individuals increases the likelihood of desired health outcomes and are consistent with current professional knowledge”
(Institute of Medicine)
Background
• Quality Indicators: explicitly defined and measurable items pertaining to the structures, processes or outcomes of care– Structures: staff, equipment, physical layout of the
department, laboratory and diagnostic imaging resources
– Processes: interactions between professionals and patients
– Outcomes: mortality, morbidity, patient satisfaction, quality of life
Quality Indicator Uses
• Improve health care and outcomes• Benchmark performance• Set minimum standards of care• Improve efficiency• Accountability• Transparency• Research• Pay-for-Performance
What Makes a Good Measure?
• Impact, Opportunity, Evidence—Important to Measure and Report
• Reliability and Validity—Scientific Acceptability of Measure Properties
• Usability• Feasibility
National Quality Forum Measure Evaluation Criteria http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx
Objectives
• To review methods for involving stakeholders in the indicator development process.
• To discuss the application of GRADE (the Grading of Recommendations Assessment, Development and Evaluation) in indicator development and selection.
• To describe the challenges in developing and testing broadly applicable performance measures for high impact, relatively low frequency, conditions.
Study Objective
• to use a systematic process involving multiple stakeholders to develop evidence based quality of care indicators for pediatric conditions requiring high acuity ED care.
Research Team
• Antonia Stang MD MBA MSC, Principal Investigator• Astrid Guttmann MD MSc, Co-Investigator• David Johnson MD, Co-Investigator• Sharon Straus MD MSc, Co-Investigator• Lisa Hartling MSc PhD, Collaborator/methodological
expert• Francois Belanger MD, Collaborator/decision maker• Angelo Mikrogianakis MD, Collaborator/decision maker • Jen Crotts RN, Research Assistant• Janie Williamson RN, Pediatric Emergency Research
Team Coordinator
Funding
• Funded by an operating grant from the Canadian Institutes of Health Research (CIHR)
Rationale
• Lack of research on quality indicators specific to the pediatric population.
• Quality measures that are part of pediatric emergency medicine practice have not been systematically validated.
• Performance measures specific to pediatrics and pediatric emergency medicine have been identified as a research priority.
Phase 4: Data Collection
Phase 3: Expert Panel Process
17 new indicators 114 considered 62 selectedPhase 2: Systematic Review
47 existing indicators 51 guideline and evidence based
Phase 1: Condition Selection
6 conditions selected for indicator development
Stakeholder Involvement
• Goals:–to represent different stakeholder
perspectives
–to incorporate scientific evidence and expert opinion
Stakeholder Involvement
• Systematic methods to combine expert opinion and medical evidence– Consensus development conferences– Guideline based– Delphi technique– Nominal group technique– RAND/UCLA appropriateness method
Phase 1 Condition Selection
• 32 Member advisory panel
• Data on the main diagnosis for high acuity (resuscitation and emergent at triage) pediatric patients (age 0-19 yrs) seen in all EDs in Ontario and Alberta.
• Criteria for Condition Selection; – importance (morbidity or mortality) – impact (potential to address gap between current and
best practice)– validity (adequacy of scientific evidence linking
performance of care to patient outcome)
Phase 1: Condition Selection
Table 1: High Acuity (Resuscitation and Emergent ) Pediatric visits in 2006/2007 and 2007/2008 for all EDs in Ontario and Alberta for Selected Conditions
Condition Number of ED VisitsDiabetic ketoacidosis 1138Status asthmaticus 489Anaphylaxis 1334Status epilepticus 439Severe head injury 941Sepsis/septic shock 240
Phase 2: Indicator Development
• Systematic Review of the Literature for each condition– Existing Indicators– High quality national and international guidelines
(AGREE), Systematic Reviews (AMSTAR), Randomized Controlled Trials
• Criteria for Indicator Development– High quality evidence linking care structure or process
to patient outcome– Strongly recommended– Consistency across guidelines
Literature Review
Condition Search Results Full Text Reviewed
Articles Guidelines
Diabetic ketoacidosis 2480 374 3 3
Status asthmaticus 4564 172 8 5
Anaphylaxis 5889 276 4 3
Status epilepticus 870 28 3 5
Severe head injury 4789 95 5 5
Sepsis/septic shock 3866 78 3 2
GRADE
Grading of Recommendations Assessment, Development and Evaluation
1-Very low quality: Any estimate of effect is very uncertain 2-Low quality: Further research is very likely to have an
important impact on our confidence in the estimate of effect and is likely to change the estimate
3-Moderate Quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
4-High Quality: Further research is very unlikely to change our confidence in the estimate of effect
02468
1012141618
Num
ber o
f Ind
icat
ors
Condition
GRADE Rating by Condition
1
2
3
4
GRADE
GRADE Inter-rater Reliability
*Cohen’s Kappa unweighted
Condition Kappa* AgreementDiabetic Ketoacidosis 0.74 0.83Status Asthmaticus 0.73 0.81Anaphylaxis 1.00 1.00Status Epilepticus 0.15 0.50Severe Head Injury 0.66 0.85Severe Sepsis/Septic Shock 0.55 0.73General Measures 1.00 1.00Overall 0.68 0.78
Challenges using GRADE
• Subjective• Lack of concordance between guidelines• Time consuming• Requires substantial knowledge of clinical
condition and research methodology• Variable inter-rater reliability
Phase 3: Indicator Selection
Expert Panel Process: 2 rounds of a web-based survey and a face-to-face meeting
Indicators were selected based on two criteria rated on a scale of 1(strongly agree) to 9 (strongly disagree):– Relevance – Impact
Indicators rated ≥7 on both criteria by 70% of panelists were retained
Relevance
Impact
Disagree Disagree Disagree Disagree Neutral Agree Agree Agree Agreestrongly moderately somewhat somewhat moderately strongly
1 2 3 4 5 6 7 8 9
Indicator Type Source GRADE Numerator Denominator
% of patients with anaphylaxis with documentation of specialist referral
P 25, 26, 28-31
2 Number or patients with anaphylaxis with documentation of specialist referral including primary MD follow-up for referral, or documentation of existing specialist relationship
Total number of patients with anaphylaxis (based on ICD-10 codes)
Disagree Disagree Disagree Disagree Neutral Agree Agree Agree Agreestrongly moderately somewhat somewhat moderately strongly
1 2 3 4 5 6 7 8 9
0
10
20
30
40
50
60
Structure Process Outcome
Type of Indicator
Challenges in Indicator Development
• Lack of high quality evidence on the link between treatment/processes and outcomes, particularly in the pediatric setting
• Difficulty in identifying performance measures applicable to all settings
"The only man I know who behaves sensibly is my tailor; he takes my measurements anew each time he sees me. The rest go on with their old measurements and expect me to fit them."
George Bernard Shaw
Phase 4:Data Collection
*Based on ICD-10 code
Condition Number of ED Visits*Diabetic ketoacidosis 112Status asthmaticus/severe asthma 852Anaphylaxis 269Status epilepticus 133Severe head injury 108Sepsis/septic shock 79
Diabetic Ketoacidosis
*Includes only visits with fluids or insulin started in study ED
Indicator Results GRADE Kappa% of Emergency Departments (EDs) with guidelines N/A (single site) 1
% of ED visits with:Intravenous (IV) fluids within 60 minutes of ED arrival* 36 (17/47) 1 1.00Isotonic solution as initial IV fluid* 72 (34/47) 2 1.00IV insulin given 88 (57/65) 4 1.00Appropriate initial insulin dose and route* 88 (52/59) 4 1.00Potassium replacement 91 (59/65) 4 1.00Bicarbonate given 0 (0/61) 3 N/C
Time (median minutes with interquartile range) from:Triage to initiation of IV fluids* 78 (45,114) 1Arrival to insulin 115 (60,148) 1Arrival to expert consultation 161 (130,201) 1
Anaphylaxis
* for food and insect sting induced reactions
Indicator Results GRADE Kappa% of EDs with:
Clinical guidelines for the treatment of anaphylaxis N/A (single site) 2% of patient visits with:
Epinephrine given in ED (or in 3 hours prior to ED visit) 68 (144/211) 3 0.89Epinephrine given in ED by the appropriate route 94 (77/82) 3 0.68Documentation of epinephrine auto-injector at discharge 85 (180/211) 2 0.02Documentation of discharge instructions to avoid offending allergen* 17 (29/173) 2 0.41Documentation of instruction for epinephrine self-administration 38 (81/211) 2 0.39Documentation of specialist referral 56 (119/211) 2 1.00
Feasibility and Reliability
*Number of ED visits based on ICD-10 code
Condition ED Visits* Missing Chart/Visit Meets Criteria KappaDiabetic ketoacidosis 112 34 65/78 (83%) 1.00Anaphylaxis 269 19 211/250 (84%) 1.00Status epilepticus 133 6 85/127 (67%) 0.67
Challenges in Indicator Testing
• Feasibility and Reliability– Retrospective– Proper diagnosis is in itself a quality
issue– Accuracy of ICD -10 codes – Cost/effort of data collection
• Small and variable sample size
Lessons Learned
• Allow ample time for systematic review and evidence grading
• Composition of expert panel is key• Need an experienced moderator • Formal qualitative analysis of expert panel
meeting• Include patient/care-giver perspective
Next Step
• Multicentre data collection on select high acuity indicators combined with existing pediatric and emergency department performance measures– Reliability and feasibility testing– Process to outcome link
Campbell 2003
Quality Improvement and Indicator Development
Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academy Press. 2001: Washington, DC.
Campbell, S.M., et al., Research methods used in developing and applying quality indicators in primary care. BMJ, 2003. 326(7393): p. 816-9.
Chassin MR et al. Accountability Measures-Using Measurement to Promote Quality Improvement. N Eng J Med 363:7.
Donabedian A. The quality of care. How can it be assessed? JAMA. Sep 23-30 1988;260(12):1743-1748.
http://www.qualityforum.org/docs/measure_evaluation_criteria.aspxCenter for Health Policy/Center for Primary Care and Outcomes
Research & Battelle Memorial Institute. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement (Prepared by Battelle, under Contract No. 290-04-0020). Rockville, MD: Agency for Healthcare Research and Quality. May 2011.
Pediatric and Emergency Indicators
Alessandrini E, Gorelick MH, Shaw K, Kennebeck S. Using Performance Measures to Drive Improvement in Pediatric Emergency Care 2010; http://webcast.hrsa.gov/postevents/archivedWebcastDetail.asp?aeid=534
Bardach NS, Chien AT, Dudley A. Small Numbers Limit the Use of the Inpatient Pediatric Quality Indicators for Hospital Comparison. Academic Pediatrics, 2010. 10(4).
Guttmann A, Razzaq A, Lindsay P, Zagorski B, Anderson GM. Development of measures of the quality of emergency department care for children using a structured panel process. Pediatrics, 2006. 118(1): p. 114-23.
Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, Zwarenstein M, Anderson GM. Prioritizing performance measurement for emergency department care: consensus on evidence based quality of care indicators. CJEM 2011. 13(3):300-309.
Rating the Quality of Evidence• http://www.gradeworkinggroup.org/index.htm (accessed February 23, 2012)• Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE
Working Group. Rating quality of evidence and strength of recommendations: What is "quality of evidence" and why is it important to clinicians? BMJ. 2008 May 3;336(7651):995-8.
• Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ, for the GRADE Working Group. Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-926.
• Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW Jr, Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH; GRADE Working Group. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008 May 17;336(7653):1106-10.
• Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009AGREE Next Steps Consortium (2009).
• The AGREE II Instrument [Electronic version]. Retrieved March 23 2012, from http://www.agreetrust.org .