Upload
alexander-mitchell
View
214
Download
0
Embed Size (px)
Citation preview
VA HSR&D Center for the Study of Healthcare Provider Behavior
Maren T. Scheuner, MD, MPH, FACMGMaren T. Scheuner, MD, MPH, FACMGChief, Medical GeneticsChief, Medical Genetics
VA Greater Los Angeles Healthcare SystemVA Greater Los Angeles Healthcare SystemDirector, Clinical Genetic Services, VISN 22Director, Clinical Genetic Services, VISN 22
Dept of Medicine, David Geffen School of Medicine at UCLADept of Medicine, David Geffen School of Medicine at UCLA
[email protected]@va.gov
Evaluation of Genetic Education Evaluation of Genetic Education
Programs for Healthcare ProvidersPrograms for Healthcare Providers
VA HSR&D Center for the Study of Healthcare Provider Behavior
DisclosuresDisclosures
No conflicts of interest relating to the content of this presentation.
Funding source for content:
• CDC Office of Public Health Genomics
• VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior
VA HSR&D Center for the Study of Healthcare Provider Behavior
33
OverviewOverview
Educating healthcare providersEducating healthcare providers
Evaluating success of an education programEvaluating success of an education program
– Formative evaluationFormative evaluation
– Summative evaluationSummative evaluation
– Use of theoryUse of theory
Case studyCase study
VA HSR&D Center for the Study of Healthcare Provider Behavior
What is the goal of healthcare What is the goal of healthcare provider education?provider education?
VA HSR&D Center for the Study of Healthcare Provider Behavior
Behavior change
Improved health outcomes
What is the goal of healthcare What is the goal of healthcare provider education?provider education?
VA HSR&D Center for the Study of Healthcare Provider Behavior
Is education (didactic teaching) Is education (didactic teaching) sufficient to change healthcare sufficient to change healthcare
provider behavior?provider behavior?
VA HSR&D Center for the Study of Healthcare Provider Behavior
Generally, not.
Oxman et al., 1995; Mazmanian and Davis, 2002; Mansouri and Lockyer, 2007
Is education (didactic teaching) Is education (didactic teaching) sufficient to change healthcare sufficient to change healthcare
provider behavior?provider behavior?
VA HSR&D Center for the Study of Healthcare Provider Behavior
Informational interventions
Clinical interventions
Behavioral interventions
Components of an effective Components of an effective
education programeducation program
Continuing medical education objectives as defined by Mazmanian and Davis, 2002.
VA HSR&D Center for the Study of Healthcare Provider Behavior
Evaluation of Genetics Evaluation of Genetics Education ProgramsEducation Programs
VA HSR&D Center for the Study of Healthcare Provider Behavior
How do we measure success?
Measures of implementation success
Implementation strategies
Clinical Innovation
Process Outcomes
HealthOutcomes
Intervention
Other factors affecting progress and success
Adapted from: Luska CV, Hall C. Challenges in measuring implementation success. 3rd Annual NIH Conference on the Science of Implementation and Dissemination. Methods and Measurement. March 15-16, 2010, Bethesda, MD
VA HSR&D Center for the Study of Healthcare Provider Behavior
Implementation DefinedImplementation DefinedEfforts designed to get evidence-based practices and related products into use
Implementation typically follows dissemination and includes:– Identifying barriers, facilitators and strategies
to reduce, overcome, leverage them– Adapting the targeted practice to the context– Developing a tailored implementation strategy
VA HSR&D Center for the Study of Healthcare Provider Behavior
Implementation ResearchImplementation Research
‘the scientific study of methods to promote the uptake of research findings for the purpose of improving quality of care” McDonald et al., 2004. Toward a Theoretical Basis for Quality Improvement
Interventions in K.G. Shojania et al., Closing the Quality Gap.
“…scientific investigations that support movement of evidence-based, effective health care approaches (e.g., as embodied in guidelines) from the clinical knowledge base into routine use.” Rubenstein & Pugh, 2006.
VA HSR&D Center for the Study of Healthcare Provider Behavior
Types of EvaluationTypes of Evaluation
Formative evaluation– Rigorous assessment process designed to
identify potential and actual influences on the progress and effectiveness of implementation efforts
Summative (impact) evaluation– Systematic process of collecting and analyzing
data on impacts, outputs, products, outcomes and costs in an implementation study
Stettler CB, Legro MW, Wallace CM, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 2006;21(Suppl 2):S1-8.
VA HSR&D Center for the Study of Healthcare Provider Behavior
Summative Evaluation in Summative Evaluation in Implementation ResearchImplementation Research
Outcomes Assessment– A priori measures defined at outset of project to
assess intervention impact or effectiveness– Defined at patient-, provider-, clinic-, facility-, and/or
system-level– Involves use of administrative data, chart review,
and/or primary data collection
Cost Assessment– Assess costs associated with implementation strategy
to inform decision makers on value and feasibility of implementing the intervention
VA HSR&D Center for the Study of Healthcare Provider Behavior
Four Stages of Formative EvaluationFour Stages of Formative Evaluation
Developmental
Implementation-focused
Progress-focused
Interpretive
VA HSR&D Center for the Study of Healthcare Provider Behavior
Formative evaluation according to implementationFormative evaluation according to implementation
1. Developmental
Identify determinants of current practice
Identify barriers and facilitators
Assess feasibility of proposed intervention
Integrate findings into intervention design, and refinement prior to implementation
2. Implementation-focused
Assess discrepancies between implementation plan and execution, exploring issues of fidelity, intensity, exposure
Understand and document nature and implications of local adaptation
3. Progress-focused
Monitor impacts and indicators of progress toward project goals
Use data to inform need for modifying original strategy
Provide positive reinforcement to high performers; negative reinforcement to low performers
4. Interpretive
Assess intervention usefulness/value from stakeholders perspectives
Elicit stakeholder recommendations for further intervention refinements
Assess satisfaction with intervention and implementation process
Identify additional barriers / facilitators
Pre-Implementation Implementation Post-Implementation
VA HSR&D Center for the Study of Healthcare Provider Behavior
Developmental
Identify determinants of current practice
Identify barriers and facilitators
Assess feasibility of proposed intervention
Integrate findings into intervention design and refinement prior to implementation
Implementation-focused
Assess discrepancies between implementation plan and execution, exploring issues of fidelity, intensity, exposure
Understand and document nature and implications of local adaptation
Progress-focused
Monitor impacts and indicators of progress toward project goals
Use data to inform need for modifying original strategy
Provide positive reinforcement to high performers; negative reinforcement to low performers
Interpretive
Assess intervention usefulness/value from stakeholders perspectives
Elicit stakeholder recommendations for further intervention refinements
Assess satisfaction with intervention and implementation process
Identify additional barriers / facilitators
Pre-Implementation Implementation Post-Implementation
Formative evaluation according to implementationFormative evaluation according to implementation
VA HSR&D Center for the Study of Healthcare Provider Behavior
Need for Formative Evaluation in Need for Formative Evaluation in Implementation ResearchImplementation Research
Captures information on factors that hinder or facilitate successful implementation
Addresses interpretive weaknesses– Avoid “implementation assessment failure”– Avoid explanation and outcome attribution
failure– Enhance understanding of study outcomes
VA HSR&D Center for the Study of Healthcare Provider Behavior
Assessment Methods / Tools for Assessment Methods / Tools for Formative EvaluationFormative Evaluation
Quantitative– Structured surveys / tools– Instruments assessing organizational culture, readiness to change,
provider receptivity to evidence-based practices– Intervention fidelity measures– Audit / feedback of clinical performance data
Qualitative– Semi-structured interviews with clinical stakeholders (pre-/post-)– Focus groups– Direct observation of clinical structure and processes in site visits– Document review
Mixed methods (i.e., quantitative + qualitative)
VA HSR&D Center for the Study of Healthcare Provider Behavior
Usefulness of TheoryUsefulness of Theory
In terms of…
– Planning the implementation strategy
– Conducting evaluations
– Identifying unanticipated elements critical to successful implementation, but may be unexplained by selected theory
– Gaining additional insights about the theory
– Helping to understand findings, including relationships between domains or constructs
VA HSR&D Center for the Study of Healthcare Provider Behavior
Types of TheoriesTypes of Theories
Multiple theories often needed
Explanatory theories (aka descriptive, impact)
– Hypotheses and assumptions about how implementation activities will facilitate a desired change as well as the facilitators and barriers for success
Process theories (aka prescriptive, planned action)
– How implementation should be planned, organized and scheduled
Mixed theories– Elements of both
VA HSR&D Center for the Study of Healthcare Provider Behavior
Choosing TheoryChoosing TheoryConsider nature of the theory– Process vs. explanatory– Context (e.g., policy, organization)– Discipline (e.g., social science, psychology)
Consider level at which it will be applied– Individuals– Teams– Organization– System
Consider previous findings, experience
Consider greatest potential for adding to the knowledge-base
VA HSR&D Center for the Study of Healthcare Provider Behavior
RE-AIM Evaluation FrameworkRE-AIM Evaluation Framework
Purposes
– Broaden and standardize criteria used to evaluate clinical QI programs
– Evaluate issues relevant to program adoptions, implementation and sustainability
– Help close the gap between research studies and practice
Glasgow et al., Ann Behav Med 2004;27(1):3-12.
VA HSR&D Center for the Study of Healthcare Provider Behavior
RE-AIM to help plan, evaluate and RE-AIM to help plan, evaluate and report studiesreport studies
R Increase Reach
E Increase Effectiveness
A Increase Adoption
I Increase Implementation
M Increase Maintenance(Sustainability)
www.re-aim.org
VA HSR&D Center for the Study of Healthcare Provider Behavior
SummarySummaryGenetics education is necessary but not sufficient for change in healthcare provider behavior
Effective education programs are multi-faceted, including: informational, clinical and behavioral interventions.
Measuring success of education programs:
– Implementation success
– Process outcomes
– Health outcomes
VA HSR&D Center for the Study of Healthcare Provider Behavior
SummarySummaryFormative evaluation can be key to effective intervention design and success, interpretation and replication of results
Summative evaluation is used to assess relevant clinical outcomes and costs associated with implementation of evidence-based practices
Formative and summative evaluation are complementary in developing, implementing, evaluating and refining implementation interventions
Important to use theory in planning and evaluation of implementation research/QI, and evaluate theory
VA HSR&D Center for the Study of Healthcare Provider Behavior
CDC OPHG Translation ProgramsCDC OPHG Translation Programs
Goal: Promote evidence-based clinical and public health practice in genomics
Focus: Genetic testing applications with evidence of clinical utility (e.g., CDC EGAPP reviews, USPSTF recommendations)
Supported activities: education, policy, surveillance
VA HSR&D Center for the Study of Healthcare Provider Behavior
““Family History Education to Improve Risk Family History Education to Improve Risk
Assessment for Hereditary Cancer”Assessment for Hereditary Cancer”
Translating Clinical Guidelines for Translating Clinical Guidelines for
Family History Risk Assessment into PracticeFamily History Risk Assessment into Practice
VA HSR&D Center for the Study of Healthcare Provider Behavior
GoalGoal
To develop a multi-component
education program for primary care
clinicians that improves recognition
and referral of patients at risk for
hereditary cancer syndromes.
VA HSR&D Center for the Study of Healthcare Provider Behavior
PopulationFamilial Risk Stratification
Genetic Consultation for Hereditary Cancer
Morbidity & Mortality
Strategies to Increase Genetics Referrals
Strategies to Increase
Familial Risk Assessment
Strategies to Increase Risk-
Appropriate Interventions
Interventions
Outcomes
Logic Model
VA HSR&D Center for the Study of Healthcare Provider Behavior
Formative Evaluation - Developmental workFormative Evaluation - Developmental work
Identify determinants of current practice
Identify barriers and facilitators
Assess feasibility of proposed intervention
Integrate findings into intervention design, and refinement prior to implementation
Pre-implementation PhasePre-implementation Phase
VA HSR&D Center for the Study of Healthcare Provider Behavior
3434
Priority Setting PanelPriority Setting Panel13 Experts13 Experts
VA HSR&D Center for the Study of Healthcare Provider Behavior
3535
Availability of:Availability of:
– Genetics education for health professionalsGenetics education for health professionals
– Clinical decision support tools in the electronic Clinical decision support tools in the electronic health recordhealth record
– Prevention/management options available for Prevention/management options available for genetic conditionsgenetic conditions
– Clinical guidelinesClinical guidelines
Key Factors for Adoption of Key Factors for Adoption of Genomic Medicine at the VA Genomic Medicine at the VA
VA HSR&D Center for the Study of Healthcare Provider Behavior
3636
Highest Priorities for Health Services Highest Priorities for Health Services Research at VA in the Next 5 YearsResearch at VA in the Next 5 Years
Genetics education Genetics education
Development of clinical guidelines Development of clinical guidelines
Development of tools in CPRS for:Development of tools in CPRS for:
– Familial risk assessment Familial risk assessment
– Ordering and interpreting genetic testsOrdering and interpreting genetic tests
VA HSR&D Center for the Study of Healthcare Provider Behavior
3737
Family History Content Review in the Family History Content Review in the Electronic Health RecordElectronic Health Record
VA HSR&D Center for the Study of Healthcare Provider Behavior
Our EHR lacks standards for family Our EHR lacks standards for family history documentation history documentation
1,416 shared templates for progress notes (Aug 11, 2007 - Aug 12, 2008)– Family history mentioned in 8%
– Disease checklist most common format, 46%
– Family history open text box, 38%
– List of first-degree relatives with text box, 14%
None captured information about specific diseases in specific relatives.
VA HSR&D Center for the Study of Healthcare Provider Behavior
3939
Key Informant Interviews with Key Informant Interviews with Primary Care ProvidersPrimary Care Providers
VA HSR&D Center for the Study of Healthcare Provider Behavior
4040
To Improve Process of To Improve Process of Family History DocumentationFamily History Documentation
PCPs want:
– Template in the electronic health record
– Better organization of the family history in the electronic health record
– Patient-provided data (through kiosk or personal health record)
VA HSR&D Center for the Study of Healthcare Provider Behavior
High Ratings for Clinical Reminders
Stratify familial risk
Recognize inherited conditions
Prompt referrals for consultation or testing
Reasons for high ratings: – Lack of knowledge, familiarity and confidence in
genetic risk assessment, diagnosis and testing
VA HSR&D Center for the Study of Healthcare Provider Behavior
First draft: Family History Red First draft: Family History Red Flags for Hereditary CancerFlags for Hereditary Cancer
VA HSR&D Center for the Study of Healthcare Provider Behavior
Focus Group FeedbackFocus Group Feedback
Not useful
As primary care providers, we need to document complete family history
Once history is documented, we can recognize the red flags
Tool should have a few stem questions that can be completed quickly for most patients
VA HSR&D Center for the Study of Healthcare Provider Behavior
Baseline Knowledge & Attitudes of Enrolled Clinicians
% Correct answers
Somewhat relevant, %
Relevant, %
Very relevant, %
Basic molecular genetics, concepts, terminology
77 50 25 25
Familial/genetic risk assessment 47 4 46 50
Recognizing hereditary cancer syndromes 45 0 50 50
Genetic testing 33 63 17 20
Management of hereditary cancers, including referral
58 6 13 81
Ethical issues for patients and clinicians 67 0 37 63
VA HSR&D Center for the Study of Healthcare Provider Behavior
Implementation StrategyImplementation StrategyClinical interventions– Cancer family history reminder in EHR
– Self-administered, patient questionnaire
– Practice-feedback reports each quarter
Informational interventions:– 7-part CME lecture series on cancer genetics
– Information sheets for providers
– Information brochures for patients
– Web site
Behavior interventions– Review of family history generated by reminder with
feedback by opinion leaders
VA HSR&D Center for the Study of Healthcare Provider Behavior
Implementation-focused:Implementation-focused:
Assess discrepancies between implementation plan and execution, exploring issues of fidelity, intensity, exposure
Understand and document nature and implications of local adaptation
Implementation Phase Implementation Phase
VA HSR&D Center for the Study of Healthcare Provider Behavior
Progress-focused:Progress-focused: Monitor impacts and indicators of progress
toward project goals Use data to inform need for modifying original
strategy Provide positive reinforcement to high
performers; negative reinforcement to low performers
Implementation Phase Implementation Phase
VA HSR&D Center for the Study of Healthcare Provider Behavior
Pre/Post design:Pre-implementation Oct - Dec 2009Post-implementation Apr 2010 - Jun 2011
Abstraction of random 10% of progress notes each month. Assessed change in documentation of:Cancer family historyReferral for genetic consultation
Monthly monitoring of health factors generated by cancer family history reminder, and quarterly practice-feedback reports for enrolled clinicians
Evaluation Plan:Evaluation Plan:
VA HSR&D Center for the Study of Healthcare Provider Behavior
Completion of Cancer Family History ReminderCompletion of Cancer Family History ReminderApril 2010 - March 2011April 2010 - March 2011
3,548 patients seen with reminder due
– Avg, 413 per provider; range, 54 - 771
1,087 reminders completed when due
– Avg, 30% per provider; range, 23% - 98%
105 (10%) referred for genetic consult
– 54% of patients with a strong familial risk
– 14% of patients with a moderate familial risk
– 1% of those with a weak familial risk
VA HSR&D Center for the Study of Healthcare Provider Behavior
Cancer Family History Documented in Progress NotesCancer Family History Documented in Progress Notes
0
10
20
30
40
50
60
70
(n=76) Q1 (n=101) Q2 (n=109) Q3 (n=112)
By text By Template Previously By Template
%
Post-implementationPre-implementation
28%
50%54%
60%
VA HSR&D Center for the Study of Healthcare Provider Behavior
Quality of Cancer Family History Documentation Quality of Cancer Family History Documentation in Progress Notes with Cancer Family Historyin Progress Notes with Cancer Family History
Pre-implementation
(n=21)
Post-implementationa
(n=117)
1st degree relatives, % 76 81
2nd degree relatives, % 48 62
Lineage of relatives, % 14 62
Age of cancer onset, % 19 43
Jewish ancestry, % 0 45
VA HSR&D Center for the Study of Healthcare Provider Behavior
Trends in Genetics Referral Generated by Template Trends in Genetics Referral Generated by Template
0
5
10
15
20
25
30
Apr May Jun Jul Aug Sep Oct Nov Dec
%
134 131 130 90 78 84
On average, a genetic consult was requested for 9% of patients and 3.5% declined a genetic consult.
Missed opportunities decreased over time.
82
Consult requested Patient declined consult Familial risk but consult “not indicated”
N = 75 82
VA HSR&D Center for the Study of Healthcare Provider Behavior
Interviews with Primary Care ProvidersInterviews with Primary Care Providers
“My documentation of cancer family history has improved… I had a template I was using and it was limited to the colon, breast, uterine and ovarian cancer, so now it’s expanded because we have all those other options.”
“Now my documentation is very detailed, whereas before I would just mainly ask about mom and dad.”
It helps a lot. I had a couple of patients who never mentioned before that their grandmother died of ovarian cancer.”
VA HSR&D Center for the Study of Healthcare Provider Behavior
“I probably wasn’t doing that in-depth of a family history before, especially not focused on cancer.”
“The template is much broader and more detailed than what I probably would have gotten before. I don’t know if I would have gone down to all those relatives…, and it certainly triggered a number of consultations in some people who probably deserved it a long time ago. So I think this has greatly improved my history-taking.”
Interviews with Primary Care ProvidersInterviews with Primary Care Providers
VA HSR&D Center for the Study of Healthcare Provider Behavior
Interpretive:Interpretive: Assess intervention usefulness/value from
stakeholders perspectives
Elicit stakeholder recommendations for further intervention refinements
Assess satisfaction with intervention and implementation process
Identify additional barriers / facilitators
Post Implementation Phase Post Implementation Phase
VA HSR&D Center for the Study of Healthcare Provider Behavior
Evaluation plan:Evaluation plan:
Semi-structured interviews ongoing.
Repeat knowledge and attitudes survey.
Look at theory used
VA HSR&D Center for the Study of Healthcare Provider Behavior
ConclusionsConclusions
Our implementation strategy is working.Some interventions effective, others not so much.
Goals met:More comprehensive family history documentation
necessary for familial risk assessment. Improved recognition and referral of high-risk patients. High-risk patients utilizing genetic services.
Future studies: Summative evaluationDisseminate implementation strategyAssess utilization of preventive services, health
outcomes and costs according to use of reminder
VA HSR&D Center for the Study of Healthcare Provider Behavior
SummarySummary
When developing education program, consider determinants of provider behavior
Select theoretical model to inform development, implementation and evaluation
Conduct formative evaluation in addition to summative evaluation
VA HSR&D Center for the Study of Healthcare Provider Behavior
AcknowledgementsAcknowledgementsVA Greater Los AngelesVA Greater Los Angeles Advisory BoardAdvisory BoardElizabeth M. Yano, PhD, MSPH Deborah Duquette, MS, CGCLisa V. Rubenstein, MD, MSPH Michael Kelley, MDBrian Mittman, PhD Barbara Lerner, MS, CGCStuart Gilman, MD, MPH Douglas Olson, PhD, RNCaroline Goldzweig, MD, MPH Holly Peay, MS, CGCAlison Hamilton, PhD Ann Chou, PhDColletta Austin, RN, MSN Barbara Simon, MA CDC, OPHGMartin Lee, PhD Rodolfo Valdez, PhD, MScShannon Rhodes, PhD Shelley Reyes, PhD, MSNina Smith, MPH Mack Anders, MPAErin Schalles, MS, CGC Cecelia Bellcross, PhD,Heather ZmyewskiHeather Zmyewski MS Daurice Grossniklaus, PhDAdrie Young