Upload
ophelia-tucker
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Evaluation Plan
Steven Clauser, PhD Chief, Outcomes Research Branch
Applied Research Program
Division of Cancer Control and Population Sciences
NCCCP Launch June 25, 2007
Presentation Overview
• Evaluation Approach
• Key Evaluation Features • Evaluation Design Components
• Patient Survey
• Evaluation Timetable
NCCCP Evaluation Approach
• Formative not Summative Evaluation – What is the feasibility, sustainability and replicability of
the NCCCP model? – How does strong institutional commitment and executive
leadership support influence results?– Less emphasis on classical intervention or impact
assessment– More emphasis on measurement of improvement from
baseline assessment • Emphasizes multi-method evaluation techniques
– Qualitative assessments of organizational change – Quantitative assessment of trajectory of change
• Pre-post evaluation design– Performance assessment largely limited to intra- not inter-hospital and program comparisons
Key Evaluation Features
• Logic maps to rigorously define pilot site hospital and cancer program interventions
• Participatory research to achieve consensus on evaluation metrics and data collection approaches
• Baseline and follow-up patient surveys to assess improvement in patient program knowledge, program experience, and health-related quality of life
• Formal cost analysis of program implementation and operations • Continued feedback of performance to
NCI staff, program leadership and pilot sites
NCCCP Evaluation Design Components
• Year One – Refine Evaluation Design, Metrics, and Measures of Performance for hospital and program– Phase I – Evaluability Assessment and site visit (2-4 months) – Phase II – Participatory Research to select evaluation hypotheses/metrics – with pilot group input
(4-7 months) – Phase III – Finalize evaluation design and collect baseline data (8-12 months)
• Year Two – Interim site visit, data collection, and first progress review by evaluation contractor
• Year Three – Final site visit, data collection and second progress review
• Year Four – Final data collection and report
Phase I – “Evaluability Assessment”
• Baseline data collection on pilot program components
• “Logic mapping” of program components (including hospital and community) to evaluation questions
• Anchored by evaluation contractor 2-day site visit to each pilot program – including multi-system leadership interviews
• Assessment completed within 4 months of contractor award
Phase II – Selecting Measures and Metrics
• NCI staff prepares initial set of evaluation measures and metrics
• Evaluation contractor refines measurement set and distributes to the sites for comment
• Sites meet with evaluator and provide feedback on the appropriateness, feasibility, and availability of metrics
• Evaluation contractor submits final recommendations to NCI
• NCI makes final decision on measures and metrics within 7 months of award
Phase III – Finalize evaluation plan and collect baseline data
• Evaluation contractor completes detailed evaluation plan for NCCCP pilot program
• NCI approves evaluation plan within 9 months of contractor award
• Sites complete baseline data collection on all key NCCCP evaluation questions by end of year one
• The “intervention” period begins for all hospital and community-based projects– Progress reviews annually– Final assessment at conclusion of year three
Pilot Site Role in Supporting Patient Surveys
• Evaluation contractor develops survey content based on their expertise, review of pilot site surveys, and NCI input – NCI approves final survey instrument
• Pilot sites administer patient survey with evaluation contractor support and training – Contractor develops survey administration protocols
(mailed surveys with telephone follow-up)– Pilot Sites obtain their IRB approvals for patient survey– Contractor trains pilot site survey personnel – Pilot sites administer survey to eligible patients – Pilot sites return completed surveys to evaluation
contractor
Patient Survey Schedule
• Patient survey administered twice:– Baseline survey completed at the end of year one
• Baseline survey report prepared for NCI and pilot sites to illustrate differences in site populations
• Results may prove useful in targeting NCCCP program interventions
– Follow-up survey completed at end of year three • Final report assesses change in patient experience and
patient knowledge/use of pilot patient resources • 300 patients will complete the survey per site
and per survey round – Multi-system pilots will administer additional
surveys for each component site
Patient Survey Measurement Domains
• Outcome Metrics: – Health-related quality of life (physical, mental, and social
domains) – Overall program satisfaction
• Process Metrics – Self-perception of needed resources to assist in
personal management of cancer treatment • e.g., understanding disease, treatments, emotional and
financial assistance services – Knowledge and use of pilot site patient resources
• Including access to clinical trials– Patient Experience with care
• trust, communication, decision-making • access to care, appointments, and waiting time• Experience with multi-disciplinary team practice
Evaluation Timetable – Key Year One Events
Activity* DateProjected Contract Award Date July 20, ‘07
Initial Contractor Site Visits Oct-Nov ’07
Sites meet with Evaluation Contractor Dec’07 or Jan’08
Sites get trained on patient survey** April ‘08
Sites receive final evaluation design April ’08
All pilot baseline measurements final May ‘08
*Timetable depends on contract award date** Site survey training depends on OMB clearance requirements