Evaluation Plan Steven Clauser, PhD Chief, Outcomes Research Branch Applied Research Program...

Evaluation Plan

Steven Clauser, PhD Chief, Outcomes Research Branch

Applied Research Program

Division of Cancer Control and Population Sciences

NCCCP Launch June 25, 2007

Presentation Overview

• Evaluation Approach

• Key Evaluation Features • Evaluation Design Components

• Patient Survey

• Evaluation Timetable

NCCCP Evaluation Approach

• Formative not Summative Evaluation – What is the feasibility, sustainability and replicability of

the NCCCP model? – How does strong institutional commitment and executive

leadership support influence results?– Less emphasis on classical intervention or impact

assessment– More emphasis on measurement of improvement from

baseline assessment • Emphasizes multi-method evaluation techniques

– Qualitative assessments of organizational change – Quantitative assessment of trajectory of change

• Pre-post evaluation design– Performance assessment largely limited to intra- not inter-hospital and program comparisons

Key Evaluation Features

• Logic maps to rigorously define pilot site hospital and cancer program interventions

• Participatory research to achieve consensus on evaluation metrics and data collection approaches

• Baseline and follow-up patient surveys to assess improvement in patient program knowledge, program experience, and health-related quality of life

• Formal cost analysis of program implementation and operations • Continued feedback of performance to

NCI staff, program leadership and pilot sites

NCCCP Evaluation Design Components

• Year One – Refine Evaluation Design, Metrics, and Measures of Performance for hospital and program– Phase I – Evaluability Assessment and site visit (2-4 months) – Phase II – Participatory Research to select evaluation hypotheses/metrics – with pilot group input

(4-7 months) – Phase III – Finalize evaluation design and collect baseline data (8-12 months)

• Year Two – Interim site visit, data collection, and first progress review by evaluation contractor

• Year Three – Final site visit, data collection and second progress review

• Year Four – Final data collection and report

Phase I – “Evaluability Assessment”

• Baseline data collection on pilot program components

• “Logic mapping” of program components (including hospital and community) to evaluation questions

• Anchored by evaluation contractor 2-day site visit to each pilot program – including multi-system leadership interviews

• Assessment completed within 4 months of contractor award

Phase II – Selecting Measures and Metrics

• NCI staff prepares initial set of evaluation measures and metrics

• Evaluation contractor refines measurement set and distributes to the sites for comment

• Sites meet with evaluator and provide feedback on the appropriateness, feasibility, and availability of metrics

• Evaluation contractor submits final recommendations to NCI

• NCI makes final decision on measures and metrics within 7 months of award

Phase III – Finalize evaluation plan and collect baseline data

• Evaluation contractor completes detailed evaluation plan for NCCCP pilot program

• NCI approves evaluation plan within 9 months of contractor award

• Sites complete baseline data collection on all key NCCCP evaluation questions by end of year one

• The “intervention” period begins for all hospital and community-based projects– Progress reviews annually– Final assessment at conclusion of year three

Pilot Site Role in Supporting Patient Surveys

• Evaluation contractor develops survey content based on their expertise, review of pilot site surveys, and NCI input – NCI approves final survey instrument

• Pilot sites administer patient survey with evaluation contractor support and training – Contractor develops survey administration protocols

(mailed surveys with telephone follow-up)– Pilot Sites obtain their IRB approvals for patient survey– Contractor trains pilot site survey personnel – Pilot sites administer survey to eligible patients – Pilot sites return completed surveys to evaluation

contractor

Patient Survey Schedule

• Patient survey administered twice:– Baseline survey completed at the end of year one

• Baseline survey report prepared for NCI and pilot sites to illustrate differences in site populations

• Results may prove useful in targeting NCCCP program interventions

– Follow-up survey completed at end of year three • Final report assesses change in patient experience and

patient knowledge/use of pilot patient resources • 300 patients will complete the survey per site

and per survey round – Multi-system pilots will administer additional

surveys for each component site

Patient Survey Measurement Domains

• Outcome Metrics: – Health-related quality of life (physical, mental, and social

domains) – Overall program satisfaction

• Process Metrics – Self-perception of needed resources to assist in

personal management of cancer treatment • e.g., understanding disease, treatments, emotional and

financial assistance services – Knowledge and use of pilot site patient resources

• Including access to clinical trials– Patient Experience with care

• trust, communication, decision-making • access to care, appointments, and waiting time• Experience with multi-disciplinary team practice

Evaluation Timetable – Key Year One Events

Activity* DateProjected Contract Award Date July 20, ‘07

Initial Contractor Site Visits Oct-Nov ’07

Sites meet with Evaluation Contractor Dec’07 or Jan’08

Sites get trained on patient survey** April ‘08

Sites receive final evaluation design April ’08

All pilot baseline measurements final May ‘08

*Timetable depends on contract award date** Site survey training depends on OMB clearance requirements