RRFSS Evaluation: Issues and Strategies RRFSS Workshop, June 19th 2002 Catherine Bingle Sarah Feltis

RRFSS Evaluation:RRFSS Evaluation: Issues and StrategiesIssues and Strategies

RRFSS Workshop, June 19th 2002

Catherine Bingle

Sarah Feltis

RRFSS Evaluation Project 2001-2RRFSS Evaluation Project 2001-2

Project background and developmentEvaluation objectives and frameworkMethodology, tools, respondentsResultsKey issues and strategies

Evaluation Project ContributorsEvaluation Project Contributors

RRFSS Evaluation Work Group RRFSS Working Group 2001 Research assistant and project consultant PHRED Funding

BackgroundBackground Evaluation Work Group was formed May 2001 Evaluation framework developed Identified need for support PHRED funding supported hiring of Project

Consultant and Research Assistant

Evaluation ObjectivesEvaluation ObjectivesEvaluate the current state of RRFSSEvaluate the effectiveness of system

processesEvaluate the utility of systemIdentify areas for improvements and

successes of RRFSS

Expectations of the EvaluationExpectations of the Evaluation

Inform changes to RRFSS procedures

Provide a baseline/template for future RRFSS evaluations

Contribute to continuous quality improvement

Evaluation FrameworkEvaluation Framework

Developed by Evaluation Work Group Looked at CDC and WHO surveillance

system frameworks

4 KEY components• Process Issues• Collaboration Among Sites and Key Players• Utility or Usefulness• Cost Effectiveness

Data Collection ToolsData Collection Tools 1st drafts completed in October At least 4 rounds of comment and revision

completed in November after review from the Working Group

In total 5 self- administered questionnaires and one semi-structured interview were developed and used to collect data

Data Collection ToolsData Collection ToolsRRFSS Representative Questionnaire

• Focus– questionnaire development, project administration,

awareness and use of results, data analysis and quality, costs, partnerships, benefits and areas for improvement

MOH Questionnaire• Focus

– familiarity and satisfaction, perceptions of value and impact on health units, feedback on ways to sustain development

Data Collection ToolsData Collection Tools

Program Questionnaire• Focus

– program staff awareness, satisfaction with content of the questionnaire, decision making process, access to results, use of results, impact on programs, and areas for improvements

Non-participating Health Units• Focus

– familiarity, health unit interest in future participation, reasons for not participating, and perceived value of RRFSS

Data Collection ToolsData Collection ToolsISR Interview

• Focus– working relationship between ISR and health units,

questionnaire development process, costs issues and expectations, quality of data and methods, future development of RRFSS

MOHLTC Questionnaire• Focus

– familiarity, impact and value, interest in partnership with RRFSS, strategies for increasing visibility and participation

All 6 tools were disseminated December 2001 All data collected by end of January 2002 109 of 151 (72%) of targeted respondents

participated• RRFSS Representative - 11 of 12 HU’s

• MOH - 10 of 12 HU’s

• Non-participating HU’s - 17 of 24 HU’s

• Program - 69 of 101 program managers/staff

• MOHLTC - 1 of 1

• ISR interview

Data Collection and Data Collection and Response RateResponse Rate

Data AnalysisData Analysis

Quantitative data • Entered and analyzed in SPSS• Frequencies

Qualitative Data• Entered into Word files for content analysis

Process Issues Collaboration Use and Usefulness Cost Issues The Future

ResultsResults

Questionnaire development processes Satisfaction with RRFSS questionnaire Quality of data / methods Data analysis RRFSS dissemination RRFSS awareness

Process IssuesProcess Issues

RRFSS Reps (RR) - RR’s and Program Staff (PS), mostly managers, involved in processes for Q development and/or revision

PS - about 2/5 were involved RR and PS - nearly all very or somewhat satisfied with

processes RRs more satisfied with new module development than with

revision processes


Questionnaire Development Questionnaire Development ProcessesProcesses

PS concerns: – Q dev. process complexity and length– Need to involve staff more

RR concerns and suggestions:– greater clarity and documentation of policies, responsibilities, procedures, timelines,

decision rationale– greater commitment to process

ISR suggestions (in addition to some of above themes) :– procedures to assure more equality of HU access to opportunities for new questions /

changes


Questionnaire Development ProcessesQuestionnaire Development ProcessesAreas for ImprovementAreas for Improvement

RR - generally meets HU needsMOHs - split bet. very and somewhat satisfiedPS - more somewhat than very satisfied, esp.

with number of questions relevant to their programs


Satisfaction with QuestionnaireSatisfaction with Questionnaire

PS and RR - opportunities for more questions - both new topics and expansion of topics

RR - explore rotating coreRR - explore adjusting core : optional balance


Satisfaction with QuestionnaireSatisfaction with QuestionnaireAreas for ImprovementAreas for Improvement

Most RRs see sample size as adequate Those who do not indicated need for larger sample for

subpopulation analyses Most viewed response rate as good, some excellent.

Likely based on partial info 67% completion rate, June ‘01.

Question quality viewed (ISR) as good overall


Quality of Data / MethodsQuality of Data / Methods

RR and ISR - further strengthen Q quality procedures - Q review, editing, & testing.

ISR - longer Q testing - pilot study (n=100) where possible (responsiveness important too)

Assessment of existing modules - keeping flawed modules over time problematic

Need feedback mechanism for problems from data user back to Q design Need yearly conference to discuss methods


Quality of Data / MethodsQuality of Data / MethodsAreas for ImprovementAreas for Improvement

RR - dissatisfaction with timeliness of data receipt at HU

RR - All HU's had begun to analyze RRFSS dataRR - HU's have not enough skilled staff to

analyze data and make it usable



ISR, RR - analysis getting short shriftMOH, RR - analytic capacity a key sustainability challenge RR - major staff capacity differences among HU'sKey factor for RR's who were somewhat dissatisfied

overall with RRFSSKey concern of HU's not to participate



RR - most said most or all of HU staff aware of RRFSS PS - 2/3 somewhat, 1/3 very familiar with Q content MOH - most very familiar with current status of RRFSS,

most somewhat familiar w/content RR - ½ assess awareness raising processes in HU's as

very effective, ½ somewhat eff.


RRFSS Awareness in HU’sRRFSS Awareness in HU’s

RR - most HU's had disseminated results in some manner; ½ to broader community

PS - 2/3 had accessed results. Of these: Most had via their epi/analyst staff Most very satisfied w/how results provided 1/3 needed support to use results - all said it was available.

15% weren't sure if needed supports; ½ didn't need.


RRFSS Dissemination in HU’sRRFSS Dissemination in HU’s

PS - dissatisfaction w/access to results most frequently due to results not being available

PS - need for repeated dissemination PS - need better explanation of data methods


RRFSS Awareness & RRFSS Awareness & Dissemination in HU’sDissemination in HU’sAreas for ImprovementAreas for Improvement

Most RR very satisfied. Those "somewhat" concerned about cost and staff resources needed for participation

Most PS were satisfied, 42% very satisfied w/how RRFSS is implemented in their health unit


Overall Satisfaction w/RRFSS Overall Satisfaction w/RRFSS ImplementationImplementation

At non-participating HU's (epi / alt.), ½ very, ½ somewhat familiar w/RRFSS status

Most somewhat familiar w/RRFSS content, some very familiar

PHB rep - very familiar w/status, somewhat familiar w/content


RRFSS Awareness Among RRFSS Awareness Among Non-ParticipantsNon-Participants

Satisfaction with partnershipContributions to decisions, activitiesPerceived work group effectivenessWays to improve partnership

CollaborationCollaboration

MOH - most very, some somewhat satisfied with how partnership works RR - half very, half somewhat satisfied RR - most say partnership is worth investment ISR very satisfied with effectiveness of working relationship among

partners ISR - benefits of partnership:

– provides ISR w/feedback on data use

– local value of study

– collegial, knowledgeable, committed working group

Collaboration IssuesCollaboration Issues

Satisfaction with PartnershipSatisfaction with Partnership

RR - most feel all HU's have equal voice in decisions, although those involved longer seem to influence decisions more

HU's do not make equal contributions, due to differences in HU staffing, ability to support involvement, choices to be involved

Expectation that these differences will recede as more HU's gain experience


Contributions to Group Contributions to Group Decisions and ActivitiesDecisions and Activities

RR- Working G, Advisory G, and Ad Hoc WG: roughly split 1/2 very, 1/2 somewhat effective

Analysis G and Evaluation G - more somewhat than very effective

ISR perceived as very effective ISR very satisfied with working relationship among partners


Perceived Work Group Perceived Work Group EffectivenessEffectiveness

RR - some duplication between Adv. G and Work. G; difficulty of managing WG meetings

ISR - – Adv. G'd challenge representing larger group– Increasing # of HU's increases coord. difficulty– Range of expertise among HU's – Meeting structures and scheduling– Critical role of Adv. Group - stability must be ensured


Perceived Work Group EffectivenessPerceived Work Group EffectivenessChallenges; Areas for ImprovementChallenges; Areas for Improvement

RR - dedicated funding to Adv. G functions Develop MOU outlining policies, procedures, expectations, roles,

conflict resolution proc. Funding for central coordination of RRFSS Annual face to face meetings Schedule chairs and recorders for WG meets Better documentation of Adv. G and TOR


Ways to Improve PartnershipWays to Improve Partnership

Use of ResultsBenefits to Health Units from RRFSSImpact on Program Planning & EvaluationImpact of Not Having RRFSSImportance for Public Health SurveillanceNeeds Not Met By RRFSS

Use and UsefulnessUse and Usefulness

Virtually all HU's had used RRFSS dataUse highest among epi's and program staff. Managers,

media, and MOH also cited.Use highest among CDP programs. Use also frequently cited for Env. HP & VPD programs;

research & communications staff

Use IssuesUse Issues

Use of Results - RRUse of Results - RR

38% used results in their programsOf those who had not, ½ said this was due to results not

yet being availableRemainder said they had not yet had timeSome indicated data not of sufficient importance /

relevance to use


Use of Results - PSUse of Results - PS

Barriers to use:– time required for data analysis and administration– awareness among PS

Key needs:– funding for central project coordination– common syntax files – better dissemination efforts in HUs


Use of Results - RRUse of Results - RRBarriers; Areas for ImprovementBarriers; Areas for Improvement

Current, timely, local data, filling data gapsData for emerging and locally relevant issuesStaff use for planning and evaluationBetter quality than typical HU surveysStrengthened partnership among HUs and epidemiologists, and

w/community partners Increased understanding of surveillance


Key Benefits from RRFSS - RRKey Benefits from RRFSS - RR

All MOH -- RRFSS results will have impact on HU ability to meet PP&E standards

8 RR agreed, 3 "too early to say"44% of PS agreed; 26% did not; 30% not sureNon-part. HU's - 14 agreed; 3 "too early"PHB rep - agreed


Expected Impact on Program Expected Impact on Program Planning and EvaluationPlanning and Evaluation

MOH, RR, & PS identified same range of negative impacts on PH effectiveness:

Reduced access to local, timely, adaptable, continuous data Reduced ability to measure program objectives, reduced

effectiveness in PP&E, and overall effectiveness / accountability Need for additional investment in HU- and topic-specific surveys


Impact of Not Having RRFSSImpact of Not Having RRFSS

Majority (8/11) of RRFSS Reps were “very satisfied” w/RRFSS as surveillance method

Majority (8/10) of the MOH’s from participating health HU’s view RRFSS as a “very important” tool for public health surveillance

11 of 17 non-participating HU’s also view as a “very important”


Importance for Public Health Importance for Public Health SurveillanceSurveillance

Virtually all HU’s indicated the need for other community health survey work beyond RRFSS

• Child and youth health assessment• Surveying of sub-populations e.g. pregnant women• HU specific topics• Sensitive topics• Information not obtained through telephone survey


Needs not Met by RRFSSNeeds not Met by RRFSS

Key Staff Involvement in RRFSSReturn on InvestmentCost Reduction Strategies

Costs and Cost-Effectiveness Costs and Cost-Effectiveness

Time• Annual estimated key staff = 0.68 FTE per HU (range 0.16 to 2.0 FTE per HU)

WHO• Epidemiologist• Other research staff

Activities• New module dev., data analysis, and RRFSS WG meetings• Data dissemination and presentation of data to HU staff

Cost and Cost Cost and Cost EffectivenessEffectiveness

Staff Involvement in RRFSSStaff Involvement in RRFSS

RRFSS Reps - (5/11) rated as “good” and 3 rated as “excellent” (time, money, and resources)

MOH - divided between “very satisfied” and “somewhat satisfied” with their HU’s investment in RRFSS

Program Savings - most RRFSS Reps indicated “to early to say” whether RRFSS had resulted in any program saving or would create revenues for HU in future


Return on InvestmentReturn on Investment

Support for central coordination of RRFSS

Encourage partnerships with other organizations

Over time - improvements in organization and efficiency


Cost Reduction StrategiesCost Reduction Strategies

Sustainability Challenges Expanded HU and Provincial ParticipationStrategies for Increasing ParticipationSuggestions for Raising RRFSS Visibility

The FutureThe Future

• MOH Perspective• Adequate staff/resources • Rapid analysis of data• Funding/Cost• Keeping program staff involved

• RRFSS Rep’s Perspective• Cost/Funding • RRFSS administrative work• Questionnaire content


Sustainability ChallengesSustainability Challenges

• ISR Perspective

• Whether or not able to do all 38 HU

• Need signed commitment from HU earlier, and for longer term

• Working group needs support for methodology improvements and analysis


Sustainability ChallengesSustainability Challenges

• HU not currently participating in RRFSS, over half indicated an interest in participating in the future

• 6 reportedly were considering • 4 were actively preparing

• HU not interested or not considering participating most common reason given for this was COST

• lack of staff• time


Expanding HU and Provincial Expanding HU and Provincial ParticipationParticipation

• RRFSS Rep’s and MOH were asked to agree or disagree with following statements

• Important for all health units to participate• Important for MOHLTC to participate• There should be a province wide sample• Funding for RRFSS should be on 100% provincial basis



• RRFSS Reps Perspective• Agreement strongest for important for MOHLTC to participate and there should be a

province wide sample • 6 Strongly, 2 agree important for all HU to participate (3 disagree)• 2 strongly, 5 agree funding 100% provincial basis (4 disagree)

• MOH Perspective• Agreement strongest important for all HU to participate • Most agreed important for MOHLTC participate and funding 100% provincial basis• Agreement was least strong there should be province wide sample



• Ministry agrees to cost share• Provision of core questionnaire funding• Bring attention to non-participating HU• Benefits realized, participation should increase• Culture of evaluation in HU• Use of RRFSS data


Strategies for Increasing Strategies for Increasing ParticipationParticipation

• Publish/present data• Share results and information • Target dissemination • Link to academic centres• Provide key results to PHB to support internal government processes• HU provide key results to local politicians


Strategies for Raising VisibilityStrategies for Raising Visibility

ConclusionsConclusionsWhat conclusions should be drawn from the

results, and what are the Next Steps?To be determined … in part by our

discussions todayWhat strategic issues are raised?How will they be addressed?

Issues, Strategies, and Issues, Strategies, and Next StepsNext Steps

Adequate staff/resources for analysis and making data useful

Increasing use of data in program planning and evaluation Dissatisfaction with getting question accepted on

questionnaire (limited # of questions) Increasing sample size Process for revising old and developing new modules RRFSS coordinators overwhelmed by work

Issues, Strategies, and Issues, Strategies, and Next StepsNext Steps

Need for stability and support for central work of RRFSS partnership Unequal HU access to ISR, decision-making, workload and resources Upgrading RRFSS web site Sustaining funding for existing participants and new funding for

interested ones Provincial representation of data Sharing data among organizations Improved timeliness of data ISR ability to handle increase

Thank you

Catherine Bingle

[email protected]

Sarah Feltis

[email protected]

Documents

RRFSS Evaluation: Issues and Strategies RRFSS Workshop, June 19th 2002 Catherine Bingle Sarah Feltis