Designing Survey Instrument to Evaluate Implementation of Complex Health Interventions: Lessons Learned

Eunice ChongAdrienne Alayli-GoebbelsLori Webel-EdgarJennifer RobertsonHelen Cerigo

Anne Philipneri Carly HeungQuynh HuynhHeather Manson

The Canadian Public Health Association Conference, May 28th 2014

PublicHealthOntario.ca

Why do we need to study implementation?

• Implementation is a complex process involving multiple steps at multiple levels by multiple people

• Only a small fraction of evidence-based interventions are actually implemented in the real world

• Implementation research helps:• Understand interaction with the local context• Identify factors that hinder or facilitate implementation

2

PublicHealthOntario.ca

Challenges in evaluating implementation

• There is an increasing body of evidence on factors that affect implementation, but:

• Different theories/frameworks, different terminology/definitions used • Knowledge about relationships between factors is limited • Available tools tend to focus on a limited number of factors and have

weak reliability and validity

3

How do we decide on what to measure?

PublicHealthOntario.ca

Objective of this presentation

To share lessons learned from identifying and measuring factors contributing to implementation of the enhanced Ontario Healthy Babies Healthy Children (HBHC) Program.

4

PublicHealthOntario.ca

The HBHC Program

• Aims to help children in Ontario have a healthy start in life and targets at children from the prenatal period to transition to school

• Funded by Ministry of Children and Youth Services (MCYS)• Early 2013 MCYS introduced mandatory enhancements to the HBHC

program to be implemented by 36 public health units

• Three key elements of the enhancements: • Funding of 36 screening liaison nurse positions• Introduction of a new validated screening tool • Introduction of training to use evidence-informed interventions during home-

visiting

• February to April 2013: staggered implementation process

5

PublicHealthOntario.ca

Aims of the process implementation evaluation

• To what extent has the program been implemented as planned?• Fidelity with protocol and program goals• Reach of target population for each program component

• Which factors hinder/facilitate the implementation? • Multiple program components• Multiple HBHC staff and community partners • Multiple organizations• Different local settings and population characteristics

6

→Administrative database analysis

→ Surveys & focus groups with HBHC staff

PublicHealthOntario.ca

Evaluation framework:multi-level factors contributing to

implementation outcomes

7

System

Organization (Public Health Unit)

ProviderProgram participants

Innovation

Contributing Factors

Reach

Fidelity

Local adaptations

Impact on program change goals

Implementation Outcomes

Facilitation

Feedback

Adapted from: Chaudoir et al., 2013. Measuring factors affecting implementation of health innovations: a systematicreview of structural, organizational, provider, patient, and innovation level measures. Implementation Science. 8:22

PublicHealthOntario.ca

How did the framework help in developing the survey?

• Identified several multi-level factors that have shown to contribute to implementation outcomes

• Encouraged a structured stepwise method to identify relevant constructs to measure for each multi-level factor 1. Review of the implementation science literature to identify

measurable constructs at each level

2. Search for existing instruments that measure the identified constructs

3. Expert consultation to identify additional instruments and verify the relevance of the identified constructs

8

PublicHealthOntario.ca

Stakeholder engagement

• To determine which of the identified constructs are most important to measure for the HBHC program implementation

• Brainstorm groups with HBHC program managers and directors of 36 health units (May 2013)

9

Better understanding of constructs that are most important in the language of the field AND additional factors relevant to consider

Question: Based on your knowledge and experience, what aspects of <multi-level factor> do you think are relevant/important to consider in the evaluation?

• Multi-voting exercise: to prioritize identified aspects

PublicHealthOntario.ca

Constructs examples selected for each multi-level factor

Innovation level constructs (Characteristics of the enhanced HBHC program)• Adaptability & flexibility• Consistency (additional)• Complexity• Evidence• Feasibility

Provider level constructs (Characteristics of health unit staff implementing the program)• Job stress• Job satisfaction• Knowledge• Professional confidence • Preparedness for implementation

(additional)

10

Organizational level constructs (Characteristics of the public health units in which the program is implemented)

• Organizational culture

• Communication

• Leadership

System level constructs (The external environment and broader socio-cultural context)

• Community characteristics (additional)

• Funding and staff resources

• Policies and mandates (additional)

• Involvement of external partners (additional)

PublicHealthOntario.ca

Development of specific survey items

• Survey items (open/closed-ended) were developed for all selected constructs in collaboration with program specialist

• Questions were derived from previously used/validated instruments, where possible

• Questions often adapted to the specific context of HBHC and the role-perspective (5 versions)

• For additional constructs we developed new questions using ‘field language’

• Pilot-testing with 2 staff from each role

11

PublicHealthOntario.ca

Example survey items

12

Strongly Disagree Disagree Neither Agree

nor Disagree Agree Strongly Agree

Prefer not to answer

□ □ □ □ □ □

Innovation level: “The enhanced HBHC program leaves enough room for me to make my own practice decisions” (Adaptability/Flexibility, adapted from Barriers and Facilitators Assessment Instrument )

Provider level: “I feel stressed by my new responsibilities and tasks” (Job stress)

Organizational level: “Colleagues in my health unit are willing to innovate and/or experiment to improve clinical practice ” (Organizational culture, adapted from ORCA)

System level: “My health unit has sufficient support to facilitate the implementation of the enhanced HBHC program in terms of budget” (Funding resources, adapted from ORCA)

Facilitation level: “ Change champions in my health unit support me to implement the program changes relevant to my role” (Change champions)

Feedback level: “Management and/or senior leadership in my health unit welcomes feedback from staff regarding the implementation of the new HBHC protocol” (Feedback)

Statements with the following response categories:

PublicHealthOntario.ca

Conclusions and lessons learned

• The evaluation framework was useful to inform systematic survey development for evaluating the HBHC implementation

• Integrating factors from other theories/frameworks into the evaluation framework encouraged a more comprehensive understanding of the implementation process

• The multi-level nature of the framework allowed for multiple perspectives and the use of multiple data sources

• Stakeholder engagement was essential to ask the right questions

• The survey contributed to the limited instruments available to measure factors that hinder and facilitate implementation of the multiple levels

13

PublicHealthOntario.ca

Limitations and challenges

• Our survey instrument used validated tools, but has not been validated

• The multi-level approach resulted in a long survey (feasibility depends on level of commitment of the sample)

• Due to a lack of consistency in definitions and terminology for constructs, comparisons with other studies are difficult

• Richness of data can be challenging in terms of making sense of the results

14

PublicHealthOntario.ca 15