Evaluation of a Statewide Prevention Program: An Illustration of a Research-Practitioner Model

Marcia Hughes, Ph.D.

Tim Black, Ph.D.

Center for Social Research

University of Hartford

Purpose

• Show usefulness and importance of bringing evaluation research and practice together in what we refer to as a research-practitioner model.

• Using the stages of research and program development of a statewide home visitation model in Connecticut, will illustrate how well-designed research has been used to inform and refine program practices.

Overview of the talk

• Why use a research-practitioner model?

• Connecticut’s Home Visitation Program: chronology & evolution

• Highlights of findings from national research

• Highlights of research-practitioner stages of research and program development

• NFN standing in comparison with national models/research

• Conclusions

Why Use a Research-Practitioner Model?

• Research-practitioner model: Conduct meaningful evaluation work that informs program practices, strengthens organizational framework, and drives stages of program development.

• Increasing awareness of overall gap between research and practice: different cultural practices, different rules of evidence, distinctive languages.

• Evaluation Research that is communicated to and used by practitioners and policymakers.

• Open-ended, complex problems vs. closed-systems problems.

Connecticut’s Home Visitation Program: chronology & evolution

• The charge of Children’s Trust Fund: Prevention

• Adopted a National model: Healthy Families America

• From 2 sites in 1995 to 42 sites in every geographic region

• Urban focus: Hartford (2005) & New Haven (2007) “go to scale”

• Statewide infrastructure: All 29 birthing hospitals, ob-gyn clinics

• Healthy Families CT (1995) Healthy Families Initiative (2000) Nurturing Families Network (2003)

Goals and Services of Home Visiting

• Primary goal: prevent child abuse and neglect; ensure child safety and well being

• Ensure child health and wellness

• Improve parenting attitude and behavior and improve relationship between mother and child

• Improvement in mother’s developmental trajectory and life course: education, employment, and self-sufficiency

• Provides high-risk families intensive parent education and support in the home and also helps to link families with needed resources and assistance for up to five years.

Nurturing Connections

1,172

2,047

3,702 3,754

5,472

3,151

3,4783,416

0

1,000

2,000

3,000

4,000

5,000

6,000

1999 2000 2001 2002 2003 2004 2005 2006

Number of first time mothers screened each year

Nurturing Home Visiting Participation

407509

600 637 639 668 689859

1201

0200400600800

100012001400

1998 1999 2000 2001 2002 2003 2004 2005 2006

Families Starting Families Active During the Year

Program Participation by Year Since 1998

A statewide infrastructure

2007: New Haven goes from 2 to 8 program sites

2005: Harford goes from 2 to 10 program sites

Program sites are located in every geographic area of CT and in all 29 hospitals, pre-natal and ob-gyn clinics.

Is NFN home visitation making a difference?

• Research over the past eleven years overall have yielded positive results.

• A good job of identifying & recruiting a high-risk population

• A good job of reducing child physical abuse

• An excellent job of linking families to services in community

• Mothers who remain in the program for one or two years often achieve educational and employment goals

• Making improvements in parenting capacities, attitudes & behaviors.

Interpretation of Pre-Post Evaluation Design

Home Visiting at the National Level: Research (RCTs) show mixed results

• Program implementation and quality assurance (Duggan et al., 2007; Duggan et al., 2004)

• Programs need an explicit “change theory” (DeLarosa et al., 2005)

• Change theory is not comprehensive enough. (Hebbeler et al., 2002)

• Who delivers the service (Olds et al., 2002)

• Who is being targeted (Dumont et al., 2008)

• Staff training & supervision (Gomby, 2007; Halperm 1993; Hiatt et al., 1997)

• Families w/ multiple problems (Chaffin, 2004; Duggan et al., Gomby, 2007

Interpretations of the research

• Research design in evaluating programs (Olds, 2007)

National Research on program impact on child abuse and neglect (Gomby, 2005)

• Population being served (low SES, often young, first-time mothers with low coping skills at program entry)

•Onset of services (prenatal, birth, families with children under 3 years of age)

• Case management services and curriculum that targets specific issues and outcomes and includes parent-child activities

• Intensity of services (consistently and frequently engage mothers and families)

• Performance monitoring to ensure fidelity to the model

• Staff training & supervision to address risk factors

Research-Practitioner Model Center for Social Research & Children’s Trust Fund

• Pre-post design and analysis of outcome data

• Cultural Broker Model (1998-2000)

• Study Circles (2001)

• Life Stories of Vulnerable Families in Connecticut (2002-2003)

• “Reflections on a program” (1996)

• Process evaluations: interviews, focus groups, surveys & ethnographic field work

• Expanded analysis of child abuse & neglect reports (2004 and ongoing)

• Urban Focus: neighborhood analysis (Hartford in 2005 and New Haven in 2007)

• Continuous Quality Improvement (2002 and ongoing)

• Research team Analysis of NFN in comparison with National models

• Clinical trial of In-home Cognitive Behavioral Therapy (3-year study starting in 2009)

Bridging Research and Practitioners: researchers, practitioners, policymakers

• Research Team Meetings

• CQI Team

• Multiple Research Methods

• Roles of project manager and data base managers:

• Provide technical assistance to program staff

• emphasize relationships with providers

• Quarterly reports and annual site visits

Process Evaluation of Home Visitation (Black & Markson, 2001)

• Ethnographic analyses of program practices; developed & articulated theoretical rationale for a paraprofessional model.

• Following needs were identified:

• Protocols for conducting home visits

• Organizational channels for home visitors to communicate perspectives on family needs and to participate in statewide decision-making

• Strategies for addressing specific challenges to delivering home visiting services (3 malleable risk factors)

• Strategies for effective supervision

Study Circles (Diehl, 2001)

• Participatory Action Study: promote dialogue among practitioners; discover common voices, and propose solutions to practice and policy concerns identified in the process evaluation.

• Outcome of study circles:

• Changes and additions to staff training.

• Changes to parenting education curricula.

• Changes in organizational structure of sites (solidifying the role of the clinical supervisor

• Development of the Continuous Quality Improvement (CQI) team with broad representation that includes staff at all levels.

Cultural Broker Model (Black & Markson, 2001)

• Interaction between home visitor & family member is the heart of the program:

• Success is often dependent on personal and cultural experience with the population she is serving

• Two important patterns to identification.

• Professionalism is embraced by the home visitors

• Home visitors reject professionalism as a viable role.

Quality of clinical supervision

• Role of clinical supervisor is also articulated and included as central component to paraprofessional model:

• Managing feelings and reactions to families • Frequency and scheduling of supervision• Joint home visits• Group supervision • Professional development/education

• Interaction between home visitor & clinical supervisor is equally important

• In order for home visitor to buy into the professional model, clinical supervisor has to be willing to learn from the home visitor’s experience and knowledge of the families and communities

Continuous Quality Improvement Team (2002 and ongoing)

• Creative Outreach Policy• Family Engagement• Looking for the Little Steps• Clinical Supervision• Length and Content of Visits• Working with Significant Others• Families with Cognitive Delays• Families with Multiple Children Policy• Families with Acute Problems Policy • Professional Development Policy

Major Policy that have been developed

Life Stories of Vulnerable Families in Connecticut (Black, Erdmans, & Dickinson, 2004)

• Roles of the Home Visitor: Mothers described their program experiences and perspectives.They articulated what they valued most about their relationships with their home visitors.

• Typologies of families receiving services: Analysis of 171 interviews with program participants conducted in 2002 and 2003 identified patterns of vulnerability among mothers in Connecticut who were identified as at risk for poor parenting.

Connecticut’s Vulnerable FamiliesN=171

12%

23%

33% 32%

0%

5%

10%

15%

20%

25%

30%

35%

Patterns of Vulnerability

CognitivelyImpairedMothers

Young-YoungMothers

Mothersliving incrises

Mothers inLess Distress

Roles of the Home Visitor

Life story interviews with NFN mothers in 2002 and 2003 provided an opportunity for mothers to describe their program experiences and perspectives. They articulated what they valued most about their relationships with their home visitors.  

• Home Visitor as “Baby Expert”

• Home Visitor as Advocate

• Home Visitor as Friend

• Home Visitor as “fictive kin”

Annualized Rates of Abuse and Neglect

1.6%3.0%

6.4%6.1%

2.2%1.8%0.0%1.0%2.0%3.0%4.0%5.0%6.0%7.0%

7/1/00-6/30/01

7/1/01-6/30/02

7/1/02-6/30/03

7/1/03-6/30/04

7/1/04-6/30/05

7/1/05-6/30/06

Annualized Rates of Child Maltreatment in the NFN Program

Expanded analysis of child abuse & neglect reports (2004 and ongoing)

• Analyzed variation in rates between program sites (Damboise & Black, 2004)

• Compared maltreatment rates with rates reported in studies of similar high-risk mothers not receiving home visitation (Murphy et al., 1985; Stevens-Simones et al., 2001)

• Compared rates with rates documented by research on similar home visitation programs across the country (range from 1% -8%)

• Analysis of reports: issues surrounding domestic violence, substance use are prevalent among perpetrators, as well as high prevalence of a parent with mental illness, and cognitive delay

NFN Compared at National level

• Program implementation and quality assurance

• Programs need an explicit “change theory”

• Problem is the change theory itself

• Who is delivering services

• Who is being targeted

• Staff training and supervision

• Home visiting is not addressing the Big 3

√√

√√

√

√√

• Research design ?

Conclusions and some of the challenges

• Changes and new initiatives often addressed/mirrored critical areas identified in national research.

• Now ready to do more rigorous evaluation: RCT

• Research-practitioner model lead to increased public awareness, funding and support for program expansion

• Program change and development largely resulted from carefully designed research.

• Challenges: limits due to state system; don’t always get cooperation from program staff; learn to be accepting of new language; requires program leaders be tolerant at same time are vulnerable