Maryland Early Childhood Mental Health Consultation (ECMHC) Evaluation

Webinar Presentation by Deborah F. Perry Ph.D.Georgetown UniversityOctober 27, 2010

Overview of Presentation Overview of

statewide ECMHC evaluation in Maryland

Maryland’s approach Logic model and

components Levels of data

collection and other features

Future directions

Maryland ECMHC Evaluation

Funding: Maryland State Department of Education (MSDE),

Division of Early Childhood Involves 12 sites across Maryland Three year evaluation (2008-2011), with two option years Principal Investigators:

Sharon Stephan: University of Maryland Bruno Anthony, PhD & Deborah Perry, PhD:

Georgetown University Minority Business partner: CKD Communications, LLC

Early Childhood Mental Health Consultation (ECMHC)

Collaborative, capacity building services provided to early care and education (ECE) providers and family members

Can be initiated in response to concerns about an individual child’s social-emotional or behavioral development

The impact can be measured in the domains of child, parent, staff, and family

Approach to Evaluation

Collaborative Participatory Comprehensive Sensitive to respondent burden No site comparisons Technical assistance related to

evaluation and data collection provided

Funded Projects and Evaluation Liaisons

Evaluation Logic Model

Skills and Qualifications

Satisfaction

Relationship

Early Childhood Mental Health Consultation

Early Care and Education

- Attitudes, Beliefs, and Stress

- Classroom Environment

Parenting

- Behavior and Attitudes

- Stress

Young Children

↓Challenging Behavior

↑Social Skills

Expulsions

Evaluation Components

ECE ProvidersSDQTOSRQS

DECA

ParentsPSIPBI

DECA

ConsultantsClimate

RQS

ECE DirectorsGAS

Implementation Factors

Core Outcomes

Program DirectorsModel

Description,Quarterly Interview

ConsultantsKnowledge

& Skills,Service Log,

Exiting Child Form,Roster, Preschool

Climate Scale

Exiting Child Study

Consultants, ECE Directors, ECE Providers,

and Parents

QualitativeInterviews

Note: instruments in pink will only be collected for child-specific referrals

Levels of Data Collection

Program-level outcomes Measures assess changes in: ECE provider behaviors and

beliefs; job stress; structural and interpersonal characteristics of the ECE setting; and the quality of the relationship between the MHC and ECE providers.

Child-/Family level outcomes For child-specific referrals:

Measures assess changes in the child’s behavior (rated by teacher and parent), parenting stress and behaviors; and the quality of the relationship between the MHC and ECE providers.

ECMHC Program Directors

Tools Construct Frequency

Model Description Program Structure

Baseline, Annual

Quarterly Site Interviews Program Changes

Quarterly

Mental Health Consultants

Tools Construct Frequency

Knowledge and Skills Inventory Consultant Characteristics

Baseline, Annual

Service Log ECMHC Services Ongoing

Classroom Roster Administrative Tool

Ongoing

Preschool/Infant MH Climate Scale Classroom Environment

Baseline, every 4 months (ongoing)

Consultant Final Perceptions Form Quality of Relationship

4 months, or at Discharge

Exit Study (Qualitative Interviews)Perceptions and predictors of expulsion

Following an exit due to behavior issues

Preschool/Infant Climate Scale

These tools are completed by the mental health consultants for every child-specific and programmatic consultation they initiate. Preschool Mental Health Climate Scale (Gilliam, undated) Infant Climate Scale (FSU, 2009)

The instruments assess multiple dimensions of the classroom environment and ECE providers’ behaviors.

Consultants complete these scales at baseline, at 4 months or at discharge; and if ongoing, every 4 months thereafter.

Child Care Directors

Tools Construct Frequency

Goal Achievement Scale Opinions and Attitudes

Baseline, 4 months or at

discharge

Exit Study Interviews Perceptions of Expulsion

When child exits due to

behavior

Early Care and Education Providers

Tools Construct Frequency

Strengths and Difficulties Questionnaire Child Functioning

Baseline, 4 months or at discharge

Teacher Opinion Survey Teacher Attitudes

Baseline, 4 months or at discharge

Provider Job Stress Index Job Stress Baseline, 4 months or at discharge

Teacher Final Perceptions Form Quality of Relationship

Baseline, 4 months or at discharge

Exit Study Interviews Perceptions of Expulsions

When child exits due to behavior

Parents

* For those who consent to long-term follow along

Tools Construct Frequency

Parenting Stress Index-Short Form Parent stress Baseline, 4 months or at discharge

Parent Behavior Inventory Parenting practices

Baseline, 4 months or at discharge

Parent Final Perceptions Form Quality of relationship

Baseline, 4 months or at discharge

Devereux Early Childhood Assessment Child functioning

Baseline, 4 months, 12* & 24* months

Exit Study Interview Perceptions of expulsions

If their child is expelled

Exit Study Interviews

Motivated by MSDE’s interest in this policy-relevant (but infrequent) outcome

Always interview the consultant; seek the additional perspectives of ECE Director, provider and parent

Includes open-ended questions such as: What behaviors led to the child’s exit? What was done to try to maintain the child in the

placement? Did you expect this child to be expelled? Why or why not?

Professional Activity Units

Working with MSDE, established process for ECE Directors and Providers to get professional activity units when complete baseline and follow-up measures for evaluation

Increase participation and help with follow-up data

Current Status and Future Directions Data collection in the field: good

participation from all sites Exit interviews completed with

consultants, and a few ECE providers Comparison group Addition of CLASS as objective measure

of change Paper on Preschool Mental Health

Climate Scale with Walter Gilliam