Evaluating Impact of Training related to Implementation of Early ACCESS in Iowa

Presenter: Janet MelbyCo-authors: Chris Rubino, Neil Rowe, Jiong YangIowa State University

National Human Services Training Evaluation Symposium

Pittsburgh, PAMay 23, 2013

About the TrainingTraining topic: Power of Teaming: Department of Human Services and Early ACCESS, Allies for Infants and Toddlers

Training objectives:1. Understand impact of trauma on infants and toddlers2. Identify how federal law supports child well-being3. Understand purpose of CAPTA, ASFA, and IDEA law4. Determine action steps between local DHS and EA providers to

facilitate improved collaboration.

Delivery: Nine, in-person, one-day, around Iowa, February-April, 2013.

Process: Content delivery, plus DHS/EA discussion and action steps

Evaluation: (a) Post-training paper survey-13 items, (b) action steps, (c) on-line training feedback

Evaluation Focus Effect of training on improvement in trainee:

Knowledge Approach to work with children Collaboration

Link between training effects and trainee description: DHS or Early ACCESS Role length Age, gender, race Current collaboration status

Sioux City

Council Bluffs

Fort Dodge

31

Waterloo

Cedar Rapids

Ottumwa

Burlington11

15

16

1921

30

29

Des Moines

15

Davenport

DHS

Early ACCESS

Community Provider

FSRP (Family, Risk, Safety & Permanency)

Other

About the TraineesRegistered for training: N = 212Paper post-training survey: N = 187

DHS = 61, Early ACCESS = 112, Other = 14Non-Caucasian = 3Males = 5

On-line follow-up: N = 132

Low High 2 As a result of today’s training, how likely are you to consider

trauma related issues when interacting with young children and families?

1 2 3 4 5

3 To what extent will this training help you serve young children better?

1 2 3 4 5

4 How much did you improve your knowledge of Child Development?

1 2 3 4 5

5 How much did you improve your knowledge of Child Trauma?

1 2 3 4 5

6 How much did you improve your knowledge of federal law regarding CAPTA (Child Abuse Prevention and Treatment Act)?

1 2 3 4 5

7 How much did you improve your knowledge of federal law regarding IDEA (Individuals with Disabilities Education Act)?

1 2 3 4 5

9 In your opinion, how important is collaboration in the area of Early ACCESS?

1 2 3 4 5

10 To what extent did today’s training change how you approach collaboration in delivery of services?

1 2 3 4 5

12 As a result of today’s training, to what extent do you know how to connect to people with whom you could collaborate?

1 2 3 4 5

13 How often will you collaborate in the future?

1 2 3 4 5

Impact Work

α=.615

Impact Knowledg

eα=.747

Impact Collaborati

onα=.625

Post-training Survey Items and Domains

8 Are you currently collaborating with others in the area of Early ACCESS?

YES (go to 8a)

NO (go to 8b)

8a. If yes: How often do you currently collaborate? YES OR

1 2 3 4 5

8b. If no: How likely will collaborate in the future? NO

1 2 3 4 5

Survey Numeric Results

2.71

4.15

4.59

4.13

4.18

4.16

3.47

4.56

4.42

3.48

Overall: EA higher than DHS on impact of training on Future Work with young children and on Collaboration (p < .01). Within Knowledge, improvements in CAPTA, child trauma, and IDEA.

Current collaborators: EA and DHS, both higher on Future vs. Current Collaboration; EA sig. higher than DHS on both Future and Current Collaboration (p < .01)

Action-Step Coding ResultsParticipants grouped by local areas (counties):

Discussed local issues and protocols Developed action steps with the goal of enhancing collaboration Action steps synthesized by trainer and coded by researcher

Main themes identified: Meetings

“Invite DHS/EA to FTM and IFSP meeting and attend if at all possible.”“Quarterly staffing for DHS, EA, and FSRP agencies (face to face or conference calls).”

Communication“Better communication through email.”“Get more information from DHS worker on family goals in case plan to possibly use as family goal for IFSP.”“Make sure EA has family interaction schedule and FSRP worker contact.”

On-line Training Feedback Results

Trainee item responses (% agree or strongly agree; N = 132): Training material informative and relevant (92%) Communication methods effective (92%) Appropriate pace (94%) Trainer knowledgeable on topic (94%) Gained better understanding (87%)

Open-ended comments: What did you like best about training?

Challenges Training development:

Matching DHS and EA participants for groups by geographic location and balancing the groups by individual roles within each agency

Developing a method to provide information about each agency’s work to the other agency in an engaging way

Getting people to sign up for the training

Evaluation design: Post-training survey

Developing survey items Revising survey form Inability to distinguish among DHS roles

Obtaining on-line evaluation data Inability to link paper survey to on-line evaluation

Conclusions and Implications Early ACCESS “Power of Teaming” training is

effective in impacting the knowledge, future work with young children, and future collaboration of participants.

Current work role of trainees (specifically DHS or Early ACCESS) is a major factor in the effect of the training in these three domains, and age is also an important factor regarding the impact of training on future work with young children.

Adding training-specific evaluation tools enhances assessment of training impact.

Being intentional in designing evaluation approaches has potential benefits.