Why Everyone Should Assess Program Implementation

Jerusha Gerstner Tisha McCoy-Ntiamoah Sara FinneySarah SundeMatthew SwainAssessment

Graduate AssistantDirector of Orientation Associate Assessment

SpecialistAssociate Director of

OrientationAssessment

Graduate Assistant

2

Goals

Define implementation fidelity assessment Explain the importance of implementation fidelity assessment Describe how to practically collect implementation fidelity data Create an implementation fidelity assessment checklist Integrate results from both outcomes and fidelity assessment

3

Transfer Orientation at JMU

History of Transfer Summer Springboard Who are JMU Transfer students? Phases of Orientation Many programming aspects are implemented by staff outside

of Orientation, so out of direct control

4

Transfer Summer Springboard

Academic Requirements Knowledge:o Upon completion of Transfer Summer Springboard, students will be

able to correctly identify the academic requirements for major, degree, and graduation completion at JMU.

Resource Knowledge:o As a result of attending Transfer Summer Springboard, students will

demonstrate an increase in knowledge of JMU resources by correctly matching university resources to student needs.

Social Acclimation and Community Building:o As a result of attending Transfer Summer Springboard, students will

indicate a significant increase in their cohesion to the JMU community.

5

Transfer Summer Springboard

Programming involves:o Instruction in academic policieso Instruction about resources available to them on campuso Welcoming students to the JMU community

Assess outcomes using a pretest/posttest design, no control group

6

Transfer Summer Springboard

7:45 – 8:15 a.m. Check-in Warren Hall 2nd & 3rd floorConfirm your e-ID username and have your JAC (JMU Access Card) photo taken. Refreshments are available on the Warren Hall patio.

8:25 – 9 a.m. University Welcome Grafton-Stovall TheatreDr. Linwood H. Rose, President of James Madison University, will welcome you to the community. Dr. Roger Soenksen, a JMU faculty member, will offer a special message to prepare you for the JMU learning experience.

9:10 – 11 a.m. Peer discussion with your ANT & TOPA Taylor HallHere’s your chance to get your questions answered, meet other transfer students, get the scoop from your ANT (Assisting New Transfers) and TOPA (Transfer Orientation Peer Adviser). You will also learn more about GenEd, course registration and transfer credits. Check out the green insert in your folder for your meeting location.

11:10 – 11:35 a.m. JMU Student Services Grafton-Stovall TheatreThis informative video presentation will provide essential information about dining, health services, information technology, financial aid, on- and off-campus housing, banking on campus and the University Business Office. Follow along using the handout in your folder.

11:40 a.m. – 12:25 p.m. Student Resource and Housing Fair Warren Hall, 2nd FloorYou won’t want to miss this! Full of information and resources, the fair will provide you with the details you need to navigate your way at JMU. Still need to find housing for the fall? Come talk with local property managers and find a place just right for you!

12:25 – 1:15 p.m. Lunch Gibbons Hall Entrance 4 & 5

1:30 – 5 p.m. Advising and Registration Check your folder labelStudents will meet with an advisor in departmental offices and register for courses online in one of the following computer labs: Ashby Hall (ground floor entrance), ISAT 336 and Showker Hall 208.

Don’t leave for the day without stopping by Taylor Down Under! Come on down to TDU for: Complete your program evaluation ◊ Your JAC! ◊ Refreshments ◊ Chatting with ANTs & TOPAs ◊ Campus tours

7

Assessment Cycle

Establish Objectives

Map Objectives to Programming

Select/Design Instrument

Collect Information

Analyze/Maintain

Information

Use Information

12

34

5

6

8

Implementation Fidelity

Objective Treatment Outcome

B DCA

Program Components

9

Implementation Fidelity

Objective Treatment Outcome

B DCA

Program Components

“Black Box”

Assessment Cycle

Establish Objectives

Map Objectives to Programming

Select/Design Instrument

Collect Information

Analyze/Maintain

Information

Use Information

12

34

5

6

Implementation Fidelity Check

11

Fidelity Components

Program Differentiationo A determination of the specific features of a program.

Exposureo Measures the duration of each program component in actuality and

how many students actually attend the various components. Adherence

o Assesses whether all features of the program were administered. Quality

o Assesses how well the program features were delivered. Responsiveness

o The extent to which the participants were engaged in the treatment.

12

Program Differentiation

What is covered or presented in Transfer Summer Springboard? What are the specific features associated with each program component? What do implementers do to enable students to ultimately meet the outcome? o Important phase, essential for other phaseso Involves collaboration with stakeholders

13

Exposure

Exposure measures (a) the duration of each program component in actuality and (b) how many students actually attend the various componentso Record actual duration of program component o Record attendance at events (if applicable)

14

Adherence

Was the full, prescribed program dose administered? Were all features covered?

15

Quality

How well were the specific features presented? May include presenter enthusiasm and method of delivery

o Rating of the quality of presentationo We used a 3-point scale, but qualitative comments are appropriateo Only applicable if the feature was presented at all (adherence)

16

Responsiveness

Engagement of participants during the program Example Measure:

o Observe and rate in audit the responsiveness of studentso Survey administered to students regarding responsiveness to material

How attentive were you throughout the day?

How effective were the presenters in providing information?

How engaged were you during the day?

1. Not all attentive2. Somewhat attentive3. Very attentive

1. Not all effective2. Somewhat effective3. Very effective

1. Not at all engaged2. Somewhat engaged3. Very Engaged

17

Audit Checklist

Program Differentiation Exposure Adherence Quality

18

Fidelity Audit

Day of auditingo Pretended to be studentso Attended all programming

Evaluate if aspects of programming were completed to match the outcomeso Utilized audit checklist when observing the programo Requested that implementers evaluate their programo Incorporated into results

Provide information beyond the scores on the outcome measure

Give insight into reasons for the outcome scores

19

Using Fidelity Results

Objective-level results from 2010 and 2011

2010 2011Pre Fidelity

Score a Post Pre Fidelity Score a Post

Academic Requirements Knowledge Scale

- 2.2 76.4% 65.8% 2.42 85.6%

Resource Knowledge Matrix - 1.75 40.5% 42% 2.43 49.8%Perceived Cohesion Scale b 6.72 3 7.27 7.15 3 7.57 Note: a Mean quality scores. b Mean scores range from 1 to 9.

20

Closing the Assessment Loop

In 2010, our outcomes and fidelity results directed our attention to programming mapped to the Resource Knowledge objectiveo Specifically, the Student Services video’s script was updatedo Where do students go to purchase a meal plan:

2010Changed

programming to increase

clarity

2011Option Percent

University Business Office 2.5Card Services 12.1Office of Admissions 0.8Dining Services 83.0University Registrar 1.1Major Department Head 0.2University Advising 0.4

Option PercentUniversity Business Office 3.2Card Services 25.6Office of Admissions 0Dining Services 71.0University Registrar 0.2Major Department Head 0University Studies 0

21

Closing the Assessment Loop

Additionally, we wanted to increase retention of important information presented about Academic Requirementso Using the Outcomes and Fidelity information, we informed our

stakeholders and program implementers of areas of weaknesso Minimum number of credits granted by any 4-year institution:

2010Changed

programming to increase

clarity

2011Option Percent

30 credits 10.460 credits 72.090 credits 0.2120 credits 17.4

Option Percent30 credits 7.260 credits 87.390 credits 0.7120 credits 4.8

22

Benefits

Implementation fidelity assessment has a myriad of benefits:o Before the program, the program differentiation stage assists in

structuring the program A good way to begin the process Helps the Student Affairs

practitioner and stakeholder A tool to establish a curriculum

with program implementers Focuses our efforts as a staff

23

Benefits

Implementation fidelity assessment has a myriad of benefits:o Before the program, the program differentiation stage assists in

structuring the programo During the program, the audit informs practitioners of program drift

and the student’s perspective

Benefits

Implementation fidelity assessment has a myriad of benefits:o Before the program, the program differentiation stage assists in

structuring the programo During the program, the audit informs practitioners of program drift

and the student’s perspectiveo After the program, all data is considered with the fidelity data:

Outcomes Data Fidelity Data Conclusions

Positive High More conclusive evidence that the program planned was actually implemented and works as planned

Positive LowParticipants were able to perform well on the outcomes measures but it was not due to the program that was implemented—participants may have the knowledge outside of the planned program

Negative High As the planned program and the actual program have high fidelity, these results are indicative that the program does not work as planned

Negative LowParticipants performed poorly but the planned program was not implemented fully or with low quality meaning that it could have been effective if implemented fully

25

Limitations

Fidelity is resource intensiveo Preparation of checklist and structuring programo Human resources

Different implementers of same program Inability to audit all programming

26

Discussion

Everyone should utilize implementation fidelity assessment Using what you have learned today, what is one program that

you are involved in that could use implementation fidelity?

27

Thank you!Questions?