Accountability Through

Evidence-Based Practice

Alan Burkard, Ph.D.

Marquette University

American School Counselor Association

Presentation Goal

The goal of this session is to

learn how to easily and

effectively use accountability

strategies to drive selection and

use of interventions and to

focus program decision making.

Assumptions about this presentation….

1. Participants want to learn about evaluation and

evidence-based practice strategies for school

counseling programs (e.g., group counseling,

guidance curriculum, workshops).

2. Participants have every intention of implementing

these evaluation and evidence-based practices at

their school.

3. Participants lack some, if not a lot, of confidence

designing and using evaluation and evidence-based

practices.

4. Participants want to start developing initial plans for

evaluating at least one intervention in their school.

Research vs. Evaluation

Research…

Is theoretical based

Uses the scientific

method to understand

phenomenon of interest

Is about hypothesis

testing

Seeks the

generalizability of results

Evaluation…

Is focused on a local

intervention or program

Seeks to understand the

effectiveness of the intervention

for a local population

Is used to facilitate local

decision-making about the use

of interventions in the school

Is used to make decisions

about resources

ACCOUNTABILITY/

PROGRAM EVALUATION

Getting Started:

1. You need a partner (first year)

A. Set a timeline

B. Established tasks to be

completed

2. Develop a Data-Based

Decision-Making Team

(second or third year)

Accountability Overview: PIES

P: Problem Definition What student or school concern would you like to address?

What would you like to be different (i.e., outcome)?

I: Intervention How will you intervene to change the student?

Summarize the problem and the intervention in the form of a question

E: Evaluate Your Intervention (Summative Evaluation) Who will you evaluate (triangulate)?

What will you measure (i.e., change in outcome)?

When will you collect the data (i.e., research design)?

How will you analyze the data (i.e., statistics)?

Make sense of the results

Consider the implications of the results

Communicate your results and the implications

S: Systemic Review

PIES-Program Evaluation Cycle

Problem

Definition

Evaluation Intervention

Systemic Review

ASCA National Model

LET’S TAKE A LOOK AT A

PROBLEM…

Doe High School 09-10: Truancy Current seniors are at 25.7% truancy rate, as freshman they were at 4%

0

5

10

15

20

25

30

T ruancy-Grade Level

9th

10th

11th

12th

Doe High School 09-10: Truancy Current seniors are at 25.7%, as freshman they were at 4%

0

2

4

6

8

10

12

14

16

18

20

T ruancy-Gender

F emale

M ale

•Drop out rates and

suspensions are not

high in comparison

to these figures

There was nearly a

40% drop in

participation of

academic activities

from 2005-05 to

2009-10

PIES: Problem Definition

Five types of needs assessment

1. Data based (State exams, ACT/SAT, attendence)

2. Organization based (AYP, School Improvement

Goals, School Mission, DBDM Team, ASCA

Program Audit)

3. Standards based (ASCA Student Standards)

4. Perception based (Stories, stakeholder beliefs,

surveys/questionnaires)

5. Theory based

All types are important, all types should be considered when

developing a program

What do we do with needs assessment information?

Answer: It depends on what you want to know.

1. Disaggregate the data/Achievement Gaps Gender

Race/Ethnicity

Socio-economic status (i.e., free/reduced lunch)

Limited English Proficiency

Family configuration (e.g., single parent, foster care, two parents)

2. Identify problems that need intervention

3. Evaluate which problems align with the

school mission, school improvement goals,

school counseling program goals

4. Prioritize the problems for intervention

EXAMPLES OF DATA TO EXAMINE

Test Scores

Achievement

State

National

Enrollment

Honors/AP Classes

College Track

Special Education

LEP

Graduation Rate

By Gender

By Ethnicity

By SES

Attendance

Absences

Tardiness

By Grade Level

Discipline

By Classroom

Types of Problems

Gender

GPA/Class Rank

By Gender

By Ethnicity

By SES

Retention Rates

By Subject Area

By Grade Level

By Gender, Ethnicity

Post Secondary

Plans

Special Education

By Gender

By Ethnicity

By SES

Dropout Rate

Grade Levels

Gender, Ethnicity…

Reasons Why

1. Criticality

a. School Mission

b. School Improvement Goals

c. School Counseling Program Goals

(i.e., Foundation)

2. Closing the Achievement Gap concern

3. ASCA Student Standards

4. Frequency: Triangulation or number of

stakeholders reporting this concern

Prioritizing Needs Assessment Data:

Identifying Areas for Intervention

PIES: Intervention

Intervention = Delivery System in the ASCA

Model

Guidance Curriculum (developmental guidance,

small group activities, parent workshops)

Individual Student Planning (appraisal and

advisement)

Response Services (e.g., individual and group

counseling, crisis intervention, consultation, in-

service training, referrals)

System Support

The Center for School Counseling Outcome Research

have established criteria for evaluating the

effectiveness of interventions

Write down one problem in

your school

How do you know this problem is a concern?

PIES: Evaluation – Two Goals

1. Summative Evaluation: Are we creating change?

2. Formative Evaluation:

A. Dose of the intervention

B. Treatment Fidelity (match between the intervention

design and implementation)

I would encourage you to start with evaluating

change.

PIES: Evaluation – Steps for

Evaluation Pre-Intervention Evaluation Planning

1. What are you trying to change? Write as Goal

2. What interventions will be used to change this attitude,

skill, knowledge or behaviors?

3. What will you measure?

4. When will you collect the data?

5. How will you analyze the data?

Post-Intervention Evaluation Process

6. State your results

7. Interpret the data (make meaning of the results)

8. Make adjustments to your intervention(s)

9. Share your findings

PIES: Evaluate-What will you measure?

Types of Data

Process

Data

Perception

Data

Results Data

Competency-

Attainment

Data

Behavior

Change

Achievement

-Related

Data

Achievemen

t Data

Guidance

Lessons,

groups, etc.

Who?

What?

When?

Where?

How long?

Attitudes

Skills

Knowledge

Attendance

Discipline

referrals

Parent

Involvement

Homework

Completion

Course

Enrollment

WCKE

SAT/ACT

scores

Graduation

rates

GPA

AP tests

College prep

course

completion

PIES: Evaluate-When will you collect

the data?

There are several research designs and

each has strengths and limitations

Pre-test and Post-test is a common

approach and a particularly strong design

Post-test design is ineffective and invalid in

evaluation of activities

Factors to consider

1. Availability of data

2. Type of data

3. Goal of the evaluation

PIES: Evaluate-How will you analyze

the data?

For most school purposes, intermediate or

advanced statistics is not important

Recall, the goal is evaluation not theoretical

research

Appropriate Tools

Means

Percentages

Behavioral counts

Changes in test scores

As you gain confidence with the tools above you

can add other statistical and analytic strategies

Small Group Activity:

Plan the Evaluation Design

PIES: Evaluate-Make sense of the results

and consider the implications of the results

Start with the results

Did scores change?

How specifically? Did they go up or down?

Disaggregate the results (ethnicity/race, ELL, SES)

Which groups changed and which groups did not

change?

Consider the implications of the results

Examine each result and consider reasons for the

findings

Have you answered your question?

Consider implications for the intervention, or, are

students different as after your intervention?

Consider whether the intervention needs modification

PIES: Evaluate-Communicate your

results and the implications

MONITOR and ANNOUNCE:

How will you use your findings?

What are the recommendations?

How will you use your findings for improvement?

How will you present your findings and recommendations?

Who will you share them with?

What are the implications?

PIES: Systemic Review

Work with your DBDM team

Consider the implications of your findings for

your program, revision of the intervention

Develop a program evaluation report

Present your results to administrators, the

school board

Consider completing a SPARC

Apply for an ASCA RAMP Award

Remember: The New Question is….

How are students different as

a result of your

school counseling program?

Any other questions?

Thanks for your interest in evidence-

based practices!

Contact Information:

Alan Burkard, Ph.D.

Alan.Burkard@Marquette.edu