Christopher R. Gareis, Ed.D . The College of William & Mary [email protected]

Principles of Program EvaluationA Workshop for the

Southwest Virginia Professional Education Consortium

Christopher R. Gareis, Ed.D.The College of William & Mary

[email protected]

Pre-Session Understandings

Perspective on Our Time Together Today

• The Profession of EducationThe regular and expert exercise of learned judgment and skills in service of one of an individual’s life needs and one of society’s greater purposes.

• Zen and the Art of Professional Development

• The Principle of Collective Wisdom

Let’s graph ‘em

Shared Understandings

• What is a “program”?A planned, multi-faceted series of activities leading to intended outcomes for individuals, groups, and/or a community.

• What is “evaluation”?The use of valid and reliable information/data to make judgments.

Definitions of “Program Evaluation”

• Judging the worth of a program. (Scriven, 1967)

• The process of systematically determining the quality of a school program and how the program can be improved.

(Sanders, 2000)

• The deliberate process of making judgments and decisions about the effectiveness, direction, and value of educational programs in our schools.

(me)

DO YOU KNOW OF…?

• …a program that you thought was effective but was discontinued without explanation?

• …a program that you believe is likely a waste of time, money, and effort, and yet it continues?

Random Acts of Improvement

Our Intentions

Our Efforts

Aligned Acts of Improvement

Our Intentions

Our Efforts

Perspective

Planning Evaluation

The Role of Evaluative Thinking

Planning

Redesigning Evaluative Thinking Implementing

Frechtling, 2007

• > 20 major models of program evaluation• Seminal Model: Daniel Stufflebeam’s CIPP

model (1971)

Our focus: Creating a Logic Model Focusing the Evaluation Identifying Performance Indicators EVALUATIVE THINKING

Our Objectives Today1. Understand basic concepts and principles of program evaluation

at the school level

2. Create a logic model that accurately depicts the essential components of and relationships among input, process, and outcome variables of a school-based educational program

3. Select the appropriate focus for the evaluation of a program

4. Identify appropriate data sources aligned to the intended educational outcomes of a selected school-level program

5. Appreciate the importance of engaging in intentional program evaluation as a teacher leader

Talk About Your Program

1. What is your program?2. What does your program do?3. Who participates in your program?4. Whom does your program serve?5. How do they benefit?6. How do you know, or how would you know,

that you program is a success?

Think Visually

• Sketch a visual metaphor of your program.

Every educational program:• Uses resources or INPUTS, such as…

– Materials– Funds– Teachers– Students

• Engages in activities or PROCESSES, such as…– Instruction– Interventions

• Intends produce certain results or OUTCOMES, such as…– Learning– Achievement– Engendering of certain attitudes or values– Eligibility for a next step in life– Bringing about some change

Visualizing an Educational ProgramThe Logic Model

Inputs Processes Outcomes

What is a logic model?A diagram with text that illustrates and describes the reasoned relationships

among program elements and intended outcomes to be attained.

A visual theory of change.

We use these

resources…

For these activities…

So that these

students / teachers …

Can produce

these results…

Leading to these

changes for the better…

How Why

Modified from guest lecture by John A. McLaughlin (November 16, 2009) at The College of William & Mary

Simple Logic Model

Inputs Processes Initial Outcomes

Intermediate Outcomes

Ultimate Outcomes

Everyday Logic

Get an

antihistamine

Take the

antihistamine

Feel better

Try This

• You’re hungry.

• Sketch a logic model to address your need.

• Oh, but one twist: You have no food in the house.

Assumptions

Inputs Processes InitialOutcomes

Ultimate Outcomes

Core Teachers• Math

Resource Teachers• Special Education• Gifted Education

Common planning by teachers• Frequency• Enthusiasm

Student affect for instruction

Student engagement in instruction

Student achievement• Quarterly class grades in math

Student achievement• 6th grade Math SOL• 7th grade Math SOL

Greenlawn Middle SchoolCommon Math Planning Initiative


The Reach of Intended OutcomesInitial

OutcomesIntermediate

OutcomesUltimate Outcomes

Time Immediately or very closely following program activity

End of a term or year, typically

End of a year, several years, or beyond formal K-12 schooling

People Targeted individual students and/or aggregate groups

Targeted individual students and/or aggregate groups

• Targeted individual students and/or aggregate groups• May have an organizational or societal impact

Indicators • Discrete/Finite• Measurable and/or observable

• May be sets of discrete indicators• Measurable and/or observable

• May be sets of discrete indicators• May or may not be readily measured or observed

Now Try This

• The Great Recession has had a broad and, in many places, deep impact on Americans.

• One “solution” to our economic morass has been a call from policymakers for improved financial literacy of our citizens.

• K-12 schools are seen as the logical place to teach young citizens financial literacy.

• Sketch a logic model that depicts this “theory of change”Causality

Source: STEM Education Symposium (retrieved 9/16/12).

A Simple Logic Model:9th Grade Transition Program

9th Grade TeachersGuidance

CounselorsStudent Mentors

New Students

Mentor TrainingSummer

OrientationMentor

ActivitiesMilestone Activities

Sense of Belonging of 9th

Graders

Academic Success of 9th

Graders

Promotion to 10th Grade

Reduced Dropout Rates

Graduation Rate of 9th Grade

Cohorts

Stephen M. Millett, Susan Tave Zelman, (2005) "Scenario analysis and a logic model of public education in Ohio", Strategy & Leadership, Vol. 33 Iss: 2, pp.33 - 40

Source: SUNY NGLC Grant Proposal (retrieved 9/16/12)

Context & Inputs

Processes of Professional Development

Outcomes for Teachers Impact on Student Learning

Experience & Expertise of Professional Developers

Experience & Expertise of Participants- Pre-service teachers- In-service teachers- Instructional leaders- Administrators

Psychometric Principles-Translated into practical terms & skills for teachers

State Standardized Assessments- de facto Curriculum?

State &/or School District Curriculum

District Goals, Initiatives, or Imperatives

Explore AlignmentC=Ia=A

Unpack Curriculum- Content- Cognitive levels

Create a Table of Specifications or “blueprint”

Critique an assessment for validity & reliability

Explore uses of ToS- Create an assessment- Critique & improve an assessment- Create a unit plan assessment- Plan instruction- Analyze assessment results

Create good assessment items, prompts, assignments, & rubrics

Understand role of C=Ia=A Alignment

Understand relationship between assessment & evaluation

Understand & apply concepts of validity & reliability

Use a ToS to:- Create an assessment- Critique & improve an assessment- Create a unit plan assessment- Plan instruction- Analyze assessment results

Apply principles to the spectrum of assessment types & practices

Create assessments that yield valid & reliable data

Use assessment data to make decisions:- About student learning- For student learning- About assessments- About instruction

Contribute to the design & development of common assessments

Provide “opportunity to learn” throughaligned instruction

Exposure to the full curriculum (e.g., depth & skill development)

Increased student achievement

Deeper, more meaningful learning

A Model for Improving Assessment Literacy

TRY THIS

Task: Create a logic model for your program

Materials: • Chart paper (in “landscape layout”)• Post-It notes• Markers• Answers to the earlier questions about your

program

Gallery Walk(with Docents)

• Can you read the logic model (without the docent’s assistance)?

• Questions that the logic model raises for you?

• Feature(s) you want to “borrow”?

• Suggestions to strengthen the logic model?

Why bother with logic modeling?

• Seems like a lot of work.• Logic models are too complex—how could I

realistically ever know all the variables at play in a complex program!

• I don’t “think” this way.• Even if we created a logic model, what would

we do with it?!

Limits of Logic Models

• Represent intentions not reality• Focus on expected outcomes, which may

mean missing out on beneficial unintended outcomes

• Challenging to know causal relationships• Does not address whether what we’re doing is

right

Benefits of Logic ModelingMcLaughlin, J. A., & Jordan, G. B. (1999). Logic models: a tool for telling your program’s

performance story. Evaluation and Program Planning 22, 65-72.

1. Builds a shared understanding of the program and expectations for inputs, processes, and outcomes

2. Helpful for program design (e.g., identifying critical elements for goal attainment and plausible linkages among elements)

3. Points to a balanced set of key performance indicators

Are you going to New York or by train?

Your question IS your FOCUS

1. Implementation FidelityQuestion: “Are we implementing the program as

designed?”Focus: Inputs & Processes

2. Goal AttainmentQuestion: “Are we seeing evidence that we are

achieving our intended outcomes?”Focus: Initial, Intermediate, and/or Ultimate Outcomes

ESL Dual-endorsement ProgramApproved teacher

preparation programs

MDLL TESOL/ESL

courses

VDOE regulations for ESL prep

ESL summer school in

LEAs

Recruitment of

candidates

Scheduling MDLL

courses

Advising candidates

Locating field sites

Program approval

Arranging field experiences

• Orientation• Transportation• Supervision

Coursework

Field Experiences

Elem/Sec/SPED program

Dual

ly-e

ndor

sed

teac

hers

Satis

facti

on w

ith

prep

arati

onIm

pact

on

stud

ent

lear

ning

Once you have a logic model and performance indicators, what do you do?

1. Determine the intent (and “audience”):– Formative evaluation (e.g., implementation fidelity and

improvement—an assumption of continuation)– Summative evaluation (e.g., to continue or discontinue)

2. Articulate a limited number of relevant evaluation questions

3. Identify who is needed to conduct the evaluation4. Determine how the evaluation will be conducted

(time, money, data collection & analysis, compilation & reporting)

http://www.google.com/imgres?imgurl=http://meera.snre.umich.edu/images/copy_of_logicmodelnoindbig.png/image_preview&imgrefurl=http:// …

Where would IF evaluation

occur and where would GA

evaluation occur?

http://www.google.com/imgres?imgurl=http://meera.snre.umich.edu/images/copy_of_logicmodelnoindbig.png/image_preview&imgrefurl=http://

What is the focus of the program evaluation that would answer each of the following questions?

a. Do methods used by teachers at Smith High School correspond to the principles of block scheduling introduced last fall?

b. Did the new reading curriculum result in improved reading abilities among 2nd graders? Increased interest in reading?

c. How consistently are Instructional Teams developing standards-aligned units of instruction for each subject and grade level (IIA01)?

d. Which recruitment strategies (of the three that we used) were most effective in involving fathers in Early Head Start?

e. How effective was the character education program in our middle school?

f. Did our new guidance program help new ESL students transition socially to our school?

g. How many AVID teachers have we trained during the past three years?

Adapted from Fitzpatrick, Sanders, & Worthen. (2004). Program evaluation: Alternative approaches and practical guidelines. (3rd ed.) Boston: Pearson (pp. 373-374).

i.e., “Implementation Fidelity “or “Goal Attainment”

What Information Do You Use to Answer Your Questions?

Examples of Assessment Sourcesin Schools

• Student attendance rates• Teacher attendance• PTA membership• Achievement gap among subgroups• Inequity of class enrollment (e.g.,

proportionally fewer minority students in AP classes)

• Special education referral rates• Behavior referrals• Reading levels• SOL scores• PALS scores• DIBELS scores• Benchmark test scores• SOL recovery data• SAT scores• AP scores• Surveys

• Class grades• Grade distributions • Algebra I enrollment / completion• College acceptances• Summer school rates• Dropout rates• Retention rates• Acceleration rates• Identification for gifted services• Athletic championships• Debate team competitions• Student demographics• Staff demographics (e.g., years of

experience, licensure status)• Family demographics (e.g., income,

educational levels)• Financial resources (budget)• Per pupil costs

What assessment sources could you use to gather relevant, accurate, dependable information/data to answer each question?

a. Do methods used by teachers at Smith High School correspond to the principles of block scheduling introduced last fall?

b. Did the new reading curriculum result in improved reading abilities among 2nd graders? Increased interest in reading?

c. How consistently are Instructional Teams developing standards-aligned units of instruction for each subject and grade level (IIA01)?

d. Which recruitment strategies (of the three that we used) were most effective in involving fathers in Early Head Start?

e. How effective was the character education program in our middle school?

f. Did our new guidance program help new ESL students transition socially to our school?

g. How many AVID teachers have we trained during the past three years?

Adapted from Fitzpatrick, Sanders, & Worthen. (2004). Program evaluation: Alternative approaches and practical guidelines. (3rd ed.) Boston: Pearson (pp. 373-374).

Avoid DRIP:Align Your Data Sources to Your Logic Model

Inputs

• Data source(s) to assess attainment or completion

Processes


Initial Outcomes


Long-term Outcomes


Aligning Focus and Assessment Sources:Process (P) or Outcome (O) Indicator?

[͟ Using a classroom observation protocol to determine % of student engagement

[͟ Advanced pass rate on high school end-of-course SOL test

[͟ Questionnaire of 4th grade math students to determine degree of “mathphobia”

[͟ Review of teacher-made lesson plans over the course of a 9-week period to determine % including explicit phonics instruction

[͟ Graduation rates

[͟ Number of “leveled” books available in the media center tracked over a 3-year period

[͟ Survey of students’ attitudes about participating in an after-school tutorial program (e.g., “On a scale of 1-to-5, how much did you enjoy attending the after-school program? How helpful was your after-school tutor?”)

[͟ 3rd grade SOL pass rates

[͟ % enrollment of minority students in foreign language courses in the high school

[͟ Implementation of an advisory period at the middle school level

[͟ Change from a standard 7-period bell schedule to an alternating-day block schedule

[͟ Average AP scores

[͟ Student attendance rates

[͟ Teacher attendance rates

[͟ Review of committee meeting agendas to determine % that focus on discussion of achievement data

[͟ Budget allocation per student

[͟ Grade Point Averages

AVID: Advancement Via Individual Determination

Retrieved 9/16/12)

TRY THIS

Tasks:

1. Imagine undertaking an Goal Attainment evaluation of your program. a. Articulate at least 2 evaluation questionsb. Identify 1-2 performance indicators (aka, data sources) that

could help answer each question

2. If time allows, try doing the same for an Implementation Fidelity evaluation.

Materials: • Your logic model

Sage Advice

Remember that not everything that matters can be measured,

and not everything that can be measured matters.

Know your means from you ends

To make “evidence-based decisions," you must have data that are:

• Valid Necessary for drawing appropriate inferences

• Reliable Necessary for avoiding erroneous judgments

• Triangulated Necessary to confirm data

• Longitudinal Necessary to determine trends and patterns

• Disaggregated Necessary to provide "fine-grain" analysis, rather than over-generalizing

How do SOL results hold up as a data source?

“Without evaluation, change is blind and must be taken on faith”

-- Sanders (2000)

Managing Complex ChangeAdapted from Knoster, T. (1991) presentation at TASH Conference, Washington, DC.

Vision Skills Incentives Resources Action Plan CHANGE

Vision Skills Incentives Resources Action Plan CONFUSION

Vision Skills Incentives Resources Action Plan ANXIETY

Vision Skills Incentives Resources Action Plan RESISTANCE

Vision Skills Incentives Resources Action Plan FRUSTRATION

Vision Skills Incentives Resources Action Plan TREADMILL

Vision Skills Incentives Resources Action Plan

Evaluative Feedback

CHANGEfor the better

Time to Evaluate

1. Understand basic concepts and principles of program evaluation at the school level

2. Create a logic model that accurately depicts the essential components of and relationships among input, process, and outcome variables of a school-based educational program

3. Select the appropriate focus for the evaluation of a program

4. Identify appropriate data sources aligned to the intended educational outcomes of a selected school-level program

5. Appreciate the importance of engaging in intentional program evaluation as an teacher leader

Empirical Research ≠ Program Evaluation

• Identifying focus• Hypotheses testing

• Value judgments• Replication of results• Data collection

• Control of variables• Generalizability of results• Focus

• Organizational need• Comparison of outcomes to

intended program outcomes• Contextually driven• Low likelihood• Broad and heavily dependent

upon feasibility• Little to none• Little to none• Dependent variables (i.e., results)

more so than on Independent variables (i.e., the cause)

Investigating the Efficacy of a Clinical Faculty Program

Christopher R. Gareis, Ed.D.Leslie W. Grant, Ph.D.

The College of William & Mary

Virginia Standards for Mentoring/Clinical Faculty

Partnership divisions/schools

Experienced classroom teachers

School of Education faculty/staff

School of Education teacher preparation program competencies and expectations

Context & Inputs

Processes Program Activity

Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection

2 + 1 Credits 2-week

summer course

4 follow-up meetings (fall and spring)

Knowledge, Skills, and Dispositions of Mentoring Relating Assessing Coaching Guiding

(Portner, 2003)

Knowledge, Skills, and Dispositions of SOE teacher preparation program competencies and expectations

Sense of efficacy for CF role (during and upon completion of training)

Sense of efficacy for CF role (after ≥ 1 year as CF)

Impact on student learning as new teachers (perceived)

Impact on student learning: Classroom

measures School/ division

measures State

measures

Evaluation of field experience by new teachers ≥ 1 year post-graduation

Formative feedback for teacher candidates during student teaching

Mid-term evaluations o CF/CT o TC o US

Teacher Candidate competence by conclusion of student teaching Final Evaluations

o CF/CT o TC o US

Administrator evaluations

Portfolio evaluations

Impact on student learning by teacher candidate

Evaluation of field experience by teacher candidates

Employer satisfaction with new teacher performance

Sense of self-efficacy for teaching (after ≥ 1 year post-graduation)

Intent to continue in the profession (after ≥ 1 year post-graduation)

Growth of Clinical Faculty network

What we have already known






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









Indicators of Improved Effectiveness of Recruitment, Selection, & Training

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100

5

10

15

20

25

30

35

40

45

50

Total Trained

Targeted Re-cruiting

Restructured Program

Level-specific cohorts begin

Currently Active

Percentage of Placements with CT vs CF

Indicators of Perceived Effectiveness Training

Coherence between field and university:• I have had student teachers in the past with absolutely no guidance or direction

from the college. I’m not sure how effective I was. This class has been so beneficial and I feel I am more confident in mentoring this time! Thank you!

Skill development as a mentor:• I gained awareness in many of the philosophies behind coaching. As I learned

techniques, I grew as a mentor. • This program is essential for bringing professionalism to the training of student

teachers. Mentoring does not always come naturally and having techniques and strategies to address problems and guide students is invaluable.

Outcome orientation:• I am prepared to mentor aspiring teachers. I am prepared to help them become

independent and effective teachers.Professional growth as a teacher:• I always feel very good about what I’m doing when I leave class. I like having the

professional growth and being able to take a good look at my teaching. I have really enjoyed it. What a great program.

• I didn’t realize how much I didn’t know about being a CT.

Overall Evaluation of Experiences in the Clinical Faculty Training

1=Poor, 5=Excellent

What We Want to Know:

Cooperating Teachers Clinical Faculty






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









Research Questions1. To what degree do CF differ from CTs in their sense of self-

efficacy for roles of a cooperating teacher?

2. To what degree do mid-term evaluations of student teachers placed with CF differ from those placed with CTs?

3. To what degree do final evaluations of student teachers placed with CF differ from those placed with CTs?

4. To what degree do new teachers who had been placed with CF differ from those who had been placed with CTs with regard to sense of efficacy for teaching, perceived impact on student learning, and intent to remain in the profession?






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









Research Methods – Data Collection

Surveys– Clinical Faculty & Cooperating Teachers (R1)

• 1998 – 2011 • n = 101• Response rate: 37.0%

– Graduates of the School of Education (aka, new teachers) (R4)• 2005 – 2010 • n = 94• Response rate: 44.76%

Mid-term & Final Student Teaching Evaluations (R2 & R3)– 2008 – 2011 (n = 319)

• Student Teacher self-evaluations• CT/CF evaluations • University Supervisor evaluations

Research Methods – Data Analysis

Unit of Analysis – Cooperating teacher designation

– Clinical Faculty (CF) – Trained through the School of Education’s Clinical Faculty Program

– Cooperating Teacher (CT) – Not trained through the School of Education’s Clinical Faculty Program

Statistical Analyses

– T-tests for significant differences according to CT/CF designation






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









RQ 4

Research Questions 2 & 3

2. To what degree do mid-term evaluations of student teachers placed with CF differ from those placed with CTs?

3. To what degree do final evaluations of student teachers placed with CF differ from those placed with CTs?






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









Statistically Significant Differences in Mid-Term Ratings (RQ2)

Competency CF Rating CT Rating

1. Understands subject matter and pedagogy...

2.30 2.44

17. Implements assessments for learning 2.20 2.38

26. Participates in and applies professional development

2.41 2.60

30. Demonstrates potential for teacher leadership

2.34 2.51

4 out of 30 Competencies

Differences in Mid-Term Ratings (RQ2)

1 3 5 7 9 11 13 15 17 19 21 23 25 27 292

2.1

2.2

2.3

2.4

2.5

2.6

2.7

CT RatingCF Rating

Statistically Significant Differences in Final Ratings (RQ3)

Competency CF Rating CT Rating

16. Creates and selects appropriate assessments for learning

2.36 2.52

1 out of 30 Competencies

Differences in Final Ratings (RQ3)

1 3 5 7 9 11 13 15 17 19 21 23 25 27 292

2.1

2.2

2.3

2.4

2.5

2.6

2.7

2.8

2.9

CT RatingCF Rating

“Overall Teaching Effectiveness” Ratings (R2 & R3)

Mid-Term Final

CF CT CF CT

Student Teacher Self-Evaluation Ratings

2.10 2.12 2.37 2.48

Cooperating Teacher Evaluation Ratings

2.29 2.39 2.63 2.69

University Supervisor Evaluation Ratings

2.10 2.07 2.69 2.64

Overall Rating Comparison

Student Teacher

CF/CT

University Supervisor

Research Question 1 (RQ1)

1. To what degree do CF differ from CTs in their sense of self-efficacy for roles of a cooperating teacher?

Sense of Efficacy for the Role of CF/CT (R1)

QuestionClinical Faculty (trained)

n=76Cooperating Teachers

(non-trained) n=25T-Test

Equal Variances Assumed

T-TestEqual Variances Not

Assumed

Convey Role as CF/CT* 4.51 4.04 0.007 0.038Four Fundamental Roles of Mentoring* 4.24 3.65 0.002 0.015

Foster Relationship 4.57 4.62 0.768 0.772

Effectively Observe* 4.59 3.92 0 0.001

Use a Variety Supervision Strategies** 4.51 4.19 0.066 0.111

Effectively Conference 4.47 4.23 0.169 0.198

Summatively Evaluate ST* 4.55 4.04 0.001 0.006

Impact Professional Abilities of ST 4.49 4.38 0.423 0.447

Provide High Quality Field Exp. 4.57 4.54 0.784 0.772Likelihood ST Positively Affects Pupil Learning 4.37 4.38 0.941 0.939

Likelihood ST Enters Teaching 4.37 4.31 0.663 0.687

Likelihood ST Remains in Teaching 3.99 4.04 0.774 0.796Likelihood ST Positive Impacts Pupil Learning in First Year 4.41 4.31 0.454 0.51Likelihood ST Emerges as a Teacher Leader 4.08 4 0.662 0.686

*significance < .05

**significance < .10

To what experiences do CF and CT attribute their mentoring acumen?

Group Experience % Indicating Most Important Experience

Cooperating Teachers Experience as a classroom teacher

28.0%

Clinical Faculty Clinical Faculty training* 38.7%

* 20% of CF selected “experience as a classroom teacher” as Most Important.






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









RQ 2

RQ 3

RQ 1

What could explain...

...differences in student teaching outcomes associated with Clinical Faculty?

....the lack of differences between CF and CTs in both intermediate and long-term outcomes?






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









Should we continue,

discontinue, or change

the W&M Clinical Faculty Program?






Context & Inputs


Initial Outcomes


Long-term Outcomes

Ultimate Outcomes

Recruitment

Application

Selection


summer course



(Portner, 2003)







measures State

measures





o CF/CT o TC o US









Program Evaluation in Use

Berkeley Middle SchoolSOL Jam & Cram

Spring 1999

INPUTS PROCESSES OUTPUTS

INITIAL

OUTCOMES

INTERMEDIATE

OUTCOMES

ULTIMATE

OUTCOMES

Analyzed SOL results and class performance

Selectedstudents: C or B

avg. LPT 140-

160 7th gr. SOL

350-400

Invited 175 “in the middle” students• Moti-

vational “recruit-ment”

Analyzed SOL test blueprints

Targeted specific SOLs for focused review

Review & enrichment activities

High-activity, novel instructional strategies

8 sessions (1/2 hour each)

Timely (week prior to SOLs)

Volunteer teachers (but paid)

Collaboratively planned lessons

Lunch & door prizes provided

Sense of “efficacy” for SOL success (survey of students)

SOL assessment results

Academically prepared for success in high school

Independent thinker

Responsible citizen

Lifelong learner

SOL Jam & Cram

Why Did Students Attend?

7th 8th

50% 32% It is important to me to do my best in school.

33% 65% I want to avoid having to go to summer school.

10% 3% My parents made me sign up.

7% -- Some of my friends signed up, so I did, too.

Why Did Students NOT Attend?

7th 8th

4% -- I do not enjoy learning.

-- 6% I already know that I will have to attend summer school.

4% -- I don’t’ believe that Jam & Cram would help me on any of the SOL tests.

4% 6% I do not feel that I fit into the group of students who were selected to attend.

52% 56% I have a conflict on the day of Jam & Cram, so I wouldn’t be able to attend.

26% 25% I forgot to return the slip on time

9% -- I can study fine on my own

-- 6% I didn’t want to go. (“It was really stupid, dumb.”)

-- 3% I don’t like school.

Post-SurveyParticipants

To what degree do you agree or disagree with the following statement: “Jam & Cram helped me to be better prepared for the SOL tests.” [4 pt. scale]

7th Grade: 3.688th Grade: 3.25

Non-Participants

7th 8th

35% 22% I wish I had attended

45% 56% I’m glad that I didn’t attend because I think I did well anyway.

21% 22% Other (mostly explanations of why they didn’t attend)

Percentage of Students Passing SOL TestsBerkeley Middle School

Grade 8

CT-08 EngRLR-08 EngW-08 HSS-08 Math-08 Sci-080

20

40

60

80

100

1998-991999-00

HSS-08 / 7th Grade

Panthers (2.75 GPA)

Control-P (2.67 GPA)

Jaguars (2.74 GPA)

Control-J (2.67 GPA)

0%

1000%

2000%

3000%

4000%

5000%

6000%

7000%

8000%

9000%

ChowsChows (2.69 GPA)

Control-CC (2.47 GPA)

Greyhounds (3.13 GPA)

Control-G (3.03 GPA)

0%

1000%

2000%

3000%

4000%

5000%

6000%

7000%

History %

An Analysis of SOL Data: What We Can Learn

1. We did great last year with gains of 11-17 percentage points in all areas!2. Science is our strongest area: highest pass rate, highest advanced pass rate, and

relatively low disparity.3. Writing is a strong area: high pass rate and relatively low disparity…but very few

advanced passes.4. English/Reading is a relatively strong area with a high pass rate and a very high

advanced pass rate.5. Computer Technology is a consistently strong area…but has the second highest

disparity group6. Math 8 is conquerable…but the disparity is wide and our overall pass rate is still in

shaky territory7. History/Social Studies remains our greatest challenge in terms of overall pass rate and

disparity by both ethnicity and gender…BUT we had whopping gains last year!8. Our year 2002 class (last year’s 6th grade) is academically strong. Our 2003 class (this

year’s 6th grade) is even stronger!9. Jam & Cram was a successful strategy by all anecdotal accounts, but was not effective

statistically speaking. We may want to repeat the program but with definitive subgroups of students and not in a single cram sessions.

10. SOL Resource class appears to have been a success based on pass rates alone (nearly 100%), although it has not been compared to control data.

Documents

Christopher R. Gareis, Ed.D . The College of William & Mary [email protected]