Improving Teacher Quality Grants, Cycle 3:

External Evaluation ReportDecember 8, 2006

University of Missouri-Columbia Evaluation Team

Evaluation Team

Principal InvestigatorsSandra AbellFran ArbaughJames ColeMark EhlertJohn LanninRose Marra

Graduate Research Assistants

Kristy HalversonKristen HutchinsZeynep KaymazMichele LeeDominike MerleMeredith Park RogersChia-Yu Wang

Context of the Evaluation

• Improving Teacher Quality grants program, Cycle 3, 2005-2006

• Focus on high-need schools

• 9 professional development projects

• Science and Mathematics, grades 4-8

Evaluation Model

Adapted from Guskey, 2000

Purpose of Evaluation

• Formative evaluation• PD environment evaluation• Summative evaluation

– Participant reaction– Participant learning—content knowledge and

inquiry– Participant use of knowledge– Organization change – Student learning

Methods—Formative

• Site visits– Interviews: teachers and staff– Observations– Formative feedback report

Methods—PD Environment

• Teacher Participant Data Questionnaire

• Site visits– Interviews: teachers and staff– Observations

• Surveys to PIs (Teaching Philosophy Survey and Seven Principles)

• PI preliminary report

Methods-Outcomes• Participant reactions

– Site visits– Teacher Participant Survey 1 and 2

• Participant learning—content knowledge– Project-specific tests (all 9 projects)

• Participant learning—inquiry– Teaching Philosophy Survey– Seven Principles

• Participant use of knowledge– Teacher Participant Survey– Interviews– Seven Principles– Implementation Logs

Methods--Outcomes

• Organization change– Higher Education Impact Survey

• Student learning– Teacher-assessed (3 projects)– Teacher Participant Survey– MAP analyses

Participant Summary

• 252 participants• 86% female; 81% white• 40% held a masters degree or higher• 76% held their first Bachelor’s degree in a field

other than science or math• Represented 76 different Missouri school

districts, 6 private schools, and 2 charter schools• Directly impacted 16,747 students in 2005-2006

Assigned Teaching Levels and Subjects of Participants

10

56

1615

11

7

19

2

9

33 3

11

8

2

8

43 3

2

24

8

0

5

10

15

20

25

30

Elementary School Middle/Jr. High High School

Num

ber

of R

espo

nden

ts

Mathematics (n=31) Biology (n=17)Chemistry (n=12) Earth Science (n=21)Physics (n=15) General Science (n=34)Other Science Subjects (n=20) Other Subjects (n=32)

Change in Teaching Assignment

79.0%

21.0%

Same as previous year Change from previous year

Teaching Experience

4

23

17

47

9

3

13

22

54

98

36

5

34

18

0

10

20

30

40

50

60

0 Years 1-5 Years 6-10 Years 11-25 Years 26+ Years

% R

esp

on

de

nts

Current District (n=206) Current School (n=207) Overall (n=212)

Elem/Middle/Junior High Certification Status

72

41

127

7810 84

0

20

40

60

80

100

120

140

Elementary/early childhood Middle/junior high schoolmath

Middle/junior high schoolscience

Nu

mb

er

of R

esp

on

de

nts

Regular (n=240) Provisional (n=25) Temporary (n=12)

High School Certification Status

11

20

1310

6

17

1

0

5

10

15

20

25

Math Biology Chemistry EarthScience

Physics GeneralScience

Num

ber

of R

espo

nden

ts

Regular (n=77) Provisional (n=1)

PD Hours Completed in Past 3 Yrs

24

10

21

26

20

89

2525

34

0

5

10

15

20

25

30

35

40

0 Hours 5 or fewer 6 to 15 16 to 35 36 or more

Hours

% o

f Re

spo

nd

en

ts

Science education and/or science (n=209)Mathematics education and/or mathematics (n=191)

Experience with Web-based PD in Past 3 Years

14.1%

85.9%

Yes web-based No web-based

Percentage of Participants from High-Need Districts

48.8%

51.6%

% participants from high need districts % participants from non-high need districts

PD Coverage – Schools and Teachers

9.8

6.3

3.5

4.9

0

2

4

6

8

10

12

Schools Teachers

% S

erv

ed b

y C

ycle

3 P

D P

roje

cts

High Needs Non High Needs

Results

• PD Environment• Participant Reactions• Outcomes

– Participant Content Knowledge– Participant Knowledge of Inquiry– Participant Use of Knowledge of Inquiry– Organization Change– Student Learning

PD Environment--Projects by Standards Area

Science# PD

Projects

Matter & Energy 7

Force, Motion & Mechanical Energy 6

Living Organisms 1

Inquiry Approach to Science Education 9

Ecosystems 2

Earth Science 4

Astronomy 2

Scientific Inquiry 4

Science & Technology 0

Mathematics# PD

Projects

Numbers & Operations 3

Algebraic Relationships 1

Geometric & Spatial Relationships 0

Measurement 0

Data & Probability 1

Inquiry-based/Problem-centered Instruction 3

PD Environment—PI Beliefs (n=19)

M SD

Explainer vs. Facilitator 4.2 0.8

Whole class activity vs. Many things going on

4.2 0.7

Curriculum coverage vs. Sense-making 3.8 1.1

Textbook content vs. Student interests 4.3 0.7

Breadth of concepts vs. Depth of concepts 3.9 1.3

least constructivist response = 1, neutral = 3, most constructivist = 5

Participant ReactionsEnd of Summer End of Project

n M SD n M SD

Confidence in my content knowledge improved 130 3.0 0.8 117 2.9 0.7

Confidence in my ability to teach improved 130 2.8 0.8 116 3.0 0.8

I will use/have used materials and activities from this project in my classes during the year 130 3.4 0.8 116 3.1 1.0

The PD project was relevant to my teaching assignment 130 3.4 0.9 116 3.3 0.8

High quality instruction was delivered in this project 130 3.6 0.6 117 3.5 0.7

Instructors modeled good practice 129 3.6 0.7 117 3.4 0.8

Overall, I am satisfied with my experiences in this project 128 3.7 0.6 117 3.5 0.8

1-5 scale

Participant Performance on Content Knowledge—Post/Pre Tests

0

50

100

150

200

250

CMSU Rockhurst Lincoln UMKC UMSL UMC

Project

Per

cen

t

Posttest scores presented as a percent of pretest scores.

Participant Change in Inquiry Knowledge

Change (post PD-pre PD; n = 124)

Principle diff t

1. Communication 0.1 0.6

2. Social Learning 0.1 1.8

3. Active/Inquiry-Based 0.2 3.2**

4. Prompt Feedback 0.1 1.1

5. Problem Solving 0.1 1.4

6. Content Knowledge 0.2 3.6***

7. Diversity 0.1 1.6*p < .05. **p < .01. ***p < .001

Participant Change in Inquiry Usage

Change (post PD-pre PD; n = 124)

Principle diff t

1. Communication 0.2 6.3***

2. Social Learning 0.4 7.2***

3. Active/Inquiry-Based 0.3 8.4***

4. Prompt Feedback 0.4 9.1***

5. Problem Solving 0.3 6.5***

6. Content Knowledge 0.3 7.6***

7. Diversity 0.2 5.9****p < .05. **p < .01. ***p < .001

Participant Use of Knowledge Based on PD Components

M SD

Content

Improving content knowledge 3.0 0.8

Materials

Using technology effectively to enhance your teaching 2.5 1.1

Developing materials for use with your students 3.0 0.9

Collaboration

Collaborating with other teachers 3.0 1.0

n=1160-4 scale

Impact on Participant Use of Knowledge (cont)

M SD

Assessment    

Assessing student learning 2.6 0.9

Analyzing student performance data 2.6 1.1

Pedagogy

Creating lessons related to content standards 2.8 2.8

Creating inquiry-based / problem-centered classrooms 3.2 0.9

Increasing student motivation 3.0 0.7

Using inquiry-based / problem-centered teaching 3.2 0.7

Participating in classroom activities as your students would 3.0 0.8

How to implement activities 3.0 0.8

Managing inquiry-based / problem-centered classrooms 2.8 0.9

Organization Change--Impact on Higher Education

• Team members from five projects responded to HEI Survey– Establishment of new science courses related

to the PD projects– Establishment of new education courses– Redesign of courses to include more inquiry-

based labs– New or strengthened collaborations between

education and science– Increased grant writing activity on campus

School Level Performance on MAP

• Map Index and % Top 2 Levels

• Served vs not served schools by High Needs status

• Science – 2005-06 compared prior years’ average performance

• Math – no historical comparison possible: examined performance levels by group

Overall Impact of PD Projects

0.7

10.9

3.4

0.20

2

4

6

8

10

12

Teachers* Students

% S

erve

d by

Cyc

le 3

High Needs Served Non High Needs Served

Performance Levels in Science

0

20

40

60

80

100

120

140

160

180

200

High-Need Not High-Need

High-Need Not High-Need

Avg

MA

P In

de

x

PD Schools Not PD Schools

| ------------- Grade 7 ------------- | | ------------- Grade 10 -------------- |

Performance Levels in Science

0

2

4

6

8

10

12

14

16

18

20

High-Need Not High-Need

High-Need Not High-Need

Per

cen

t T

op

2

PD Schools Not PD Schools

| ------------- Grade 7 ------------- |

| ------------- Grade 10 -------------- |

Changes in MAP Science Performance – Index Scores

-6

-4

-2

0

2

4

6

8

High-Need Not High-Need

High-Need Not High-Need

MA

P I

nd

ex

Sc

ore

PD Schools Not PD Schools

| ------------- Grade 7 ------------- |

| ------------- Grade 10 -------------- |

Changes in MAP Science Performance -- Proficiency

-2

-1

0

1

2

3

4

5

High-Need Not High-Need

High-Need Not High-Need

Pe

rce

nt

To

p 2

PD Schools Not PD Schools

| ------------- Grade 7 ------------- |

| ------------- Grade 10 -------------- |

Average MAP Index Scores by Grade Level

680

690

700

710

720

730

740

750

760

4 5 6 7 8 1Grade

Ave

rag

e M

AP

Ind

ex

High Need PD Schools High Need -- Not PDNot High Need PD Schools Not High Need -- Not PD

10

Average MAP Math Proficiency by Grade Level

20

25

30

35

40

45

50

55

4 5 6 7 8 1Grade

Ave

rag

e %

To

p 2

High Need PD Schools High Need -- Not PDNot High Need PD Schools Not High Need -- Not PD

10

Summary of Results• Teachers were overall satisfied with PD

experiences– Valued most: staff, engaging in activities as

students would, opportunity to improve content knowledge, working with other teachers;

– Valued least: lectures,activities geared toward a different grade level or subject matter than they taught, loosely structured follow-up sessions with no clear purpose.

Summary of Results (cont)

• Assessment components less emphasized than content and inquiry components.

• Teachers gained content knowledge

• Evidence of some improved teacher practice attributed to projects.

• Student learning data mixed.

• Evidence of impact on higher education is limited but promising in some projects.

Conclusions: Effective Project Design Features

• Projects demonstrated effective practice to varying degrees.

• Alignment of content emphasis areas between projects and teacher/school needs is critical.

• Shared vision/collaboration with team implemented in a variety of ways.

• Effective emphasis areas: learning science/math through inquiry; collegial learning with teachers; long-term PD activities; sense of community.

Conclusions (cont.)

• The “smorgasbord” approach – while well intentioned seemed difficult to carry out.

• Emphasis on mathematics in overall cycle 3 ITQG program was somewhat limited.

• Individual projects improve over time.

• Evaluator role balance between program and projects continues to be an issue.

Limitations• Necessity of sampling. • Instruments align with overall program not specific

projects.• Low overall response rates

– Implementation Logs– End of Project instruments– Higher education impact

• Overall evaluation vs. project specific.• Lack of and alignment of student achievement

data.• Impact on evaluation due to ongoing team

collaboration with PIs and K-12 partners.

RecommendationsProject Directors:• Continue to build strong relations among PIs and

instructional staff.• Build stronger K-12 partnerships.• Balance content and pedagogy.• Emphasize and provide opportunities for practice

and feedback on classroom assessment.• Encourage participation in evaluation activities.• Take advantage of formative feedback.• Use literature on best practice when designing

and implementing PD.

Recommendations

External Evaluators:• Explore ways to reduce participant time on

evaluation.• Be proactive in working with PIs and K-12

organizations.• Continue to work with PIs through all

phases of evaluation.• Work with MDHE to examine our roles as

evaluators.

Recommendations

MDHE:• Continue funding multi-year projects.• Encourage true partnerships via RFP

wording and reward systems.• Require that the majority of participants

are from high-needs districts.• Require minimum hours of PD per project.• Support PI cross-fertilization of best

practices.

Questions

Copies of the report and Executive Summary available at:

www.pdeval.missouri.edu