Measures of Effective Teaching (MET)project

November 2011

z

2

What does it take to build better feedback & evaluation systems? (and why start there, anyway?)

3

4

5

6

7

Trustworthiness Tests

8

1. Face Validity – Do teachers recognize the observation instrument and other

measures as reflecting qualities of practice they value?

2. Coherence – Do the measures match your district’s theory of instruction?

3. Predictive Validity – Do scores on your measures correlate with outcomes that you

value, such as gains in student learning?

4. Scoring Reliability – If a different rater had been assigned to the observation or the

assessment, would the score be the same?

9

Multiple Measures of Teaching Effectiveness

10

Knowledge for Teaching

11

12

The MET Project research

How well can …

• Student Perception Surveys

• Structured Classroom Observations

• Pedagogical Content Knowledge Test

• Student Outcomes

… produce valid & reliable measures of teaching?

+

+

+

Preliminary Finding #1

13

Student Perceptions

14

Care

Control

Clarify

Challenge

Captivate

Confer

TestPrep

Consolidate

Care

• My teacher makes me feel that s/he really cares about me

• My teacher seems to know if something is bothering me

• My teacher really tries to understand how students feel about things

Control

• Students in this class treat the teacher with respect

• My classmates behave the way the teacher wants them to

• Our class stays busy and doesn’t waste time

Clarify

• If you don’t understand something , my teacher explains it a different way.

• My teacher knows when the class understands, and when we do not.

• My teacher has several good ways to explain each topic that we cover in the class.

Challenge

• My teacher asks students to explain more about the answers they give.

• My teacher doesn’t let people give up when the work gets hard.

• In this class, we learn to correct our mistakes.

Captivate

• My teacher makes learning enjoyable • My teacher makes learning interesting • I like the way we learn in this class

Confer

• My teacher wants us to share our thoughts • Students get to decide how activities are done in

this class

Consolidate

• My teacher takes the time to summarize what we learn each day

• The comments that I get on my work in this class help me understand how to improve

TestPrep

• I have learned a lot this year about [the state test] • Getting ready for [the state ] test takes a lot of

time in our class

Student Perceptions

Top 5 Correlations

15

Survey Statement Category Rank

1

2

3

4

5

• Students in this class treat the teacher with respect

• My classmates behave the way my teacher wants them to

Control

Control

Control • Our class stays busy and doesn’t waste time

Challenge • In this class, we learn a lot every day

Challenge • In this class, we learn to correct our mistakes

33 • I have learned a lot this year about [the state test] Test Prep

34 • Getting ready for [the state test] takes a lot of time in our class Test Prep

Preliminary Finding #2

16

Teacher Impact: Real or Random?

17

•For each teacher, calculate VAM using ONE class of students from ‘09-’10 school year

• Identify top and bottom quartile on VAM Heads or Tails

•For elementary teachers, RECALCULATE VAM using LAST year’s students (‘08-’09 school year)

•For secondary teachers, RECALCULATE VAM using DIFFERENT students from another class (same course, same year)

Flip Again

•Compare RECALCULATED VAM scores to see if differences in student performance persist. Compare

18

Students with Most Effective Teachers

Learn More in School

19

MET Logical Sequence

20

Measures predict

Measures combine

Measures fairly reflect teacher

Effective Teaching Index

Teaching Effectiveness Dashboard

Research Use

Measures reliable

Measures improve

effectiveness

?

Measures stable under pressure

21

Observation System Design

22

Establish ACCURATE

DATA LINKAGES

Define EXPECTATIONS

FOR TEACHERS

Ensure OBSERVER ACCURACY

Ensure RELIABLITY OF

RESULTS

Determine ALIGNMENT

WITH OUTCOMES & other measures

Quality Information

Validation Engine

23

System picks observation rubric & trains raters

Raters score MET videos of instruction

Software provides analysis of: • Rater consistency • Rubric’s relation to

student learning

Rater Training & Consistency: building trust

• Calibrate • Refine

• Test • Train

Video examples for anchor points

Rater certification: Don’t pass, don’t rate

Periodic tuning: Out of

tune, don’t rate

Adjustments to observation

framework based on data

Ensuring Reliable Observations

24

25

Video validation

26

Instrument Developer

• Classroom Assessment Scoring System, CLASS University of Virginia

• Framework for Teaching Charlotte Danielson

• Mathematical Quality of Instruction (MQI) University of Michigan

• Protocol for Language Arts Teaching Observation (PLAT0) Pam Grossman

Natl Math & Sci Initiative

• Quality Science Teaching (QST) Stanford University

• National Board for Professional Teaching Standards NBPTS

• UTeach Observation Protocol (UTOP)

27

The Measures of

Effective Teaching Project

Two school years: 2009–10 and 2010–11

>100,000 students

Grades 4–8: ELA and Math

High School: ELA I, Algebra I and Biology

Participating Teachers

28

The MET Project

The Bill & Melinda Gates Foundation launched the Measures of Effective Teaching (MET)

project in fall 2009 to test new approaches to measuring effective teaching. The project’s goal is

to help build fair and reliable systems for teacher observation and feedback to help teachers

improve and administrators make better personnel decisions. With funding from the foundation,

the data collection and analysis are being led by researchers from academic institutions,

nonprofit organizations, and several private firms and are being carried out in seven urban

school districts.

Our primary collaborator include: • Mark Atkinson, Teachscape

• Nancy Caldwell, Westat

• Ron Ferguson, Harvard University

• Drew Gitomer, Educational Testing Service

• Eric Hirsch, New Teacher Center

• Dan McCaffrey, RAND

• Roy Pea, Stanford University

• Geoffrey Phelps, Educational Testing Service

• Rob Ramsdell, Cambridge Education

• Doug Staiger, Dartmouth College

Other key contributors include: • Joan Auchter, National Board for Professional

Teaching Standards

• Charlotte Danielson, The Danielson Group

• Pam Grossman, Stanford University

• Bridget Hamre, University of Virginia

• Heather Hill, Harvard University

• Sabrina Laine, American Institutes for Research

• Catherine McClellan, Educational Testing Service

• Denis Newman, Empirical Education

• Raymond Pecheone, Stanford University

• Robert Pianta, University of Virginia

• Morgan Polikoff, University of Southern California

• Steve Raudenbush, University of Chicago

• John Winn, National Math and Science Initiative

Research Partners

www.metproject.org

info@metproject.org

30

31

Students Distinguish Between Teachers Percentage of Students by Classroom Agreeing

Still more… 1. Many short observations vs. fewer longer. 2. Weighting: How much weight to attach to each component? (Spring 2012) 3. 2nd Year Randomization: Did combined measures control for the relevant student characteristics? (Spring 2012)

32

Diff. Between Top and Bottom Quartile

01

23

4

De

nsity

-.4 -.3 -.2 -.1 0 .1 .2 .3 .4Teacher Value Added

Top and bottom quartile based on student perceptions

Top and bottom quartile based on student perceptions and value added

34