Embed Size (px)
Citation preview
Measures of Effective Teaching (MET)project
June 20, 2011
z
2
What does it take to build better feedback & evaluation systems? (and why start there, anyway?)
3
4
5
6
7
Teacher Evaluation Scores – Reading
8
Trustworthiness Tests
9
1. Face Validity– Do teachers recognize the observation instrument and other
measures as reflecting qualities of practice they value? 2. Coherence
– Do the measures match your district’s theory of instruction? 3. Predictive Validity
– Do scores on your measures correlate with outcomes that you value, such as gains in student learning?
4. Scoring Reliability– If a different rater had been assigned to the observation or the
assessment, would the score be the same?
10
The Measures of Effective Teaching Project
Two school years: 2009–10 and 2010–11 >100,000 students Grades 4–8: ELA and Math High School: ELA I, Algebra I and Biology
Participating Teachers
11
The MET Project
The Bill & Melinda Gates Foundation launched the Measures of Effective Teaching (MET) project in fall 2009 to test new approaches to measuring effective teaching. The project’s goal is to help build fair and reliable systems for teacher observation and feedback to help teachers improve and administrators make better personnel decisions. With funding from the foundation, the data collection and analysis are being led by researchers from academic institutions, nonprofit organizations, and several private firms and are being carried out in seven urban school districts.
Research Partners
13
Multiple Measures of Teaching Effectiveness
14
15
The MET Project researchHow well can …
• Student Perception Surveys• Student Perception Surveys
• Structured Classroom Observations• Structured Classroom Observations
• Pedagogical Content Knowledge Test
• Pedagogical Content Knowledge Test
• Student Outcomes• Student Outcomes
… produce valid & reliable measures of teaching?
+
++
We asked students.
16
Classroom Observation Using Digital Video
17
Knowledge for Teaching
18
Value-added on each test
Value-added on state assessment
Value-added on supplemental assessments
19
20
Validation Engine
21
System picks observation rubric & trains raters
Raters score MET videos of instruction
Software provides analysis of:•Rater consistency•Rubric’s relation to student learning
Rater Training & Consistency: building trust
Ensuring Reliable Observations
22
Real time (ear-bud) coaching
Direct Feedback
Remote Collaboration
Facilitating Feedback
23
Preliminary Finding #1
24
Student Perceptions
25
CareCare
ControlControl
ClarifyClarify
ChallengeChallenge
CaptivateCaptivate
ConferConfer
TestPrepTestPrep
ConsolidateConsolidate
CareCare
• My teacher makes me feel that s/he really cares about me
• My teacher seems to know if something is bothering me
• My teacher really tries to understand how students feel about things
• My teacher makes me feel that s/he really cares about me
• My teacher seems to know if something is bothering me
• My teacher really tries to understand how students feel about things
ControlControl
• Students in this class treat the teacher with respect
• My classmates behave the way the teacher wants them to
• Our class stays busy and doesn’t waste time
• Students in this class treat the teacher with respect
• My classmates behave the way the teacher wants them to
• Our class stays busy and doesn’t waste time
ClarifyClarify
• If you don’t understand something , my teacher explains it a different way.
• My teacher knows when the class understands, and when we do not.
• My teacher has several good ways to explain each topic that we cover in the class.
• If you don’t understand something , my teacher explains it a different way.
• My teacher knows when the class understands, and when we do not.
• My teacher has several good ways to explain each topic that we cover in the class.
ChallengeChallenge
• My teacher asks students to explain more about the answers they give.
• My teacher doesn’t let people give up when the work gets hard.
• In this class, we learn to correct our mistakes.
• My teacher asks students to explain more about the answers they give.
• My teacher doesn’t let people give up when the work gets hard.
• In this class, we learn to correct our mistakes.
CaptivateCaptivate
• My teacher makes learning enjoyable• My teacher makes learning interesting• I like the way we learn in this class
• My teacher makes learning enjoyable• My teacher makes learning interesting• I like the way we learn in this class
ConferConfer
• My teacher wants us to share our thoughts• Students get to decide how activities are done in
this class
• My teacher wants us to share our thoughts• Students get to decide how activities are done in
this class
ConsolidateConsolidate
• My teacher takes the time to summarize what we learn each day
• The comments that I get on my work in this class help me understand how to improve
• My teacher takes the time to summarize what we learn each day
• The comments that I get on my work in this class help me understand how to improve
TestPrepTestPrep
• I have learned a lot this year about [the state test]• Getting ready for [the state ] test takes a lot of
time in our class
• I have learned a lot this year about [the state test]• Getting ready for [the state ] test takes a lot of
time in our class
26
Students Distinguish Between TeachersPercentage of Students by Classroom Agreeing
Student Perceptions
Top 5 Correlations
27
Survey StatementSurvey Statement CategoryCategoryRankRank
11
22
33
44
55
• Students in this class treat the teacher with respect• Students in this class treat the teacher with respect
• My classmates behave the way my teacher wants them to• My classmates behave the way my teacher wants them to
ControlControl
ControlControl
ControlControl• Our class stays busy and doesn’t waste time• Our class stays busy and doesn’t waste time
ChallengeChallenge• In this class, we learn a lot every day• In this class, we learn a lot every day
ChallengeChallenge• In this class, we learn to correct our mistakes• In this class, we learn to correct our mistakes
3333 • I have learned a lot this year about [the state test]• I have learned a lot this year about [the state test] Test PrepTest Prep
3434 • Getting ready for [the state test] takes a lot of time in our class• Getting ready for [the state test] takes a lot of time in our class Test PrepTest Prep
Preliminary Finding #2
28
Teacher Impact: Real or Random?
29
30
Students with Most Effective Teachers Learn More in School
31
Video example – coming tomorrow!
Video validation
33
InstrumentInstrument DeveloperDeveloper
• Classroom Assessment Scoring System, CLASS• Classroom Assessment Scoring System, CLASS University of VirginiaUniversity of Virginia
• Framework for Teaching• Framework for Teaching Charlotte DanielsonCharlotte Danielson
• Mathematical Quality of Instruction (MQI)• Mathematical Quality of Instruction (MQI) University of MichiganUniversity of Michigan
• Protocol for Language Arts Teaching Observation (PLAT0)• Protocol for Language Arts Teaching Observation (PLAT0) Pam GrossmanPam Grossman
Natl Math & Sci InitiativeNatl Math & Sci Initiative
• Quality Science Teaching (QST)• Quality Science Teaching (QST) Stanford UniversityStanford University
• National Board for Professional Teaching Standards• National Board for Professional Teaching Standards NBPTSNBPTS
• UTeach Observation Protocol (UTOP) • UTeach Observation Protocol (UTOP)
MET Logical SequenceMET Logical Sequence
35
Measures predict
Measures predict
Measures combine
Measures combine
Measures fairly reflect teacherMeasures fairly reflect teacher
Effective Teaching Index
Effective Teaching Index
Teaching Effectiveness Dashboard
Teaching Effectiveness Dashboard
Research Use
Measures reliable
Measures reliable
Measures improve
effectiveness
Measures improve
effectiveness
?Measures stable under pressure
Measures stable under pressure
Diff. Between Top and Bottom Quartile
01
23
4D
ens
ity
-.4 -.3 -.2 -.1 0 .1 .2 .3 .4Teacher Value Added
Top and bottom quartile based on student perceptionsTop and bottom quartile based on student perceptions and value added
37
