Measuring Teacher Effectiveness

Laura Goe, Ph.D.Research Scientist, ETS

Principal Investigator for Research and Dissemination,

National Comprehensive Center for Teacher Quality

Oklahoma Department of EducationOklahoma City, OK August 15, 2012

2

Laura Goe, Ph.D.

• Former teacher in rural & urban schools Special education (7th & 8th grade, Tunica, MS) Language arts (7th grade, Memphis, TN)

• Graduate of UC Berkeley’s Policy, Organizations, Measurement & Evaluation doctoral program

• Principal Investigator for the National Comprehensive Center for Teacher Quality

• Research Scientist in the Performance Research Group at ETS

3

The goal of teacher evaluation

The ultimate goal of all teacher evaluation should be…

TO IMPROVE TEACHING AND

LEARNING

4

Evaluation for accountability and instructional improvement

• Effective evaluation relies on: Clearly defined and communicated standards

for performance Quality tools for measuring and differentiating

performance Quality training on standards and tools- Evaluators should agree on what constitutes

evidence of performance on standards- Evaluators should agree on what the evidence

means in terms of a score

5

Feedback (Coggshall et al., 2012)

“Feedback on whether or not instructional practices are working can come in the form of student learning data, the teachers’ own observations of student engagement, observations from a peer or a coach, a video-taped record of the practice, discussion within a professional learning community, or the results of a formal evaluation.” (p. 6)

Coggshall, J. G., Rasmussen, C., Colton, A., Milton, J., & Jacques, C. (2012). Generating teaching effectiveness: The role of job-embedded professional learning in teacher evaluation. Washington, DC: National Comprehensive Center for Teacher Quality.

http://www.tqsource.org/publications/GeneratingTeachingEffectiveness.pdf

6

Multiple Standards-based Measures of Teacher Effectiveness

• Affords many benefits to a comprehensive evaluation system Ability to triangulate results increases confidence in

evaluation outcomes More complete picture of teacher strengths and

weaknesses Each type of measure provides a different type of

evidence

• All work together to better inform professional development decisions

7

Using surveys for teacher evaluation

• Surveys are unlikely to show much change within a single year This limits usefulness in teacher evaluation as a

“pre-” and “post-” measure But they may be useful for revealing trends over

time For example, a new teacher is likely to improve

in many aspects of classroom management and discipline in the first few years, which is likely to be reflected in survey results

8

Surveys (2)

• The survey results will likely confirm what is already known through observations and other measures, but what is most important is what teachers do with the survey results Discussion with mentor, principal, colleagues Compare results with school, grade, or team

aggregates Support self-reflection & focus on areas in

need of improvement

9

Surveys (3)

• Almost no research on parent surveys• Research on peer surveys is all in higher

education• Research on student surveys is more

advanced, but only on one instrument—Harvard’s Tripod Survey Tripod was part of the Gates-funded

Measures of Effective Teaching study

10

Tripod Survey (1)

• Harvard’s Tripod Survey – the 7 C’s– Caring about students (nurturing productive relationships);– Controlling behavior (promoting cooperation and peer

support);– Clarifying ideas and lessons (making success seem

feasible);– Challenging students to work hard and think hard (pressing

for effort and rigor);– Captivating students (making learning interesting and

relevant);– Conferring (eliciting students’ feedback and respecting their

ideas);– Consolidating (connecting and integrating ideas to support

learning)

11

Tripod Survey (2)

• Improved student performance depends on strengthening three legs of teaching practice: content, pedagogy, and relationships

• There are multiple versions: k-2, 3-5, 6-12 • Measures:

student engagement school climate home learning conditions teaching effectiveness youth culture family demographics

• Takes 20-30 min• There are English and Spanish versions• Comes in paper form or in online version

12

Creating surveys (Over-simplified!)

• Convene a working group including teachers, principals, parents, students, etc.

• Determine which aspects of teaching standards can be measured with survey questions

• Create questions; try to ask some of the same questions in student, peer, parent, and self-assessment surveys for triangulation of responses

• Field-test the survey and have a psychometrician analyze results (IRT model)

• Make adjustments and pilot the survey (no stakes); then make final revisions

13

Using service learning for teacher evaluation

• Service learning has the potential to be useful in teacher evaluation, particularly when trying to measure teachers’ contributions to student learning in non-tested subjects and grades (lower elementary, high school, subjects that are not part of statewide testing)

• The goal should be comparability in results across students and teachers within a district

As much as possible, teachers within the same subject/grade should do the same service learning projects and use the same assessments

14

Service learning (2)

• Great value in service learning Teachers collaborating around student learning across

curriculum and grades Most subject/grade standards can be measures with a

service learning project Learning goals and assessments can be differentiated

for special needs students and ELLs

• This makes it important to select a service learning project that is wide in scope rather than narrow

• May incorporate “place-based” learning strategies Students and teachers strengthen relationships with local

community and learn about local geography and history

15

Service learning (3)

• There is considerable research on service learning, but mostly on outcomes related to the projects

Not a lot of research on student growth that could be directly attributed to teaching practices

• A good service learning project would have multiple measures of student learning over time (pre- and post-) and may include multiple subjects

• Could take the form of a student portfolio scored with a rubric

Scoring processes, audits, etc. would need to be put into place for validity and comparability

16

Creating a service learning template and guidance (oversimplified!)

• Convene a working group including teachers, principals, parents, students, & community leaders

• Determine several broad-scope projects that could take place across a semester or a school year

Using these projects as examples, identify the student learning standards that could be assessed

Identify measures that could show student learning growth on standards (portfolio entries, tests, performances, presentations, group projects, written reports/journals, etc.)

• Field-test projects and measures and analyze results—use to create guidance docs & templates

17

Service learning for teacher evaluation purposes: A possible approach

• Teacher teams and principals meet with district staff to select a service learning project that all or most teachers will participate in throughout the district

• Using state guidance doc and templates, teachers and principals develop plan and timeline for projects

Teachers from grades and subjects work together in teams to select appropriate assessments to measure student learning

• Using state guidance, assessments are scored and compiled and teachers’ contributions to student learning growth can be determined

18

Resources on Service Learning

Bradley, L. R. (2006). Service-Learning: A strategy for school improvement and building effective community partnerships. Columbus, OH: Ohio Department of Education.

http://www.servicelearning.org/filemanager/download/Ohio_Department_of_Education_training_manual.pdf

Colorado Department of Education. (2004). Creating high-performing schools through service-learning: A service-learning trail guide. Denver, CO: Colorado Department of Education.

http://www.ecs.org/clearinghouse/57/13/5713.pdf

Davis, K. M., Miller, M. D., & Corbett, W. T. (1998). Methods of evaluating student performance through service learning. Gainesville, FL: College of Education, University of Florida.

http://www.fsu.edu/~flserve/resources/methods/MethodsEvaluatingStudentPerformanceThroughSL.pdf

Ernst, J. A. (2004). Prairie Science Class evaluation report. Fergus Falls, MN: Prairie Wetlands Learning Center.

http://meera.snre.umich.edu/sites/all/files/Prairie_Science_Class_Evaluation_Report.pdf

19

Teacher portfolios

• It’s not a “brag book,” it’s evidence of how the teacher sets/pursues meaningful goals related to improving knowledge & instructional practice

• More like a year-long self-exploration of one or more standards related to aspects of teacher’s practice such as:

Restructuring the classroom learning environment to support diverse learners’ needs

Working with a collaborative team to improve reading outcomes

Developing expertise in classroom technology to support new methods of instruction

20

Portfolios (2)

• Portfolios do not need to be “physical” binders but can be virtual evidence collections

This is easier for most teachers to create and share and easiest for others to review

Allows teachers to include multiple types of data (videos, spreadsheets, recordings, photographs, work samples, planning documents, self-reflection, peer comments, etc.)

• Teachers should receive training in Choosing standards to focus on Choosing a way to demonstrate growth on the standards Selecting evidence to include in the portfolio

21

Portfolios (3)

• State’s role will be to provide guidance docs, templates, and examples of virtual portfolios at various score levels (like “unsatisfactory” and “exemplary”)

• Principals and others involved in scoring portfolios will need to be trained to score portfolios with accuracy and consistency in order for results to be valid indicators for teacher evaluation purposes

• Teachers and principals should be involved in the development, and the process should be piloted (no stakes)

22

Some “popular” observation instruments

Charlotte Danielson’s Framework for Teaching

http://www.danielsongroup.org/theframeteach.htm

CLASS

http://www.teachstone.org/

Kim Marshall Rubric

http://www.marshallmemo.com/articles/Kim%20Marshall%20Teacher%20Eval%20Rubrics%20Jan%

Marzano Teacher Evaluation Framework

http://www.marzanoevaluation.com/

23

Example: University of Virginia’s CLASS observation tool

Emotional Support Classroom Organization

Instructional Support

Pre-KandK-3

Positive Climate

Negative Climate

Teacher Sensitivity

Regard for Student (Adolescent) Perspectives

Behavior Management

Productivity

Instructional Learning Formats

Concept DevelopmentQuality of FeedbackLanguage Modeling

Upper Elementary/Secondary

Content UnderstandingAnalysis and Problem

SolvingQuality of Feedback

24

Domain 1: Planning and Preparation includes comprehensive understanding of the content to be taught, knowledge of the students’ backgrounds, and designing instruction and assessment.

Domain 3: Instruction is concerned with the teacher’s skill in engaging students in learning the content, and includes the wide range of instructional strategies that enable students to learn.

Domain 2: The Classroom Environment addresses the teacher’s skill in establishing an environment conducive to learning, including both the physical and interpersonal aspects of the environment.

Domain 4: Professional Responsibilities addresses a teacher’s additional professional responsibilities, including self-assessment and reflection, communication with parents, participating in ongoing professional development, and contributing to the school and district environment.

Example: Charlotte Danielson’s Framework for Teaching

25

Domain 1 from “The Four Domains of the Marzano Teacher Evaluation Model”

26

Example: Kim Marshall’s Rubric

Planning & Preparation for Learning

Highly Effective Effective Improvement Necessary

Does Not Meet Standards

a. Knowledge

Is expert in the subject areaand has a cutting-edge graspof child development and howstudents learn.

Knows the subject matter welland has a good grasp of childdevelopment and howstudents learn.

Is somewhat familiar with thesubject and has a few ideas ofways students develop andlearn.

Has little familiarity with thesubject matter and few ideason how to teach it and howstudents learn.

b. Strategy Has a well-honed game planfor the year that is tightlyaligned with state standardsand assessments.

Plans the year so students willmeet state standards and beready for externalassessments.

Has done some thinking abouthow to cover high standardsand test requirements thisyear.

Plans lesson by lesson and haslittle familiarity with statestandards and tests.

27

Creating an observation instrument

• Consider the research: lots of evidence on use of Framework for Teaching and CLASS, little for other instruments

Don’t reinvent the wheel

• Working group to design instrument should include teachers, principals, other stakeholders

• Focus on teaching standards What evidence will you need to be able to score the

teacher’s performance on this standard?

• Field test and pilot instrument• Training for scoring is crucial for valid results

28

Questions?

29

Laura Goe, Ph.D.609-619-1648lgoe@ets.orgwww.lauragoe.comhttps://twitter.com/GoeLaura

National Comprehensive Center for Teacher Quality1000 Thomas Jefferson Street, NWWashington, D.C. 20007www.tqsource.org