presented to Volusia Teachers Organization
AFT Innovation Grant SubcommitteesOctober 28, 2010
Dr. Serena Roberts
Think about the term “teacher effectiveness” and what it means to you
Turn to a partner and share your thinking
Based on what both of you have shared, come up with a common understanding to share with the group
How does your collective understanding of teacher effectiveness relate to teacher evaluations?
How does it relate to assessment-based accountability?
By the end of this session, participants will:Understand terms relating to measurements of
teacher effectivenessBe aware of other state’s value-added models
and the researchers associated with themKnow possible biases of different value-added
modelsBe able to compute simple gain scoresBe prepared to grapple with difficult questions
about teacher effectiveness and ways different measurement models can play out in assessments of teacher effectiveness
Part of a national movement
The role of AFT’s Innovation grant
Current focus with Race to the Top (RTTT) in Florida
2003 Rand Report, Evaluating Value-added Models for Teacher Accountability
“Although VAM holds great promise, it also raises many fundamental and complex issues. . . . If these issues are not adequately addressed, VAM is likely to misjudge the effectiveness of teachers and schools and could produce incorrect generalization about their characteristics, thus hampering systematic effort to improve education” (p. iii).
The entire Spring 2004 issue of Journal of Educational and Behavioral Statistics was devoted to value-added topics.
In April 2008 the University of Wisconsin at Madison hosted the inaugural National Conference on Value-added Modeling.
Value-added assessmentusually refers to a specific technique or assessment
settingValue-added measurement
is usually used in discussing attempts to measure such intangibles as the impact of school environment, the long-term effect of prior teaching, teacher experience, and socioeconomic characteristic
Value-added modelingrefers to the full range of statistical and mathematical
formulations encompassing all value-added assessment techniques
Layered Mixed Effects Models (LMEM)ignore sociodemographics, apportion credit for pupil
learning growth among multiple schools, use incomplete observations
Simple Fixed Effects Models (SFEM)highly correlated to LMEM so use of SFEM is preferred
because of their simplicity; however, use of SFEM rather than HLM can be controversial
Hierarchical Linear Models (HLM)The question of whether to adjust for minority status
and poverty “is, perhaps, the most important issue faced when developing a school accountability system. Either inclusion or exclusion of them is likely to lead to a biased system. Which bias is most tolerable may depend on whether the system is to be a high-stakes one” (Tekwe, et al, 2004, p. 12).
Tennessee Value-Added Assessment System (TVAAS)Measures teacher effectiveness based on student gains,
controlling for socio-economic status (SES) and other factors that influence achievement
Before 2004 the model was criticized for absence of explicit control. In 2004 they introduced controls of SES and demographics. Controls had negligible impact on estimated teacher effects in TVAAS, but not on a SFE estimator with which the TVAAS was compared. Their explanation was that the use of covariance of tests in different subjects and grades results in a student’s history of test performance substituting for omitted background variables.
Uses LMEMSanders
LouisianaRedesign of teacher preparation programs and use
of effectiveness of experienced certified teachers as the point of comparison
Includes aspects of cost-benefit analyses and a skills-as-outcomes model
Looks at which practices are most effective and what characteristics of teachers are most powerfully tied to their instructional success
Coding and analysis of data about practices and characteristics to assess teacher effects
Results to be used to shape hiring practicesHLM uses prior achievement, Limited English
Proficiency, gifted status, section 504 status, free or reduced price lunch status, student attendance, disability status, discipline record
Noell
One way to look at the value one’s teaching adds is to gather data to document how one’s students do when they move to the next course.How do students who earn A’s in reading in your 2nd
grade class do on their 3rd grade FCAT Reading exams?How do students who earn A’s in your Pre-Algebra class
do in Algebra? The value added by changing a teaching practice
sometimes called Response to Intervention (RTI)Non-statistical value added
“When people talk about their experience with a really good teacher, they’re not talking about test scores. . . . They’re talking about a teacher who gave them self-confidence, the ability to learn, an interest and curiosity about certain subjects” (Pallas, 2010, as quoted by Stewart).
Subcommittees:
Value Added
Formative Assessment
Alternative Assessment
Please get into the following groups for this data exercise
Elementary Middle HighAnthony JohnsonBecky PittardTrish Randall
Diane ClowDavid FinkleYania GonzalezLekita Howard
Beth BosseRon PaganoTom RussellMary Thomas
John CashAlba PerezSusan PersisMarilyn Travis
John AtkinsMatt KrajewskiVickie Presley
You will be given a data activity packet consisting of 8 sheets. Directions for each set of data are at the top of each sheet. Please do not look at subsequent sheets until you complete the previous sheet or are told to move to the next.
If you have questions, please do not hesitate to ask for help.
Teacher 1, in School F, has a 74.3 average gain score in reading.
Teacher 2, in School A, has a 150 gain score in reading.
Based on this data, what can you conclude about the effect of each of these teachers on pupil learning?
What other data would be helpful in coming to a conclusion?
The importance of school effect
The New Teacher Project“Teacher effectiveness should consider multiple
sources of data and emphasize student learning outcomes.”
Teacher evaluations and support for effective instruction (October 6, 2010)
Teacher Evaluation 2.0 (October 2010)The Widget Effect: Our National Failure to
Acknowledge and Act on Differences in Teacher Effectiveness (2009)
Wisconsin Center for Education ResearchUsing Teacher Performance Assessments for Human
Capital Management (April 2010)New York Daily News
“New York City plan to grade teachers with ‘value-added’ data is destructive, says Diane Ravitch” (October 25, 2010)
Subcommittees:
Value Added
Formative Assessment
Alternative Assessment
If three teachers are ranked high, average, and low, using value-added methods, what is the difference among them in terms of the average number of test questions answered correctly by their students?
How many more correct answers does it take, on average, to raise a teacher from the low to the average ranking?
What are the factors on which these answers depend?
Which of the 13 RTTT projects require knowledge of what we talked about today for effective implementation?
Expand Lesson Study *Expand STEM Career and Technical Program Offerings Increase Advanced STEM Coursework Bolster Technology for Improved Instruction and
Assessment Improve Access to State Data Use Data to Improve Instruction Provide Support for Educator Preparation Programs Improve Teacher and Principal Evaluation Systems Use Data Effectively for Human Capital DecisionsFocus Effective Professional DevelopmentDrive Improvement in Persistently Low-Achieving Schools** Implement Proven Programs for School Improvement** Include Charter Schools in LEA Planning
One of the results of the Louisiana studies is: It is possible for teacher preparation programs to
prepare new teachers whose students demonstrate achievement comparable to the achievement of students taught by experienced teachers.
How will this effect traditional pay scales?
How will it effect the improved teacher and principal evaluation systems mandated by Race to the Top?
Please move to your grant subcommittee groups.
You will be given a data activity packet consisting of 4 sheets. Directions are on each sheet. Please do not look at subsequent sheets until you complete the previous sheet or are told to move to the next.
If you have questions, please do not hesitate to ask for help.